Sunday, July 14, 2019

#CodeConfident: Journey

As designed in my experiment for my #CodeConfident challenge, I have now five GitHub repositories of practice projects. As I had hoped, these five were already sufficient to feel confident enough to start my proof of concept project: to implement a new small product from scratch. May I present to you: my first steps of building Journey, a learning journal application.

Pre-implementation Thoughts

  • initial brainstorming
  • May 28
    • think about presentation already, maybe app about my challenge? reasons why, etc? people I collaborated with, etc. - basically built my presentation? or maybe as addition... and private repo in that case before I make it public
    • tech stack: Java, Spring, Hibernate & Angular, TypeScript, Redux flow & in memory or docker SQL database - basically what we have at work; cover with tests!!! make it especially testable
    • model the architecture!
    • have a rough goal in mind; maybe even fulfill a purpose I have myself, like keeping a list of good reads or bookmarks of things I want to do? these already exist.. does not need to be new, however!
    • then let the app evolve
    • MVP
    • did story mapping? 
    • Define a small scope first and start
    • Idea!! A coding journal app!
      • I know more what I would love to have
      • Would have free text fields, date picker etc.
      • Make it more general: a learning tracker! See your progress, visualize
      • Lots of potential features, can really be extended
      • Maybe also help define your challenge! :D
      • "You are awesome"
      • Growth
      • Add motivational cheers per day
    • Do story map, then architecture
      • Have it documented
    • For GitHub: use project this time to track progress
  • May 29
    • created mind map with all ideas around app
  • June 11
  • June 15
    • Journey is used for a blog engine, a migration tool, a router for rails, journey planner, game, MVC tool, native diary app
    • Journey
    • Ready to start :-)
    • Use the product to run all sorts of tests for, exploratory and automated, accessibility, security, static code analysis, etc.
    • Check out the security / vulnerability scanner snyk

June 17

  • created GitHub repo
  • started to initialize project with Spring Boot: adding the same dependencies as for
  • decided to use JHipster instead to quickly generate the project with the complete desired tech stack:
  • found that there's also an online generator:
  • found they recommend for Java builds
  • still wanted to use the local installation
  • found that jhipster command is not recognized on Windows, same for yeoman; normally that's an issue with the path, could not figure out why yet; checked the path and it correctly pointed to node; found that this issue can occur with multiple node instances so the package install location might be different:; learned that you can see where your package is installed: npm ls -g yo (; added this to my path, and now the commands are recognized!
  • generated project: monolithic, no JHipster registry, JWT authentication, SQL, MySQL, MySQL, ehcache, Hibernate 2nd level cache, Gradle, API first development with generator, Angular, default JHipster theme, no internationalization, Protractor, no other generators

    added 1845 packages from 1540 contributors and audited 892246 packages in 472.857s
    found 1 moderate severity vulnerability
      run `npm audit fix` to fix them, or `npm audit` for details
    Application successfully committed to Git.

    If you find JHipster useful consider sponsoring the project

    Server application generated successfully.

    Run your Spring Boot application:
    ./gradlew (gradlew if using Windows Command Prompt)

    Client application generated successfully.

    Start your Webpack development server with:
     npm start

    > journey@0.0.0 cleanup D:\Dev\journey
    > rimraf build/resources/main/static/ build/resources/main/aot

    INFO! Congratulations, JHipster execution is complete!
  • ran: npm audit fix
  • noticed: could not specify Java and Angular versions; got Java 8 & Angular 8
  • generated nice readme with instructions
  • ran: npm install
  • in 2 terminals:
    npm start
  • --> need to configure DB first, then go through everything and adapt, then commit

June 18

  • renamed package; IDE refactoring did not catch all cases, app startup told what to correct
  • learned how to start the DB in Docker (for Windows with a backslash instead):
    docker-compose -f src/main/docker/mysql.yml up
    ( but can also be found in readme file)
  • checked if app can now connect with DB: successful :)
  • updated readme
  • cleaned up for commit
  • TODO: follow further instructions to model entities etc.

June 20

  • GitHub & npm install informed about security vulnerability; still exists after upgrading Swagger UI
  • read more about how to get started with JHipster
  • started to generate first entity
  • thought about adding a date field, yet decided not to go this direction yet to avoid the whole complexity of dates and times; added only fields title and description in the end (id got generated automatically)
  • found Google calendar entries event descriptions allow up to 8192 characters, so decided to set the maximum to the 8000
  • decided to add filtering and pagination, went for links
  • finished configuring the entity, was generated successfully (only a few warnings due to the upgraded Swagger UI were thrown)
  • checking the changes, JHipster indeed did most of the stuff already! Amazing what code generators can do for you already, bootstrapping whole apps; feeling I cheated here, yet I know that a tool is there to support and at work we also use generators, e.g. for Angular; it simply saves lots of time and also lets me have a base to extend on; might bereft me of some learnings, yet I can still do that later on anyway
  • generated tests are passing
  • ran the app; database connection failed; rebuilt the database container, didn't help; couldn't create the bean due to a missing table:
    Unable to build Hibernate SessionFactory; nested exception is org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation:
     missing table [jhi_date_time_wrapper]
  • connected to the database, saw that no tables had been created at all
  • stopped DB again and recreated it from scratch (without option -d)
  • startup still fails as tables are missing like authority, yet the DB already shows a few tables; saw Liquibase runs asynchronously while bean creation already failed
  • not making sense of reading the exceptions or internet searches; assumption: the previous sample implementation interferes with my new one
  • made liquibase configuration synchronous; had the same issue (only took longer); seems unrelated with data
  • back to the start: cloned the repo again, started up the app
  • old app successfully set up database; starting the new app, it complained about missing table jhi_date_time_wrapper; found the related liquibase changeset was marked to be run for a test context only; removed that, ran again; now the new app starts up just as expected
  • tried again with
    docker-compose -f src\main\docker\mysql.yml down
  • still not working, no matter if the changeset goes with or without the test context set
  • what I know: it runs when the db is set up properly; it fails when running the changesets
  • arrrgh, found a change in one of the data sets: had typed a character there by mistake ^^
  • nope, wasn't that.. :(
  • tried to run liquibase separately, but cannot find mysql connector
  • don't have a properties file
  • tried to add dependency like in, did not help
            classpath "mysql:mysql-connector-java:8.0.16"
  • feeling I'm going down a rabbit hole; focus again on app startup
  • interesting links:
  • tried commenting out fake csv data yet it failed as well
  • needed to remove the test context from the jhi_date_time_wrapper table creation
  • the first changeset is the culprit, it fails at different steps, afterwards it works; have to make it work in one run instead; seems to be a timeout until connection is cut off
  • tried again from scratch, including the jhi_date_time_wrapper changeset and having liquibase configured synchronously before app startup - and it worked!!!
  • tested it out carefully again, removing settings I tried out; found that it was indeed only the combination of having liquibase configured synchronously and to include the test changeset as somewhere it was expected (while it wasn't before), plus resetting the database completely or releasing the database changelog lock
  • lesson learned: read exceptions more carefully, search better, test out your assumptions one at a time
  • committing for now, will have to clean up all previous sample data later
  • interesting log messages and exceptions:
    2019-06-21 00:36:19.983 WARN 6040 --- [ journey-task-1] i.g.j.c.liquibase.AsyncSpringLiquibase : Starting Liquibase asynchronously, your database might not be ready at startup!

    2019-06-21 00:36:22.559 WARN 6040 --- [ journey-task-1] com.zaxxer.hikari.pool.ProxyConnection : Hikari - Connection com.mysql.cj.jdbc.ConnectionImpl@10549e00 marked as broken because of SQLSTATE(08S01), ErrorCode(0)

    com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
    The last packet successfully received from the server was 139 milliseconds ago. The last packet sent successfully to the server was 139 milliseconds ago.

    Caused by: org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: missing table [jhi_authority]

    2019-06-21 00:36:22.542 WARN 6040 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.

    ApplicationContextException: Unable to start web server; nested exception is java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'servletEndpointRegistrar' defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration.class]: Bean instantiation via factory method failed;
    nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method 'servletEndpointRegistrar' threw exception;
     nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'auditEventsEndpoint' defined in class path resource [org/springframework/boot/actuate/autoconfigure/audit/AuditEventsEndpointAutoConfiguration.class]: Unsatisfied dependency expressed through method 'auditEventsEndpoint' parameter 0;
  • TODO: remove unused sample parts like user and registration (or keep it? nah, not the idea of the app, can re-implement in case direction changes, or still?) nah, reduce code where feasible! always a good idea, can look things up later again
  • TODO: cover with tests
  • TODO: adapt functionality where desired

June 21

  • explored more what the bootstrapped app already offers - and it's a lot! comes with user management, metrics and more; also: responsive!!
  • lots of directions to move on to, like adjusting pages, using nav bar instead of menu for journal, increasing the description field size, ... lots of feature ideas in my mind map as well, also for further entities
  • lots of things to test as well! e.g. lots of interesting output in the DevConsole (including errors ;) e.g. for health check, configuration, logs, metrics update... and the swagger warnings, have to solve them)
  • TODO: fix warnings and errors

June 22

  • pairing session with Amber Race
  • introduced her to the current state after bootstrapping the app and generating the first custom entity, journal entry
  • we agreed to switch who's at the keyboard from time to time, and did so
  • Amber was interested to explore frontend development, and so was I
  • I suggested a few things we could improve on frontend side for journal entries: hiding the id on the frontend, increasing the description input text box size to a multi-line element, adding a date input field
  • we started out with hiding the id for updating entries and could simply do it by removing the related html part
  • we wanted to change the description input to a textarea field, thinking this was an easy change as well - and we stumbled; it took us quite some trial and error and googling until we found out that textarea needed an explicit closing tag! Too easy to be true - seems you have to stumble here and then you know it; at least we got around implementing a custom component just for that
  • after implementing the textarea, we thought what happens if we press Enter in the field, does it create a new line or instantly submit the form? We found it does create new lines, yet we saw an error thrown in the console:
    Assertion failed: Input argument is not an HTMLInputElement
    Uncaught TypeError: Cannot read property 'type' of undefined
        at e.setFieldValue (onloadwff.js:71)
        at HTMLFormElement.formKeydownListener (onloadwff.js:71)

    I knew this error from work and even reported it as issue as it occurred quite frequently when using keyboard navigation; we googled the error and found out it's caused by the LastPass Chrome extension! o_o Tried it in FireFox, and indeed it didn't happen here. Guess I'm owing an apology to my fellow developers.. Lesson learned: google error messages before reporting them
  • we wanted to add a new field when updating a journal entry, a date; we created a new form-group div with an input of type "datetime-local" and got errors; right, we're missing the field in the related TypeScript file as well, so added it here too; still we got errors; we found that the formControlName attribute is linked to the TypeScript field so they have to match; googling let us try the input type "date" instead - and we found it worked! It even automatically offered us a date picker :D of course the backend part needs to be implemented as well, so we can't save the value, but that's for another day
  • we still saw an error when loading the page, yet it seems to be one of these that the bootstrapped app offers yet did not implement: http://localhost:9000/management/info; have to dig into this later
  • Retrospective:
    • Amber: that was fun! Learned something about Angular and TypeScript, how it's wired together, has some picky things that you have to learn
    • Lisi: agree, all this is a test for me how far I can get puzzling pieces of knowledge and what I learn from googling together without doing formal tutorials; seems in the end I have to go back to tutorials, yet then I often end up only knowing how to do things within the clinical environment of a tutorial, not in a real app
    • Amber: sometimes you have to learn the hard way
    • Lisi: felt that bootstrapping the app to have it as practice playground is like having a legacy app where you have to find your way through, like when starting a new job

June 22 (cont)

  • received a pull request from my learning partner Toyer Mamoojee; clarified that the tests had been auto-generated; will look into the PR when I come to this part
  • fixed SwaggerUI warnings after upgrade to new version by adapting the webpack.common.js paths
  • added new date field to the rest of frontend logic, adapted tests
  • copied backend part as far as I could; used Date in the beginning, stumbled with filters which used local dates; checked out the project one more and generated a test entity with a date field with JHipster to see how they would do it; using Java LocalDate and Moment in the frontend; adapted implementation accordingly
  • running liquibase for generating a changeset automatically failed; did it manually for now, to be investigated
  • nearly worked from the start! only had to make date nullable for the beginning as existing fields are not set yet
  • tried it out: date cannot be saved, bad request; no value had been sent and the field must not be null; sent API call directly via Postman, same result; compared again, found typo in DTO setter... o_O and that was it!
  • now working, even sorting (smoke test only ;) )
  • added not null constraint on database field; learned I need to specify the data type and a default value for those which are currently null
  • running all kind of existing tests; integration tests for journal entry fail, all other pass; found I never ran the protractor tests; installed and ran them (found alternative to running "npm run e2e":)
    npm install -g protractor
    webdriver-manager update
    webdriver-manager start
    protractor src\test\javascript\protractor.conf.js

    complained that mocha could not be found:
    E/launcher - Error: Error: Cannot find module 'mocha'

    npm install mocha

    did not help; installing globally and then running "npm install" again, however, worked! tests running now --> 2 tests fail, for logs and configuration
  • newly generated tests are passing; old ones fail, filter does indeed not work as expected, returns all existing entries; what did I forget to adapt?? copied over again criteria and query service from generated sources - now the tests are passing, seems there was a tiny little mistake somewhere I could not see
  • TODO: fix logs and configuration implementation

June 23

June 24

  • checking out project on Mac, new run
  • found that now profiles are working! but Swagger UI not; ran open api generation again, still not working; what else did I do??
  • tried again with existing project; decided to remove all generated parts like node modules and build etc.; after having everything regenerated the profile service is working again!! now Swagger does not work, same situation
  • logs show huge number of loggers but none displayed, filter does not do anything
  • Swagger errors:
    Refused to apply style from 'http://localhost:9000/swagger-ui/dist/css/typography.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
    swagger-ui.min.js:1 Failed to load resource: the server responded with a status of 404 (Not Found)
    GET http://localhost:9000/swagger-ui/dist/lib/swagger-oauth.js net::ERR_ABORTED 404 (Not Found)
    Uncaught ReferenceError: $ is not defined
        at index.html:33
  • tried to adapt the Swagger UI index.html to what is available; getting different errors now
    Uncaught ReferenceError: require is not defined
    :9000/swagger-ui/index.html:17 Uncaught ReferenceError: $ is not defined
    docs:1 Refused to display '' in a frame because it set 'X-Frame-Options' to 'sameorigin'.
  • when clicking on Swagger, I get " refused to connect"
  • could not solve issue yet;
    Uncaught ReferenceError: require is not defined
        at Object.<anonymous> (swagger-ui.js:1)
  • TODO with Amitai: solve SwaggerUI issue, move journal to navbar, create new entity to define challenge and experiment, checklist for evaluation and progress tracking, re-brand from JHipster to journey

June 25

  • pairing with Amitai Schleier on journey project
  • Lisi presented what had been done, what got generated and what we changed, also options to move forward; also shared struggles upgrading SwaggerUI
  • Amitai: I have a learned behavior: whatever is annoying, your feeling is telling you the truth; it's annoying now it will be annoying later; so let's fix SwaggerUI
  • Amitai: can we go back to how it was before? --> Lisi showed him the changes in the package.json and the wepack common js, also addressed the changes in the index.html
  • Amitai: first thing I notice it's a major version bump
  • looked for an upgrade guide together, didn't find too promising results
  • tried to find official documentation
  • decided to switch to figuring out what's actually going on, look at what we have
  • Amitai: I follow my curiosity until it seems like taking energy, always looking for the easy path; it's also fine to use all willpower and push through, sometimes does that as well, yet tend to follow his energy more
  • looked at the generated sources when the app was built; did not find anything related in the build directory, looked at the node module; found I missed to copy a JavaScript file in the Webpack configuration; added this, yet no difference
    { from: './node_modules/swagger-ui/dist/swagger-ui.js', to: 'swagger-ui/dist/swagger-ui.js' },
  • again googled for a tutorial, didn't find a promising thing
  • Amitai: we haven't asked the right questions yet; we should look at what we have, it will tell us the right questions
  • adapted the index.html to call the SwaggerUI JavaScript only, so no 404 errors anymore; also checked that the script could be loaded successfully
  • now had two different errors:
    external "deep-extend":1 Uncaught ReferenceError: require is not defined
        at Object. (external "deep-extend":1)
        at __webpack_require__ (bootstrap 6cb5c7f86a6d6e7d420d:19)
        at Object. (index.js:1)
        at __webpack_require__ (bootstrap 6cb5c7f86a6d6e7d420d:19)
        at Object. (all.js:16)
        at __webpack_require__ (bootstrap 6cb5c7f86a6d6e7d420d:19)
        at bootstrap 6cb5c7f86a6d6e7d420d:62
        at bootstrap 6cb5c7f86a6d6e7d420d:62
        at webpackUniversalModuleDefinition (universalModuleDefinition:9)
        at universalModuleDefinition:1
    (anonymous) @ external "deep-extend":1
    __webpack_require__ @ bootstrap 6cb5c7f86a6d6e7d420d:19
    (anonymous) @ index.js:1
    __webpack_require__ @ bootstrap 6cb5c7f86a6d6e7d420d:19
    (anonymous) @ all.js:16
    __webpack_require__ @ bootstrap 6cb5c7f86a6d6e7d420d:19
    (anonymous) @ bootstrap 6cb5c7f86a6d6e7d420d:62
    (anonymous) @ bootstrap 6cb5c7f86a6d6e7d420d:62
    webpackUniversalModuleDefinition @ universalModuleDefinition:9
    (anonymous) @ universalModuleDefinition:1

    index.html:16 Uncaught ReferenceError: $ is not defined
        at index.html:16
  • Amitai: this message doesn't make me feel like I'm supposed to read this; Lisi: we're not the target audience
  • Amitai: idea: set up a throw-away project with Swagger 3 to see how it should work; partition the complexity of the problem
  • searched for example projects, did not find any suitable ones
  • Amitai: code generation is a tricky bargain, it provides you lots of code you don't want to write, yet you don't want that much code; Lisi: really relate to that, had the feeling that I instantly entered a legacy project where I needed to understand first what's going on
  • looked at the script called in the index.html
  • what is Springfox?
  • googled the error message, yet did not find something meaningful
  • next question is what does the $ in JavaScript mean?
  • Reflecting:
    • Amitai: lots of things are interacting with each other, we need to know a lot more than we do; it's not clear where the boundaries are; I'm working at my best when the lack of knowledge is very contained
    • Lisi: what if I would remove Swagger completely from the project and start again? Amitai: probably the same; would rather pay someone; would look for someone who knows Swagger and pair with them
    • Amitai: if this were our job I'd come back rested and create a small project using SwaggerUI 3 from scratch, just to understand and throw away; then come back to this; the next thing would be what is going on with Webpack; but first let me write some code with my own hands
    • Lisi: feeling I really lack lots of knowledge, have these bits and pieces here and there and trying to puzzling them together, yet there's so much white space; Amitai: I need a certain amount of knowledge to be able to guess well
    • Amitai: this time I drove a lot of the questions, was that okay with you? Lisi: also noticed I stayed at the keyboard, hoped it was okay; for me it was really good to have someone else navigating as I tried already a lot and needed that fresh perspective and differences how to move on; Amitai: really comfortable in asking questions
    • Lisi: although we didn't have a happy end, this session was really valuable for me to see it really is okay that I could not fix the issue already, not that easy

June 25 (cont)

June 27

  • compared how index.html delivered with the npm package calls SwaggerUI; difference is that authentication is required for local app
  • ran new copy of the app with old version again to see the difference; npm package is highly different, also shows files that the new version does not have anymore; the API URLs you can check there are:
  • checked the JHipster sample app repos, all of them still show the old Swagger version
  • found that the script needs the jQuery scripts to run, otherwise I get the same error:
    Uncaught ReferenceError: $ is not defined
  • it's still a mystery; I learned the index.html needs to be re-written to work with the new version yet also allow authentication; also, that webpack might be another culprit; --> I put this on ice until I learned more about JavaScript to be able to fix this, or until JHipster releases a new version fixing the security vulnerability here
  • TODO: go on developing the app for now

July 1

July 2

  • decided to first create a new entity so the product takes more shape and later solve the other stuff (they will bite me, I'm sure of that; yet the end product also does not have to be perfect, it's about learning after all)
  • created new entity "Challenge" with fields that I used when defining mine
  • used the JournalEntry entity as a template, then copy & paste & adapt
  • generation of getters and setters is super useful (to call the generation popup in IntelliJ Windows edition: Alt+Insert)
  • noticed one more that IntelliJ used the asterisk import; searched again for the setting to always use explicit imports instead of wildcards:
  • got lots of unused imports, shortcut is really handy: for IntelliJ Windows it's Ctrl+Alt+O
  • finished backend part without test, tried to run the app
  • due to copy & paste I used varchar for the database update script although I needed a type that can store more data like text
  • also, caught copy & paste error in liquibase script; really need to follow-up to be able to generate the scripts automatically
  • as I ran the liquibase script halfway through before I got a checksum error; just deleted database and started from scratch, easiest thing without productive data
  • app is running! :D
  • smoke test for the new controller, called API, happy path was working
  • TODO: add relation of journalEntry to challenge
  • TODO: create tests for new controllers
  • TODO: create frontend part for challenge

July 3

July 6

July 8

  • checked existing UserMapper integration test
  • found that my new mappers for journal entry and challenge had a lot less logic, yet one part was the same; for user mapper a test had been generated, for the new entities not
  • implemented the same test for both new mappers
  • TODO NEXT: add implementation in frontend

July 9

  • tackling the frontend implementation for the challenge entity now
  • creating new challenge model, adapting the journal entry model
  • found that also the journal entry model showed attributes as optional although they shouldn't be from my perspective; learned later in the routing that this way you can call an empty constructor, so changed all to optional
  • learned shortcut in Visual Studio Code for Windows: Alt+Up/Down moves line
  • learned that optional parameters must follow all required ones
  • now trying to first make the existing part for journal entry work again before implementing the challenge frontend part
  • did not work at first try, didn't catch everything; had a look at the generated project to learn what's missing; missed quite a lot; learned they do the mapping via ids
  • seems I have to implement the rest of challenges as well to make journal entries work again; did that, still got one error in the journal update component ngOnInit; probably a copy & paste error; and it was, including a missing method
  • Visual Studio Code shortcut to optimize imports is similar to IntelliJ's: Shift+Alt+O
  • Visual Studio Code shortcut to duplicate the line down: Shift+Alt+Down
  • added all challenge fields in frontend, adapted to optional and textarea where desired
  • committed!! :D
  • thought: the copy & paste approach might be questionable, yet it gave me the working template needed to be confident enough to write the initial set, make it work, and then to do changes myself; what helped was the sheer repetitive practice of copying over, replacing all instances by hand so I really had to read the code line by line; sometimes used find and replace to reduce typos yet then I was really oblivious what the code was doing
  • lots of open todos, yet I feel it's really time for the first blog entry about this project
  • TODO: show tag instead of id for journal entry (all pages)
  • IDEA: don't show fields on detail page if they are empty?
  • TODO: adapt tests and cover the rest
  • TODO: show decent error message when you try to delete a journal entry that is linked to a challenge or vice versa
  • TODO: investigate error message on page load ("Error: Uncaught (in promise)")
  • TODO: test behavior a lot further, discover problems and risks and further ideas

July 13

To Be Continued...

This coding journal post already got way too long. Obviously, there are still lots of open todos, even for a first rough minimal application that does not need to claim perfection. Therefore my coding journal will proceed in upcoming posts.

On Another Note

I am super excited to announce that Gem Hill just started her own testing tour focused on security testing! We had a great session on my testing tour last year, and now she decided to kick off her own tour. Can't wait to learn more about it, and hope to make a guest appearance on her tour as well!