Refactoring Am I Rent Stabilized
Published on December 01, 2020
Am I Rent Stabilized? continues to be a successful project; according to Google Analytics it still gets around 1,200 views and (double that for “impressions”) a month! However the last time I significantly touched the code for AIRS was over five years ago, back when I was still a fledging programer and web developer. Not surprisingly, I more recently found the code difficult to reason about, making it hard to add new features, fix bugs, or make necessary improvements such as those relating to web accessibility. I realized then, that if this project were to have a chance of living on, it would need a significant refactor. In this post I outline how I refactored AIRS; the goals and non-goals I set for myself, decisions I made along the way, and what I learned from it.
Part of what motivated me to do this refactor was reading Martin Fowler’s infamous book, Refactoring, Improving the Design of Existing Code. MF’s book not only provides a wide range of strategies for refactoring code, but also does a good job at justifying why refactoring is (or for many teams should be) a central part of software development. Refactoring the source code for AIRS offered a different set of challenges than my current day job as a UX Engineer where I tend to focus on prototyping UIs and data visualizations for the web. Though my prototyping work often acheives a high level of complexity, it is generally considered to be “throw away code”, meaning that it never sees the light of day in the actual product. The goal of prototyping is typically to validate or invalidate a hypothesis made by UX designers, UX researchers, and project managers. Thus the implementation of a prototype is not what is important; the learnings from it or the stakeholder buy in it helps generate is.
I think there’s something to be said about the importance of maintaining successful projects, or at least ones you dearly care about, versus sticking with only doing new projects on the side. I have not worked on a side project as the sole contributor in a good number of years, so I saw the appeal in doing things how I saw fit without having to spend time discussing and debating decisions with other contributors or team members. Even though side projects are still work, this can make the work a bit more enjoyable and a relief from the occasional tension and conflict inherent in team work.
Here are the goals of the refactor I decided upon:
- Make the code easier to reason about by improving its organization and structure
- use the common
src/directory approach with sub directories for related pieces, e.g.
- reduce code bloat by aiming to keep classes and functions small (say under 300 lines)
- aim for being DRY but try not to preemptively optimize for this. To me this means waiting to abstract code after it’s apparent there’s duplication, instead of trying to abstract it right away when writing it.
- use the common
- Improve code quality:
- use ES6 syntax
- use ES modules
- use StyleLint for (S)CSS linting
- use Prettier for code formatting
- add a commit hook that runs both Prettier and ESLint so that these tools run automatically
- write unit tests using Jest
- achieve good test coverage (>= 75%)
- Remove unnecessary 3rd party JS dependencies (e.g. jQuery, CartoDB.js, Leaflet, etc.)
- many of these aren’t absolutely needed for what the app does
- some can be replaced using native browser features (e.g.
fetch) or smaller 3rd party libraries (e.g. d3-geo)
- although optimization isn’t a goal, by reducing the amount of 3rd party JS the site should load faster for users on mobile and slow internet connections
- Update the frontend build system
- the Gulp file and usage of Browserify was a bit messy and not easy to reason about, but maybe that’s just because I’ve become so used to working with Webpack
- Use Webpack as a build system with Babel for transpiling ES6+ to ES5, code splitting, code minification, source maps, Sass, etc.
- Have Continuous Integration and automatic deploys integrated with Github
- Use Netlify for deploys and preview deploys for pull requests
- Set up a Github Action for a CI build on pull requests that also runs the unit tests
Port the code to TypeScript. As much as I’ve enjoyed working with TypeScript lately, it felt like doing so would be adding more complexity to the refactoring effort than was needed. TypeScript can always be added incrementally later if I chose to do so.
Solve all the accessibility problems. There are quite a few of them that I intend to fix, and when it wasn’t too much effort to fix one while refactoring I went ahead and did it. I intend to file issues for the more significant problems and fix them later after the refactor.
Optimize for performance. In MF’s Refactoring, he talks about how prematurely optimizing for performance can really hurt the readability of code and thus the ability for humans to reason with it. He suggests focusing on the refactoring first, then finding out where performance bottlenecks may be occurring and to address them when necessary.
Focus on adding new features. I don’t really have any new features to add at this point anyway, so not a problem. I did end up adding autosuggest to the address search though, so I guess that counts as a new feature!
Ultimately I decided to:
Break up the translation JSON files by page name and two letter language code to make them easier to reason about. Previously there was one JSON locale file per HTML page, and each file contained translations for all three languages. Breaking them up makes them easier to edit and compare.
Organize the code methodically and consistently following the component pattern to decouple features using ES6 classes. Each interactive element on the page received its own
Componentclass, and inherited from a parent class so that components would retain similar functionality.
Use ReduxJS for application state management. Redux is well known among the JS community, and while it is criticized for requiring a lot of boilerplate to get started, I find it to be a super useful library for managing shared application data and state. Additionally, the Redux Dev Tools are enormously helpful for seeing what’s going on during state changes when debugging.
Use Webpack as a module bundler along with the Babel, Sass, etc. Previously I was using GulpJS as a build system, but the Gulp file I had was messy and not taking advantage of modern features such as bundling ES6 modules, transpiling, and tree shaking. I originally began rewriting the Gulp file and upgrading its dependencies, however it quickly started to feel like going down a rabbit hole to get it to do what I wanted. Ironically, with all the criticism Webpack seems to get for being complicated and difficult to configure, I feel that I know it well at this point from having used it in so many other projects, so decided to make the switch.
yarnand kept out of version control.
Favor native browser APIs such as
- For example with the map on slide four that shows properties that likely have rent stabilized apartments, I decided to use
d3-tilewith the Carto Maps API to render the map. The map is essentially what amounts to a static image (it doesn’t allow for zooming, panning, click, or hover interactions) so there’s no need for a full featured web mapping library like Leaflet or MapBoxGL. In fact, when I later saw this Observable Notebook on how to make a webmap from scratch, I realized I could even get by without
- For example with the map on slide four that shows properties that likely have rent stabilized apartments, I decided to use
However, in some cases I decided to keep an existing dependency. For example, I retained the GSAP web animation library for smooth scrolling behavior. Although there’s now browser support for native smooth scrolling, it isn’t 100% supported by Safari and would require a polyfill for IE. While investigating what seemed like the recommended polyfill, it didn’t seem like it would even solve for my use case. Even with the native browser smooth scroll, you can’t control the duration of the transition or its easing property like you can with GSAP’s ScrollToPlugin. So instead of removing GSAP I upgraded it from version 1.x to 3.x. This had the additional benefit of using GSAP and the ScrollToPlugin as ES modules, so I could incorporate them into my Webpack build system and not load them from a CDN as globals like I was previously doing.
- Write unit tests
- This is a part of what MF advocates for to make the refactoring process go smoother.
- Having unit tests can help you feel more confident that you’re not breaking anything when making changes to the code and seeing that the tests pass.
- Writing tests was a new challenge to me, I typically don’t ever write tests at all!
- It definitely made the entire refactoring process feel slower as I ended up writing 2-3x as much code as I originally anticipated.
- I learned a lot though; deciding what to test and writing good tests can be fun puzzles to solve.
- One payoff of this effort is that the next time I write unit tests for a project I’ll be faster at it.
- Use Netlify
- The preview deploys that you get for free with Netlify are enormously helpful, e.g. for cross browser testing, CI/CD, code reviews, or when presenting changes to clients and team members who are not developers.
- Netlify automates building and deploying your app to production (in this case to amirentstabilized.com) when pushing to the main git branch.
- You get hosting, an SSL certificate (so your site is served over
https), logging, and other goodies.
- The free personal account plan is fairly generous, you only need to pay if you go over build minutes or want premium features such as having multiple team members on an account.
The Refactoring Process
The first step in the refactoring processed involved reviewing the existing JS code to understand how it was structured; I needed to know what each module, code block, and function was doing. Long story short, it was a mess! I had abused the ES5 revealing module pattern, name spaced parts of the code unnecessarily deeply, and used very terse variable names at times which made it difficult to understand what any one part of the code was doing and how it might relate to another. I didn’t do a good job modularizing the code either, there were bits of related logic spread throughout various modules which definitely wasn’t a good organizational practice.
Even though I was aiming to only use vanilla JS I still had bits of jQuery in a lot of places. I teased out logic that was worth keeping and could be improved while also deciding what was worth throwing away. I mainly focused on the JS, deciding not to refactor the styles which use Sass, but I did make small changes or improvements to the Sass code when I felt it was necessary or not too big of a lift.
I began the written refactoring by first creating two entry points for Webpack, one for the main page / app, and another for the three
info/*.html pages. This enabled code splitting (for both JS and CSS), which helped keep the bundles generated by Webpack smaller. The main (
index.html) page is where the actual “app” is, the other pages are just static content, so they don’t need to load all of the same code. A user may have just bookmarked the
resources.html page for example, so when it loads we only serve JS and CSS needed for that page.
I then focused on refactoring the translation & Handlebars template loading logic. This is an important part of code to make crystal clear, as it is essentially how the entire site renders on page load and when a user toggles the language of the page.
The process is roughly:
- Check for a language code in the browser’s
localStoragethat may have been previously saved. If nothing is found then default to English.
- Check for the HTML page’s base name (e.g. “index”, “why”, “how”, “resources”).
- Load the correct Handlebars template file based on the page name.
- Load the correct locale JSON file based on the page name and language.
- Use a dynamic
importto grab the correct initialization script (for the
- Render the page using the Handlebars template and locale JSON.
- Invoke the initialization script to set up any interactivity.
To tackle the interactive elements, I created a base
Here’s what that class looks like:
And here is an example of a component that inherits from that base class:
For each component class I wrote a series of tests. Martin Fowler advocates for tests to make your code more resilient when refactoring it. One recommendation MF advocates for when writing tests that I found to be fun is to deliberately break the tests to make sure they work as expected. Or to see what happens when you deliberately use your code in unexpected ways, and to test for that. Writing tests can be a bit like solving puzzles; it’s not always apparent what to test for and how to test it. Getting to know Jest and friends (e.g. JS DOM) was also a bit of a learning curve. In the end I’m glad I put in the effort, as I know when I write more tests in the future that it won’t feel so foreign or difficult.
Here’s an example test for the above
AdvanceSlides component that checks that the component’s click handler is called when the “button” UI element is clicked on. (I know, this should be a real
<button> not an
<h3>, a pertinent example of why more folks should be taught about accessibility when they are learning web development!)
I mentioned earlier that I chose to use ReduxJS for managing the app’s state. To be honest, I have not used Redux outside of a ReactJS project before, and this proved to be interesting! Typically when using Redux with React you would use the
Provider context and
connect helper function to make a Component aware of Redux. Being that I’m not using React, I utilized an
observeStore function, which is how components I wrote respond to changes in state:
It takes three arguments: the Redux store singleton, a function that returns the piece of state that should be watched for changes, and a callback function (
onChange). It returns an
unsubscribe function, and recall that the base component class I created has an
unsubscribe method. When a component uses this
observeStore function, that method is overridden with the function returned by
observeStore. This is important, because when the app re-renders as a result of a language toggle, the entire DOM is essentially blown away and any previous component instances need to be cleaned up. That clean up work involves unsubscribing from the Redux store.
For the remainder of the refactoring effort I added Redux boilerplate as needed, e.g. the action types, action creators, reducers, middleware, etc. Redux accomplished somewhat simple tasks such tracking the active slide’s index, as well as more complex and asynchronous tasks such as fetching data from an address geocoding API (Shout out to NYC Planning Labs for the terrific API!). Other “slices” of state include whether or not someone’s address is likely to have rent stabilized apartments, and whether or not the searched address is within a catchment area of a local tenants rights group. These types of data are shared between components and thus benefit from being stored in shared application state.
I particularly like using the Redux Dev Tools for debugging the Redux state, which I find to be a much better UX then
console.log‘ing things. The Redux Dev Tools are something you would also have to create yourself if using a custom made pub/sub routine, another solution I considered for managing application state but ultimately decided against using. Lastly, Redux is fairly simple to write tests for, with the exception of asynchronous action creators which can be a little more tricky to test.
That sums up the majority of the refactoring work. Other bits included writing utility helper functions and constants (other than the Redux action types) that get shared between multiple modules. Please feel free to take a look at the app’s source code to learn more or tell me what you think!
The refactor of AIRS ended up being over four hundred commits over a period of three months! Doing this in my free time was not easy, but I found that I could chip away at it here and there to slowly make progress. There were definitely some big pushes at times and at some point I had to decide when to call it “good enough” and merge the changes. I didn’t get around to everything that I had wanted to do in the refactor, however I am now able to tackle work incrementally more easily than I was able to before. Finally, I decided on adding a Changelog file to track significant changes to the code and design of the website. I should probably add a Code of Conduct as well for contributors, as well as a License file.
Oh the things you learn as time goes on! Looking back at my original code helped me reflect on how much I’ve grown as a programmer and web developer, how much the world of browsers and web development has changed, and how my outlook on building websites and UIs has evolved. One example is that the concept of “componentizing” the UI was just beginning to take off around 2013 to 2015, and now in 2020 it’s so entrenched in how us developers build UIs it almost seems inconceivable to do otherwise. Browsers have of course changed in the past five years as well; new APIs are being unveiled while more features become standardized across browsers (well, sort of). Webpack was still fairly new in 2015 and not widely used as a frontend build tool; Grunt and Gulp were still the popular choices back then. While I’ve always valued User Experience above all, I’ve come to appreciate it even more from my current job as a UX Engineer at Google. This has motivated me to fix the accessibility issues with AIRS; for example improper uses of headings or
<div> elements that function like as buttons but without the proper
ARIA attributes and keyboard event handling.
Here are some bits of quantitative information related to the code before and after the refactor.
- 1MB total JS
- 48 kB for the
- 416 kB total JS
- 206 kB for the
- ~50 kB total for multiple source bundles
*In the original code some 3rd party dependencies were included in the bundle while others, such as jQuery, were loaded over CDNs.
vendors.js bundle that can be cached by the browser. This is beneficial, for typically the source code changes more frequently than 3rd party dependencies, and with cache busting file names thanks to Webpack, source code bundles will be received by the browser when they are updated. A few remaining scripts are still loaded via CDNs: the “Add To Calendar” widget, Google Analytics, and the “Add This” social media widget. While this is not ideal, I’m willing to live with it, and may end up using alternatives to some of these in the future anyway.
Amount of Source Code
The amounts listed below are total lines of code:
- JS: 1,880
- SCSS / Sass: 3,240
- Handlebars templates: 848
- JS, total: 5,839
- JS, excluding unit tests: 2,122
- SCSS / Sass: 3,115
- Handlebars templates: 867
Results from the Lighthouse auditing tool in the Google Chrome browser.
- 96 Performance
- 72 Accessibility
- 71 Best Practices
- 80 SEO
- 0.7s FCP
- 0.9s TTI
- 0s TBT
- 1.3s LCP
- 96 Performance
- 81 Accessibility
- 79 Best Practices
- 80 SEO
- 0.7s FCP
- 0.9s TTI
- 0s TBT
- 1.3s LCP
(FCP: First Contentful Paint, TTI: Time to Interactive, TBT: Total Blocking Time, LCP: Largest Contentful Paint)
It shouldn’t be surprising that the results from before and after the refactor do not differ by that much, as I did not focus on making changes that would drastically affect things such as SEO, accessibility, Time to Interative, etc. I did not aim to improve performance other than reducing the amount of JS sent over the wire, which seems to have not affected the performance score. It’s worth noting that the Lighthouse scores are estimates, so re-running the tool may give slightly different results each time. The Lighthouse audits were run on my Mac Mini, circa 2018 with 32 GB of RAM. The scores tended to differ drastically from one run to the next when I ran them on an older laptop.
Possible Next Steps
Here is a short list of tasks that I would like to tackle next, now that the refactor is complete:
- Add integration or end to end tests
- Fix accessibility issues
- Refactor the SCSS
- Integrate a backend service to remove the need for a CARTO account
- Instead of Handlebars, use plain old HTML and a locale JS library such as
Phew, that was a lot! If you made it this far, then thanks for reading. Hopefully this post will motivate you to do some refactoring of your own if it’s something you haven’t tried out yet. And lastly, don’t forget to read Martin Fowler’s book Refactoring!
If you found this website to be helpful please consider showing your gratitude by buying me a coffee. Thanks!