How we worked on Technical Debt to ship features faster than ever!
Technical debt best epitomizes the exuberance of youth. We take deliberate decisions in the interest of speed and bravado. And as we get older and hopefully wiser, we look back and think about how dumb we were as a kid. 🤷♂️
Varun Khona, CEO, Headout
But you can’t be old and wise without being young and dumb. Both are necessary and inevitable parts of the same story.
But as the organization grows, technical debt makes a big hit on developer productivity.
For a developer, an easy-to-use and fast feedback setup is the goal. The faster one can see the output of their code, the more they can experiment and release software faster.
In the frontend world, things like Hot-reload are prevalent. You change your code in the IDE, and it instantaneously reflects in the browser.
But the story is quite different when it comes to the backend. Most of the backend languages are static. And hence swapping is not as straightforward in Javascript. Even in a massive and sophisticated ecosystem like Java, very few solutions address this issue.
Similarly, many more issues were making the dev process slow. Let’s see how at Headout, we fixed all of them in a span of 180 days.
Pain #1: Slow Feedback Loop
💡 In a typical backend development cycle, to see code changes:Change Code → Build (5 - 60s)→ Restart Server (60s+)→ Test
So much time to see the change for even one line of code. There had to be something better. We then started to look for some tools which could help us out.
Solution: Use HotSwapAgent
After a lot of searching around, we landed at two tools
- Spring Boot Devtools is the most widely used tool to Hotswap, but it restarts the tomcat server on rebuilding. While that serves the purpose of a small project, it restarts the whole server for large codebases like ours.
- But then we finally found Hotswapagent. Hotswapagent swaps the compiled class files, so you won’t have to restart the server.
We were finally able to make Hotswapagent work with our setup. Check out the video below explaining our Hotswap process:
Impact: 12x faster feedback loop!
At Headout, now the dev cycle is: Change → Build (5 - 60s) → Hotswap (1-2s!!) → Test 🔥 This change has improved our feedback loop on an average case from 120 seconds to 10 seconds. A massive 12x decrease!
Pain #2: Slow CI Times
At Headout, we have three leading apps in the backend. Calipso, Aries, and Oberon (yes, they are named after Greek gods.)
We use Elastic Beanstalk to deploy our applications and Github CI to make the Docker builds. Our docker file was something straightforward. It would create war and start it inside of a Tomcat image.
This was all good, but the CI used to take around 10 minutes. While we were decreasing the build time, we realized that Docker builds also took quite a chunk of the CI. That was the point we came across Jib.
Solution: Use Jib to build Docker Images
💡 Jib builds optimized Docker and OCI images for your Java applications without a Docker daemon - and without deep mastery of Docker best practices. It is available as plugins for Maven and Gradle and as a Java library.
The best thing is that the docker images built with Jib start in an exploded state, which gives the application the same speed as if it were running locally.
Impact: 3x faster Github CI!
The best thing is that Jib can build images without any docker daemon. So we can skip maintaining Dockerfile and installing Docker in our CI altogether!
🔥 This brought down our CI time to 3 mins, 3 fold decrease from before.
Pain #3: Keeping API doc up to date
The team at Headout has been snowballing, and as teams grow, there is more and more need for well-maintained documentation.
Solution: Use Swagger UI and OpenAPI V3
And what better way to have API docs than to have a Swagger UI. Swagger UI allows us to have:
- Synchronizes the API documentation with the server and client at the same pace.
- Allows us to generate REST API documentation and interact with the REST API. The interaction with the REST API using the Swagger UI Framework gives clear insight into how the API responds to parameters.
- Responds in the format of JSON and XML.
- Implementations are available for various technologies, such as Scala, Java, and HTML5.
👉 And thanks to Springdoc-OpenAPI and our shift from dedicated to embedded Tomcat, this was possible with just one line of code!
Pain #4: Dedicated tomcat server
We used to have a dedicated tomcat server and used to start spring inside of it. While this method works, it is now deprecated and isn’t used anymore.
Additionally, this method required a lot of XML config and jars to run, which means a lot of maintenance costs.
The worst was that we weren’t able to leverage the excellent integrations Intellij has with Spring Boot. It was time to get rid of this legacy setup.
Solution: Don’t deploy your application in Tomcat; deploy Tomcat in your application.
Here are a few screenshots of what all we can do now with Intellij and Spring integrations
(Click here to view these pictures larger)
#Pain 5: Too much config to maintain (20+ config files removed)
Over the last eight years, the way we manage configs in spring boot has changed. And since we shifted to Kotlin, it has changed even further. This called for a config refactor as well.
Solution: Use Kotlin DSL and remove redundant config
- We had 12 config files for Newrelic with 99% duplicate configuration, reduced it to just one file
- We had six config files for
logback.xml,reduced it to 3 - All the hibernate XML configs are not needed anymore
- Shifted all context XML files to Kotlin DSL
With all of these things in place, our backend is much better for Headout to ship projects faster than ever. But removing so much technical debt was no easy challenge, and quantifying the time it will take was even harder.
How much effort did this project take?
This was at no level a small project. It involved a vast refactor and a complete understanding of the backend. Anytime I had problems understanding the backend architecture or some places I couldn’t move ahead, the whole team was ready to help.
The will to help and efficiency of our Headout team is the reason I could finish this project in 180 days. Kudos to our engineering team!
Just like our users have the time of their life on our experiences, here's to our engineering team having the time of their life building this rocket ship. 🚀
