How we learned to stop fearing technical debt and love frameworks

The war room from the movie "Dr Strangelove"

Frameworks (sometimes called platforms or wireframes) are great for software development. They allow programmers to skip over the standard functions and procedures, and jump right into the meat of development. Frameworks like Laravel and Symfony both come with in-built functions that allow developers to pull items from a database in a single line of code. This way they don’t have to spend a huge amount of time reinventing the wheel.

Framework inequality

Sadly not all frameworks are made equal, and when choosing one to work with it’s important to think about how long it is going to last. The most popular HTML, CSS, and PHP frameworks are open-source, meaning that they’re developed and maintained by the community. This is great for splitting the load of development between many people, and making sure they everyone has a say in what should be in the framework (as opposed to proprietary software).

However having a framework that is built entirely by the community means that there isn’t as much security in its foundation. If one or two of the lead developers within the community fall out of sync, and no one steps up to take their place, then the dev network that upheld the framework can quickly turn into a ghost town. Slack channels become empty, git commits dry up, and the support that came with the community ceases to exist. This is why, when choosing a framework to build with, it is incredibly important to weigh up the age of the framework with the size of the community. Community size trumps age, but consistency trumps size.

Most developers have heard the story of a company which took on a project using a platform that unfortunately fell into disarray. With the git commits slowing down and the community falling quiet the company was left to make a difficult choice: rebuild their entire code in a better supported platform, or soldier on with what they had. Moving onto a new platform can be a scary task, and incredibly expensive. However, staying with an old and less supported platform can be just a difficult and even more costly. This is where we need to discuss something called “Technical Debt”.

Technical debt

The idea behind technical debt is that often code that is implemented quickly and easily is often not the best solution in the long run, and has to be refactored or completely rewritten later on in the development cycle.

Similar to financial debt, technical debt incurs interest in the rest of your project. Because of a quick and dirty fix that was implemented to squash a bug, developers now have to spend time working around that design choice to get other things to work. The problem can either be swept under the rug (which tends to lead to further technical debt down the road), or we can invest in fixing the problem now, spending time to make sure we have fewer issues later on in the project.

Sticking with a dying framework is increasing the technical debt. It may be easier and cheaper to stay with the solution that you have now, but further into the project you will hit a bump in the road that forces you to make a major and potentially drastic change.

Playing to your strengths

In the 4th Harry Potter Book, Harry Potter and the Goblet of Fire, there’s a chapter where Harry is tasked with retrieving a golden egg from a dragon. After days and weeks of wracking his brain to think of a way to do this, he is advised by one of his teachers to play to his strengths and use a flying broom stick. It just so happened that Harry was incredibly good at flying, and was able to overcome the challenge.

The same story can be adapted for software development teams. Every project has an end goal (a golden egg), a bunch of issues in the way (a dragon), and a particular set of skills they can utilise (flying on a broom stick). This is something else to keep in mind when selecting a framework.

If your team has three developers fluent in Python, and only one adept in PHP then it makes sense to select a framework that plays off the majority of their strengths. That may sound like an obvious point, but it’s surprising how often teams can get overwhelmed and excited about the prospect of using a new technology or the latest javascript framework simply because it’s new. Sure, learning and exploring new ideas is important, but they shouldn’t replace solid foundations and best practices without proper evaluation.

Saying that, if you haven’t select your team yet, or are still in the process of building it, then you have a fantastic opportunity to build your team around the framework.

C to D

Choosing how to develop our integrated data access model for census aggregate, flow and boundary data held by the UK Data Service was an interesting and difficult decision. Our aim is to make the data from our legacy systems available in ways which will improve its discoverability and usability, cost effectively and aligned with the lastest standards for frictionless data. We had two possible paths that we could take: we could build the application from the ground up, or use an existing framework. They both had their pros and cons. Spoiler alert, we chose to use a framework.

In order to build our application from the ground up we would have had to design everything. From the database structure, to the MVC (model-view-controller) setup, every single line of code would need to be thought out, planned, and rigorously tested. Obviously this would be very time consuming and expensive. However the level of control we could have over the application would be immense. Since we would be designing everything ourselves, we could fit each function to do exactly what we wanted, when we wanted, and prioritise certain methods over others in order to suit functional needs. But the amount of time and resource available to us did not allow this approach.

So we chose to go down the framework route. Next up was deciding which one to use. The two that stuck out most to the team were CKAN and DKAN.

CKAN, the older of the two, was designed to be “the world’s leading Open Source data portal platform”. It manages to stand up to it’s aspirations very well, with services like Berlin Open Data and The Humanitarian Data Exchange and of course data.gov.uk using the CKAN platform. CKAN itself is written in Python and has been around for about 10 years. Unfortunately the community behind the framework doesn’t seem as active, and can be difficult to get in touch with. Without a Slack channel, the developers rely on mailing lists to keep in touch.

DKAN is a spin-off of CKAN, written upon the Drupal PHP framework, and is designed to perform in the same way as CKAN while being built in PHP. It’s important to know that Drupal is huge. It is the 3rd most used content management system in the world, behind WordPress and Joomla. However the strongest pro for DKAN is it’s community. The GitHub repository (where all the DKAN code is stored) is constantly being updated and maintained by a passionate group of developers who seem to genuinely care about open data access. There is also a public Slack channel where developers, maintainers, and pretty much anyone can join and chat about all things DKAN. The data sets in DKAN can be stored either within DKAN or on external sites, and it is possible to manage access control and version history with rollback. DKAN provides user analytics and data users can upload, tag, search and group data sets via a web front-end or APIs. In addition, they can also collaborate, comment, and share information via social network integration. The current deployment of DKAN instances include Open Puerto Rico portal and the city of Cologne portal in Germany. A few of the community members recently attended DrupalCon in Baltimore to meet up and discuss the current state of DKAN. Find out more about open data software evaluation at Open Data Monitor.

Testing the water

We spent a few weeks evaluating both frameworks, trying to decide which solution to pick. After running into problems with CKAN we jumped into the community to ask questions and try to find some assistance. Unfortunately the unavailability of the CKAN community made the task difficult. Similarly we faced some issues with DKAN, however the accessibility of the DKAN community meant that our issues were fixed in a matter of minutes!

Due to the ease of access we had to the DKAN developers, and the friendliness of the community there, we decided to pick up the DKAN framework and build our application on to that. Since then the team has been active within the DKAN community, contributing code to the GitHub repository and assisting other users in anyway we can.

From here on out

Building an application can be difficult, time consuming, and stressful. However having a framework to stand on and a great community to lean on can make all the difference. The team here will be blogging frequently about our experiences with DKAN in development and will be testing out user needs by posting here as we go.


Image credit: Still taken from the motion picture “Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb” directed by Stanley Kubrick with Hawk Films. Distributed by Columbia Pictures 29th January 1964

Leave a Reply

Your email address will not be published. Required fields are marked *