Let’s pretend for a minute that you aren’t a software developer, and that you are a doctor. A patient comes in with the following symptoms:
- Trouble walking
- Redness and swelling around a wound on the leg
- Give the patient a band-aid, pain killer, a walking cane, and a cold bath?
- Investigate and clean the wound, administer antibiotics, and recommend rest?
Sometimes in software development, we see things just like this. We see the symptoms of a deeper underlying issue, but we get fixated on solving or fixing the symptoms rather than the underlying cause. Let’s take a more software related example.
A customer occasionally sees a database error. Do you:
- Wrap everything in a try/except and display a nicer error message (or hide it altogether. It’s not like the customer can do anything about it, right)?
- Investigate the command that was causing the database error, and fix the underlying problem that allows the error to happen in the first place?
It seems totally obvious what the better solution is in this case, but so often things are patched with duct tape, chewing gum, bailing wire, and band-aids. After a while, it becomes a disgusting mess that no one wants to deal with it. At that point (or well before that point!) developers start crying for a re-write or a refactor. A good case of “refactoritis” can be a sign that things haven’t been properly fixed along the way. Maybe a rewrite or refactoring is necessary. Maybe it isn’t. But either way, the temporary solutions need to stop, and underlying issues need proper investigation.
You may not immediately agree with me, but hear me out. Some information really is worthless.
I had the “pleasure” of being involved with a project that loaded data from a CSV, and imported that data into a second system. The purpose of the second system was quite different, and it really only needed maybe three or four fields from the CSV (which had a lot more columns than that!). When the code was written to import the CSV, it imported everything, whether it was going to be used or not. I get why it was done (the developer didn’t know any better), but by doing so, the developer introduced more code, more data, and more maintenance cost.
I get that the value of recorded information often isn’t readily visible (especially without any sort of visualization of that data), but there are cases where information really is useless.
It’s a bit like going up and asking someone where the nearest toilet is, and they proceed to tell you not only where it is, but the entire history of indoor plumbing. For the specific use case that you are wanting, knowing about London’s historic sewage pumping station isn’t going to help you do your business any faster.
In chatting with a friend, this thought popped into my mind:
What value is there in answering questions? How does it empower the person with the answer?
It might seem a little silly, but here’s a few random questions that have “powerful” answers (in the business sense):
- How can I automate this process?
- How much is the grain in my grain bin worth?
- How much does it cost me to use Product X, and is it worth it to me?
- Where are all my assets located?
- Do I need to keep specific inventory levels?
- Which products are my best sellers, and what is the impact of dropping my lower selling products?
- What is my single biggest expense, and how can I reduce it?
- Where is the majority of my time spent?
By helping people answer questions, empower them. And, as software developers, by having the ability to put together solutions that can answer questions, we, in essence, become powerful.
Everyone has questions, but not everyone has the means of answering those questions. Some answers take more work than others, and some answers carry more value than others (e.g.: “How can I keep in communication with family members in an emergency?” vs. “Where did I put that hammer I was just using?”). Either way, there is something powerful about being able to answer questions.
I’ve seen this one happen in a few different places:
A web service that returns an HTTP status code of 200 (OK), but the message inside the response says otherwise.
HTTP status codes are there for a reason! If the response isn’t okay, then don’t say that is is okay! It’s like going into a doctor’s office and saying “Yeah, I’m feeling pretty good” but your arm is gushing blood because you accidentally cut it off on a table saw.
If the response isn’t okay, then use an appropriate HTTP status code to give an indication what the problem is. Otherwise users might try to work with your response, which no longer contains the data that they are expecting.
Recently, I’ve had the pleasure of dealing with some optimization work. I find it oddly satisfying, as it’s very quantifiable. It’s also nice to hear a happy story from an end-user, saying that something was slow, and now it’s fast.
Sometimes optimizing the user experience is a matter of actually faking the appearance of improved performance. If a user feels that it is faster, that’s really what counts. Some times this comes down to caching data, adding loading indicators, and making sure the main UI thread is still being responsive. Depending on the case, that might be all that’s necessary to give a decent experience.
Other times, it’s actually digging in, and figuring out what is slow, and fixing it. Often that is much easier said than done. Performance bottlenecks can be almost anything from memory thrashing or cache misses to poorly designed algorithms to slow 3rd party libraries. To fix some of these issues it can take switching to multi-threading, reorganizing data structures, revising algorithms, or switching libraries. Changes like that often aren’t cheap. But once the underlying issue is resolved, it still fixes the user-perceived slowness.
I prefer to find and fix the underlying problem rather than applying a band-aid. There are times when band-aids are appropriate, but more often than not, it’s better to dig in and fix the real problem. No quantity of band-aids will fix a bleeding femoral artery. Likewise, no quantity of band-aids will fix a poorly written web service or a poorly thought out algorithm.
’tis better to have loved and lost than never to have loved at all.
– Lord Tennyson
Or in programming terms:
’tis better to prevent bad data from entering the system than to have to clean it up after.
Allow me to share a recent example.
At my job, I work on a team that interacts with a lot of services. Some of these services are under our control, and others aren’t. One, in particular, is a legacy service that we are working to replace. This particular service contains a set of data about machines. We currently have a script within our main system that calls this other system, fetches data from it, and attempts to insert it into the main database.
One of the problems with the old system is that there’s a lot of user-entered data that isn’t clean. For example, let’s say there’s a database field for the manufacturer of the machines. Users could have entered values such as:
Chevorlet <- note the bad spelling
When doing string comparisons*, to a computer, all of these values are different. So as a result, when we pull data into the new system, we end up pulling in this same bad data.
Thankfully the revisions we are making to the main system will make this kind of user-entered data a lot more difficult to enter, but unfortunately we aren’t there yet. As a result, we now have bad data in the main system. What really should have been done in a case like this, is that any data coming into the main system should have been cleaned prior to being entered. The new system will eventually force a user to select the manufacturer from a drop-down list (not enter it in a text field). This fixes things going forward, but still doesn’t clean up the older bad data.
Like I said, ’tis easier to prevent bad data…
*Avoid doing string comparisons as much as possible – especially on user-entered data! You are just asking for pain if your code contains a lot of string comparisons.
Let’s set the scenario: you are sitting on a ton of data, and the owners of that data want to generate a report based on it. Running the report takes a while (say, several minutes). Do you:
1. Pre-calculate the report at a given interval (e.g.: nightly), and store the pre-calculated results so that they can be displayed quickly?
2. Make the user wait several minutes while the report is generated?
In cases I’ve seen like this, it’s a bit of a mixed bag. No one likes waiting several minutes for a report to run (especially impatient users), but from a development standpoint pre-calculating and storing results has its own set of drawbacks. What happens if we have new data being entered into the system each hour, and users want an up-to-date report? What happens if the nightly process that generates the report fails?
I don’t think there’s always a cut-and-dry answer here. But lately my experience has me leaning toward option #2. Yes, it’s an inconvenience to the user, but it may save a lot of development effort later on. It’s a bit like how fixing a bug during development can save 10x the amount of time later on if that bug eventually makes its way into production. Rather than spending that development time dealing with caching (which isn’t always simple as it seems*), it might be better to spend that time optimizing the system, and organizing your data in a better way to increase performance. I’m not really a fan of duplicated data, especially when there’s a potentially better option.
*There’s two difficult problems in computer science:
1. Naming things.
2. Cache invalidation.