Tuesday, February 15, 2011

Between The Cracks

All of computer development is centered around building systems, not the glue that holds different systems together. This is like building a wall of bricks without using mortar. Such a wall might be strong, depending on how heavy the bricks are, but it's never as strong as bricks with mortar. Yet whenever these systems fail, people tend to say, “I guess we need heavier bricks.” Something else is needed.

Many times I've seen users print data from another system and then enter it manually in their own system. The excuse given is that human eyes need to review the data before it's entered. But if the data is that important, wouldn't you want to automate the review process as much as possible? It's not hard to create a queue that would pull the data from the other system and then allow users to review it before pushing it into the new system. Then you could automate away the tedious parts of data entry, making it more likely that the data is correct. The interface between systems is typically where data integrity has to be addressed, but usually this part of the system is an afterthought. Think of it this way: if all the interfaces into and out of a system are manual, have you really gained anything by automating?

The interface between user systems is important, but it's not the only crack that gets neglected. The interface between the user system and the programmers working on it is important too. Installation and update methods are usually tacked on at the end of a project instead of being thought about from the beginning. Many times updates are in the form of a file that's manually copied to the system. But manually copying a file is an error prone task – people copy the wrong file, put it in the wrong place, or forget to delete the original file. This isn't even mentioning the mess created when trying to back out a change that didn't work. Yet many insist this manual work is needed to keep control of what's going on, even though this sort of control is exactly what computers were invented for. The exactness and consistency of computers is what can guarantee data integrity, and programs are a developer's data. Developers should be as concerned with the integrity of their software as users are with their data. Wouldn't it make more sense to have some sort of license server that grabs the right updates or at least can tell you when you try to install the wrong one? Instead this part of the system is neglected, leaving a gap that can bring down an otherwise solid system.

I've saved the worst for last. Although there are many integrated user systems and there are some programs that are able to update themselves, maintenance programming is still very primitive. Database audit information, telling who did what to which and when, is all stored in database tables where they can be searched in any way you can imagine. But log files, which are basically program audit information, are still stored in text files where they need to be read line by line. And this is if the original programmer remembered to add logging. Many times programs fail silently, leaving no clue why they gave up the ghost. Other times they display a cryptic error message to the user, which the user will have to memorize if the maintenance programmer is going to have any hope of locating what part of the program failed. In general dealing with program failure is a tediously manual task, which is why it's usually given to the least senior programmers. Yet there continues to be a lack of thought, from language designers down to everyday programmers, on how to improve this.

There are other systems that a program is plugged into, like configuration, documentation and testing. The interface with these things should also be as automated as possible. Filling in the gaps between one system and the systems it interacts with, whether these are other user systems or programmer systems removes a lot of the tedious and error-prone busy work. But it also makes a system more robust, because all the paths into and out of it have been automated.