Is rewritting really bad?

Paul Chiusano tell us his Scala success story, and while I find it interesting, there's something that attracts my attention in it. He says:

Finally, I'd like to scrutinize the folk wisdom that rewrites are generally a bad idea.

That got me thinking a bit. That wisdom predates some developments which only became widespread practices recently. In particular, BDD and Continuous Integration.

It occurs to me that if I decide to rewrite something from scratch, I'll have a large set of tests in the form of BDD (which tests for Behavior, so a lot of it should still apply), plus the tests for all bugs that were caught and fixed, for which you them add tests to help with CI.

Furthermore, because you are doing BDD and CI, and possibly TDD for low level stuff, you won't get far with the same errors in case you re-introduce them.

That's something to think about... maybe the old wisdom requires some revisiting.

At any rate, I'd like to point out that this particular piece of wisdom might have become widespread because of this article by Joel Spolsky. I'll quote one part of it, though:

These problems can be solved, one at a time, by carefully moving code, refactoring, changing interfaces. They can be done by one programmer working carefully and checking in his changes all at once, so that nobody else is disrupted. Even fairly major architectural changes can be done without throwing away the code.

Well, that's precisely what Paul Chiusano did! So, on the other hand, perhaps is too soon to dispose of that wisdom...

A Mind is a Terrible Thing to Waste

I have just seen a presentation about Agilists and Architects, which resonated with an old thought of mine: the waste of experience in the software development process.

Writing software – programming – is something that takes a huge amount of experience to get right. You have to know the language, of course. You have to know it well enough to avoid getting things wrong because of an unknown feature or characteristic of the language. You have to know the language libraries, frameworks, and development tools. This is the very basic.

You have to know patterns, and you have to know anti-patterns too. You need experience to realize that a pattern apply to a problem, or that some solution is inching toward an anti-pattern.

You have to know about security, about protocols, about distributed systems, about parallel systems, about algorithms – of so many types I can’t even list them here. You have to know about databases, persistence, XML, MVC, DSL, regex, grammars, state machines… we could go on and on. You have to know usability.

You have to know about programming paradigms. That’s easy, there’s only four or five of them. Or maybe more, depending on who’s definition you are using. And, of course, each paradigm has many variations preferred by one language or another – and who has to deal with a single language these days?

You have to know computers, and experience here will bring you understanding of performance issues. Experience will also let you identify bugs, or avoid them altogether.

Then, of course, you have to know CMMI, PMBoK, FPA or any of dozens standards and best practices and processes for software development, quality, testing, team working, etc.

Finally, of course, you have to know how to communicate with other people, understand the business needs, know the existing systems, the explicit and implicit rules, and, if possible, why it exists in the state it is, why certain decisions were taken, what were the conditions that might change and make it interesting – or even necessary – to take another path, or to avoid certain paths.

Each of those things you know or don’t have a direct impact on the quality of the code you write. Sure, you can write software with only a small glimpse of all this knowledge, but many times you would have written better, more efficient and more maintainable code if you only knew there was a word to what you were doing. Whole classes might have crumbled to a single library call. You might have avoided code which is intrinsically insecure, protocols which are unfixable. Hell, you might have avoided trying to solve problems known to be unsolvable.

While deep knowledge is clearly the domain of experts in a field, all those domains come together through the hands of the generalist, who has to know something exists before he can use it.

So there are architects, who are meant to have all that experience, and lay down the trail for those inexperienced to follow. Only, of course, it doesn’t work that way. The sheer level of detail required in developing software preclude architects from being able to know or influence but a thin, though important, layer of all there is.

It couldn’t be other way, in truth – if it were possible to write a precise specification of what the software should do, what should be its structure, how it should go about doing what it is supposed to do, well, if it were possible to write such a specification, then we could feed it to the computer and be done with it. In fact, we can write those specifications – they are called “source code”. Source code is not what a computer executes. It’s a specification detailing how to produce the instructions the computer will follow to provide the service required.

In short, or, rather, in conclusion, programming experience results in code with fewer bugs, that satisfy more completely the users needs, easier to adapt to new needs, and in a shorter time.

So why the hell does any company even contemplates hiring “junior” programmers? No, let me rephrase that. Why is the average experience of programmers working – coding – for corporations so low? There is an obvious – to me – inversion of value going on there. And looking at companies that make software to be sold as service or at retail – as opposed to producing software to someone else’s specs – I don’t see much disagreement.

I can see a few reasons why that might be true, of course. First, just because someone acquires experience doesn’t mean he or she learns new stuff. And the transition from mainframe to personal computing pretty much proved that can be detrimental – when people cling to what they know against the unknown, instead of avoiding what they know to be wrong, or taking the better one of known alternatives.

So, corporations “solve” the problem by making experienced people into architects and managers. Being a manager has a quality of its own, and having experience writing software does not really add anything to it. Being an architect takes you too far from where you can make most use of your experience.

Hopefully, if you are lucky enough to work somewhere software architects have meaningful influence on the software development process, you might get programmers to avoid huge mistakes. And if your corporation is really lucky, you’ll do so without critically impairing the software development process with decisions detached from the reality of the problems being faced.

I was happy, then, with the ideas presented by Rebecca Parsons and Martin Fowler. It pains me to see so much experience and knowledge wasted because of misguided – or misinformed – processes.