Ever since software is being developed, one of the greatest challenges is how to build exactly what the customer needs. How do you stay close to your customer in the face of forces that drive you away from him? Many methods and programming languages have promised a solution to this problem.
Take COBOL, one of the oldest programming languages. COBOL means COmmon Business Oriented Languages. The idea was that COBOL was business oriented enough so that business people could write their own programs with it. This never worked of course. COBOL is widely regarded as a difficult programming language. Good COBOL programmers are hard to find and they are badly needed to maintain the still enormous body of COBOL programs.
SQL: same story. In SQL you tell the computer, or more specifically the database, what you want with the data instead of how to do it. Should be easy enough for business people? Nowadays SQL is considered too difficult even for the average programmer. We have Object-Relational Mappers to shield programmers from the "complexity" of SQL.
Next candidate: UML. We draw diagrams, pictures that business people can understand or even draw themselves. We have a code generator that translates the pictures to programs, and the user is in control again. This approach, however recent, is already out of favour again. See my earlier post "What's new about Model-Driven Development". The use of DSL's instead of UML does help a little, because a DSL is simpler. But the expectation of some, that the domain specific nature of a DSL means that a domain expert can write programs with it, can safely be regarded as naïve. If only because the domain of a DSL is almost never the domain of the user, but that of the developer.
OK, so new programming languages or tools don't help to bridge the distance between user and programmer. Let's try something completely different. Let's have architects help the business user to describe his business processes, and align the applications to support those. And let's introduce a Service-Oriented Architecture (SOA), so the applications can communicate easily via services. And we need a process engine to directly support the business processes by calling the services in the correct order. "A business user can do this himself."
Sounds great! Only I don't believe a word of it. Firstly, SOA experts keep telling us that a SOA is difficult to implement, and that the potential benefits - easy adaptability of the software to the ever changing requirements of the business - will be only reaped after many years. That is, if the SOA is introduced properly and embedded in the organisation. Governance is the magic word here. So what do you mean: "quick adaptation to changing business requirements"? Secondly, the word governance with me evokes the idea of "programming by committee". And that should be faster than one or two programmers sitting together? Thirdly, now we have architects between the customer and the programmer, making the distance between them greater again. Not to say anything bad about the good ones (I know one or two), but many architects have little affinity with the practical problems ("irrelevant details") facing programmers trying to implement their SOA. I always hear architects complain about programmers not keeping to their guiding principles. Why would that be? And does it sound strange that those organisations that have a separate architecture department are the same whose development process has all but come to an almost complete standstill?
Is there no hope then? Yes, there is. Those projects that adopt a radical agile process, be it eXtreme Programming, Scrum, Evo or whatever, are the same that see a radical improvement in productivity, while actually building what the customer wants. In addition to the fact that the stories sound convincing, I have experienced this myself. That experience was so good that since then I don't want, and maybe even can't, use any other process.
Even though the results are good, there is still much resistance against agile methods. One objection raised sometimes is, that large systems cannot be built without architecture, and that agile methods lead to chaotic and unmanageable systems. The first is true: large systems still need architecture, but the second is not: agile does not necessarily lead to unmanageable chaos. Agile is no substitute for good craftmanship. To create excellent applications in an agile way you need excellent programmers. Only the way architecture is created is completely different. Architecture is a byproduct of software development. It emerges at the same time, not before. Only then you may hope to have an architecture that fits the application like a tailor-made suit.
Applications are so complex nowadays, that they can't be built without architecture. The same complexity is the reason why the architecture can't be thought out completely in advance. If you try you will end up with a rigid system that is hard to change, exactly what that same architecture is trying to avoid. It doesn't have to change, it is thought out in advance. No. Architecture is needed, but can only be developed via an agile process.