Ever had the feeling that you are an Information Archaeologist?

How many times do you have to find out what the functionality of a systems is and what the data means? A lot of times you come to the conclusion that the documentation is not sufficient or (most of the times) not available, and that the people who've build the system are long gone. The only thing that remains is "The Code is the Documentation". Well then you end up in what used to be so neatly called 'reverse engineering'. Actually what we are doing then is the modern form of Archeology. Scraping layer by layer of code, data and applications in order to understand the true meaning of the application landscape. The Software Archeology excercise is mostly done when we being asked to rebuild an old application into a new landscape. Most of the times we put developers to work who are trained in new technologies, but have got no clue whatsoever about the old technology in which the application is build. The result is that these teams are recreating over and over again the 'Stone of Rosette', or the key to translating the functionality from the code. We tend to forget that the system is not only build based upon functionality but also about what the technology enables and what the strenths and weaknesses are. So for instance the reason why procedural languages are used in heavy batch applications (performance....) only comes to live when the application is rebuild into an Object Oriented application, resulting in bad performance. So the AS-IS situation of a landscape, and all the reasoning behind it, is almost as (or even more) important then 'the Future state to be'. The roadmap towards the Future state to be is the task of a (Software) Architect who knows not only knows the 'New technologies' but has a strong background in the 'Old stuff' and all the reasoning behind it. Only then can we move towards a Stability Based Architecture (SBA).

The Data Refinery

Two years ago I was involved in a RFID project in the logistics market in joint Capgemini and Oracle effort. Based upon the EPCIS Architecture we had to combine movement data of crates between different storage facilities and stores. Without going into details about the case, on the technology side the biggest challenge in RFID is in the huge amount of data and how to get data out which makes sense for a particular use case. We came up with a division of the life cycle of RFID data in three main segments (in discussion with Ard Jan Vethman) (1) In real time (within seconds), you want direct actions, a crate is entering a room where it is not supposed/expected to. (2) In short term (hours, days, weeks, depending on the business case), you want to perform actions based upon a condition based upon multiple measurements. for instance when a shipment of fresh food between a factory and a warehouse normally takes 4 hours, you want to send an an alert when the shipment takes a day. (3) In longer term (more then weeks), you want to see how your processes are optimized against the measurement which are received. You can look at this as a Data Refinery, bulk data is processed as it flows in (complex event processing), the result or residue is a high quality small subset of all the data which is entered, the exceptions from all that is ok. The residue is then used as events which can be send to a Business administrator and systems administrator. Oracle provides the following technologies which can be used for this functionality. For the longer term we've known BI for a long time. But what is provided in the real/short term. For the short term data Oracle Business Activity Monitoring (BAM) has now been around for some years, and provides excellent insight, as well as graphically as with alerts. For the real time processing we had to program the event processing ourselves, but recently Oracle came up with Complex Event Processing (CEP) as a COTS product. This enables querying data on a data stream. Getting insight in a near real time data stream and do some cherry picking is still a new and under developed area and is an exciting area.

Aim for Stability Based Architecture (SBA)

Software Architecture is the art of translating Business needs into a suitable and working technical solution.
The same as in architecting and creating houses our sponsor expects that the product will be stable and last for quiete a time. Looking back in time we see that at the time of Cobol, applications were able to last more then 20 years (and even now exist, even after the retirement of their creators). But with the speed of new developments and the coming of hype driven development the application lifespan (or time that it gets outdated), is getting less and less. Java, among others, has got a bad reputation in this area, for instance Struts lasted some 2 to 3 years, after which you were the laugh of the year when you decided to choose Struts in your solution. SOA is another example, finally it paid of working with it, when somebody decided that SOA is dead and we have to run after the latest hype, Cloud computing. The half-time of a Software Architecture is getting shorter and shorter, and it happens too often that at the time of go-live we almost have to think about upgrading, because the version is almost outdated.
I'd like to propose a new architecture type, Stability Based Architecture (SBA), in which we aim for stable (core of the) application and add an expected life-time of the application. The core of the application should be stable and last for a long time, just like rock under pressure, it folds but doesn't break. Only then can we justify the costs of implementation.