Composing applications like Lego, SCA in a nutshell

Composition is all about combining functionalities exposed from different sources. Up until now SOA hasn't brought us the salvation of combining functionalities from disparate sources. There may be help on the way. Service Component Architecture or SCA offers a simple application model for wiring all functionality together.

Composition is about combining functionalities exposed within an application or from external sources. How is this implemented in programming languages:
Combining functionality within an application is standard practice in languages such as Java and .Net. The functionalities exposed and reused are fine grained, there lots of calls between objects and the communication is based upon a method type of call.
Remote invocations in the same language, from outside the applications, is hard to implement, it is error prone and mostly a lot of time is lost due to all connection and protocol complexity.
Crossing the border in a mutiple language environment calls for the SOA neutral language approach, typically done with Web Services. The usage model of a SOA type of call is completely different then in the case of combining functionality within an application. The exposed and reused functionalities are coarse grained, mostly the amount of calls between services is low and communication is based upon a document type of call. The implementation of Web Service invocation and handling, even though standard frameworks exists are still time consuming and all the error handling needs to take place within the application.
The complexity gets harder when the same service needs to be exposed with multiple communication protocols. Making a unified communication model based upon Web Services is unfortunately not the answer, due to the different usage model of the SOA and low level application type of calls. Web Services used for low level application calls, for instance, are killing for performance.

Service Component Architecture (SCA) offers an implementation model for combing functionality, from internal and external sources, into composites. It offers an environment which takes care of all the communication, or wiring, between the services. SCA is limited to combining application or data logic but is not aimed at the presention layer and the data persistency layer. SCA is a standard defined by the OSOA organization, and is backed up by the vendors IBM, SAP, Oracle (inclusing BEA), Iona and Sun.

Without going into detail here a short summary of SCA.
SCA provides four basis building blocks, Services, Components, Composites and Domain.
Service, Components and Composites are part of SCA Design Time environment, the Domain is the SCA run time environment.
The basic building block is the Component, this holds and implements functionality, which can be exposed via Services. SCA does not deal with with the programming part of the component, it only handles combining functionality together. The only thing which a developer needs to do is state in the component (for instance a Java Class) that at a certain point in the application external functionality is needed, this can be done via a Reference name added as an Annotation in the application.
The Composite is a virtual container (just an XML file) where Components are defined and wired together. Policies can be added to Components when specific requirements on the usage or communication are needed, for instance security or reliability.
SCA enables building applications easily just like combining Lego blocks. Components can be combined into composite. Composites can be reused into new composites. But be aware that the the chain is as strong as its weakest link. If anywhere in the composite there is component that is badly tuned, it affects the composite as a whole. Composites should therefore be accompanied by SLA's, what is expected of the Composite.
The run time environment of SCA is the Domain. Multiple composites can be contained into the Domain. The Domain takes care of all the communication handling and the policies. In contrast to the SOA principle of being vendor neutral, a Domain is based upon one language. This is based upon one of the lessons learned from SOA, that vendor neutrality for fine grained services is overkill from a design perspective and on run time can be resource intensive which can cause performance problems.
SCA uses a very pragmatic approach, when services and clients are written in the same language, there is no need for a language neutral representation. SCA handles all the communications between the objects, or wiring, in a way that is best suited for the language and the environment. When dealing in a hybrid distributed environment the communication is handled via Web Services.

What are typical use cases for SCA
(1) Data composites, in a SCA Data Composite, data coming from different locations can be combined into one composite and the composite returns the combined set.
(2) Application functionality Composite
(3) Process composites

SCA will be able to fullfill the promise of SOA as long as the basic principle, simplicity, will be preserved.

The risks of the SOA EDA promise, a collapse of the power grid of applications

Looking at history, where networks are created and can collapse, we need to learn in IT what the risk are of our attempts to unite the IT landscape with Service Orientation (SOA) and Event Driven Architecture (EDA). Examples in the financial world and the utility market should be a warning for IT of what could hit us. This calls for craftmanship of people in our profession who have the ability to look cross borders and control by means of Governance, in Design AND in Run time.

One example of a collapsing network is the current ongoing subprime mortgage crisis which still creates waves of unemployments, closing of companies an costing a lot of money. It is caused by globalization of the financial world, a would be believe that house prices would never go down, the growing dependancies of markets and the lack of control. At one point is became clear that the fundament of the economic rise, the ever increasing house price, was not a certainty anymore, and the network collapsed.

Another example is in the utility market. On august 14th, 2003 a disruption of the energy grid caused a blackout in the North Eastern part of the US and Canada, and hit 10M people. It started as massive power fluctuation in New York state and, in no time, affected all the areas. Here one flaw in one part of the energy grid caused the collapse of a very large part of the energy grid.

In IT we come from silo based architectures, every functionality (or set of functionalities) is contained in one silo, which may have integration points with other parts in the enterprise but stands essentially on its own. When one of these silos goes down it can cause a major problem for the enterprise, loss of sales, the going down of an essential production process, and in the end money and reputation damage. But at this point, it would NOT cause a cascading domino effect on other systems or enterprises. The promise of the SOA and EDA is the network of applications where everything can be connected and reused. Especially the reuse part can be the risk part, reuse with no insight in the relations can cause Single Points of Failures.

We are creating the power grid of applications for the future, that makes us responsible for making the right choices now.
One of these choices is to know the dependancies of the system components, for both in Design and in Run time.
At Design time, we need to be able to quickly identify all the dependencies in our landscape. This includes both the application types like custom build (based upon all types of technologies), package based, legacy and integration components. With the ultimate wish to be able to create an impact analysis list based upon all the touched components with just one push one the button. Another decision which is made at Design time, is whether we´d like to reuse an existing component, elsewhere in the enterprise landscape or even outside, and what the impact is if that (reused) component goes down. Based on that risk analysis we can decide whether to reuse that component or build it ourselves.
At Run time we need to have insight in the inter relations between all deployed components, and identify possible risk areas, for instance a service which is used by an ever increasing amount of applications.

Where should we start? In most environments people start of with Excel sheets, which in no time are not usable anymore because of the complexity, which calls for the need for proper Governance tooling (or at the basis a dependancy tracking tool). When you start with a Governance tool the environment is, of course, empty. Loading all the components and dependancies in your environment is time consuming and costly. This is the major problem why dependancy tracking tool, (SOA) Governance tools and Service Repositories are hard to start with in enterprises. The Business case at start is way too costly, even though we all know that on the long term it is essential for getting a grip on you IT environment. A diffentiator in this market will be the option to load an existing environment into the dependancy tracking tool repository.

The last piece of the process integration puzzle, looking at the UI part

Process integration is a subject that always comes back when you're defining a Software Architecture. There are different types of of Process integration, based upon integration partners involved, integration types, amount of users involved, connection type, length of the process etc. In the past loads of books have been written about process integration and a lot of patterns have been defined.
When looking at the Application to Application (A2A) and Human to Application (H2A) type of processes cross technology solutions have been defined in an XML type of language BPEL. BPEL is the conductor of a process and does call-outs to functionality by means of Web services and adapters to legacy systems.
The ease of functionality that BPEL offers for process integration however is not very usefull for a UI type of process like a close-out sequence of a book ordering process on the Web. These are typically hard-wired within UI frameworks, like in Java we do (or used to) in Struts or JSF. In these solutions there is no flexibity or reuse. Rearranging a UI process requires rework and redeployment, and on avarage there's not so much reuse of UI process components. When looking at a SOA perspective, you'd like to reuse (groups of) pages and handle them as services in order to create composite applications.
There is some movement in this market. Oracle came up in de Fusion Middleware 11 version with a UI Taskflow mechanism.
The basis for the Oracle Solution is Java Server Faces, or JSF, and is called ADF Taskflows. Components in these Taskflows are complete pages (JSF), page fragments (parts of JSF), Java call-outs and the most important one, the Taskflow definition. This Taskflow definition resembles the BPEL type of working, and is conductor using for handling a process from a UI perspective. It uses a plain XML file referencing all the needed components, or services.
Two types of Taskflow exist, the unbounded and the bounded taskflows.
Unbounded taskflows are just a predefined set of pages used to complete a task, this taskflow can be entered from all pages within the taskflow.
The Bounded taskflow acts more like a Service, only entry one entry point exists and zero or more exit points which can result in an output result. The inner behavior is 'private' and cannot be changed or interfered by the outside world.
Bounded taskflows are the most neat ones, I've been searching for these for quite some time now. Finally the possibility to reuse a group of pages within a new composite.
I hope the market picks up this initiative and creates a cross technology integration language for UI based process integration, just like we have now with BPEL.
For more information see

Oracle Application Development Framework ADF

Fusion Developer's Guide for Oracle, Part III Creating ADF Task Flows

Nice example pages

The cost of dead code

Once in a while I see this situation on the street. Thrash alinged around a garbage bin, somebody did an attempt to trow away his thrash but was too lazy to finish the job. 'Close but no cigar', so somebody else has got to finish the job, which in essence costs energy (and money).
In my job as a Software Architect I come across this behavior also a lot of times. Once in a while I've got to dust off an old application and investigate what the possibilities are for modernization or revitalization. Modernization ot revitalization comes down to, what can we do the extend the lifetime of an application for a couple of years more, and also be able to add agility, stability and new or changed business requirements to the application. It also helps identifying the roadmap for the future of the application. To make the roadmap for modernization there is a list of things to investigate, size of the application, the context in the application, complexity of the code, (future) business needs but also the amount of ´dead code´ in the application.
What is dead code? At one point in the history of the application functionality was removed or changed, but the actual code itself is still available in the application. For instance, a procedure which is not called anywhere in the application, a page which is removed from the menu, old functionality which is commented out, database objects that are not used anymore but still contain data etc etc. What you see here is scared or lazy behavior, scared as in 'I'm not sure if I removed or changed the right code', lazy as in 'I don't want to spend too much time'. The result is that the application is becoming like a jungle, you get lost in the dead wood. Making a change is getting more and more costly as a result of the increasing dead code. So let's change our behavior in the maintance phase, if functionality is not needed anymore just throw it away, like you would do with thrash in a garbage bin. When a proper versioning system is set up, you always can find the history back.

The seven challenges for Oracle

Oracle is moving towards the biggest provider of all software in the market, by means of smart acquisitions and a simple aim what they want. The aim of Oracle is to be bigger than SAP and IBM. I identify seven major challenges for Oracle the coming years. All challenges need to be guided by a sound roadmap towards the future.

What are the challenges for Oracle (and therefore also for Oracle partners and clients) the coming years:
(1) Integration of all products
Of course this is a no-brainer, and is a tough job. The roadmap helps in identifying integration points between the products to be integrated and still enabling growth for these individual products in parallel.

(2) Offering a roadmap towards the future
Last year, at the first of Juli, Oracle presented the acquisition of BEA with an excellent webcast, showing the preferred product stack for the future. Also included was a roadmap for the 100 days following the webcast which contained integration of some major BEA components into the Oracle stack. The 11 version of the technology stack would be a new milestone in the Oracle technology stack, but still is not in production and not clear what the choices are.
What I'm missing now is a decent roadmap for the future. Being a solution/software architect I need to know what is happening the coming years with a product stack when I propose a solution to one of my clients. This roadmap should contain product information, timelines and migration strategies for keeping track with the technology stack. Important is that the roadmap should not start from today, but also contain all 'old school' products of Oracle.

(3) Keep the 'old' Oracle customers on board
The impact of this fast change is that all existing clients (for instance 65K Database clients in EMEA only) see Oracle speeding away over the horizon. With the help of the roadmap we can help them move in a controlled pace towards the future, and not only for technology sake, but adding business value to their landscape.
Oracle should invest in migration tools helping the customers move towards the new technology stack. And not, as I've heard a product manager say, well open the 'old' and 'new' environment on two separate PC's and start typing... ;)

(4) Challenging other database products
Oracle is the market leader in the database area. This area however is getting commoditized more and more. Electricity was once a specialized product, but now you just get it 'out of the wall'. Speed, stability, availability and storage costs are an issue in the Database market, not extra functionality anymore. Since software is slower then hardware maybe Oracle should create hardware based Databases, on the new acquired Sun systems, with extreme speed and cheap in price. Another option would be offering the Database available in a Cloud.

(5) Challenging Open Source
In the Open Source market, more and more products are standardized and growing in experience. Products like MySQL and JBoss have the potential to be used in all types of core applications. One of the biggest advantages Oracle has over Open Source is that they can provide support and the likelihood that they will keep on existing during the lifetime of an application, and longer. Again here the roadmap is essential in providing information to clients when (not to) choose for Open Source.

(6) Challenging IBM
There's one thing where IBM beats Oracle, and that's architecture. IBM however has also the same problem as Oracle with their technology stack. They've got an extensive product stack with, not always, the best alignment among product components.
Oracle has got the opportunity to shape out a new landscape that is based upon a strong architectural base and is in complete alignment. The roadmaps shows the speed of shaping out this new architecture.

(7) Challenging SAP
SAP beats Oracle with Industry solutions. But with the introduction and work with the Oracle Application Integration Architecture (AIA) Oracle catches up with SAP. Oracle asks partners and clients helping with creating new AIA industry patterns. Here the roadmap can show the speed of development of patterns across industries. Advantage of Oracle over SAP in this arena is the momentum of development of the new patterns. SAP is struggling now with implementation of their SOA strategy and wether the choice of Java in their environment was a good one.

Of course Oracle can't do this on their own, but with the help of partners and clients we can come a good way, by helping creating a sound roadmap which will lead us the coming years in the Oracle world.

Thanks to Joost van der Vlies and Ruben Spekle

The four ‘FIT’ factors, making a Business Case for Open Source and Vendor products

Open source is hot in the market. It is free and you don't have vendor lock-in. True, a landscape based upon 100% dependency on one vendor can cause situations where your business wishes are not in agreement with the architectural possibilities (or pricing strategy…) of the vendor products. So are we better off with Open Source, where unlimited developers seem to be working day and night? As always is the answer, it depends...
So how can we decide whether to choose for Open Source or Vendor based technology?. This can be based on four ‘FIT’ factors: Functionality, Technology, Pricing and Support.

First of all, the Functionality fit has to be determined. Fit for functionality is the acceptance factor of the end users. Ability of the technology to (rapidly) adapt to functional changes also has to be checked. And the expected life time of the needed application has to be determined. How well can the needed functionality be mapped on the features of the software.

Then looking at the Technology. What is the needed stability of the application (this is related to support). What is the track record of the technology in this area. How easy can new technology be added to the stack and combined with other components. How much knowledge is available in the market. How open is the technology. How proven is this technology.

Pricing is of course also an important component. The pricing component not only consists of license- and support fees, but also development costs, and needed personnel after go-live to support the application.

Support requirements differ according to the type of application and it’s usage. This may range from an FAQ list, 24*7 help line all the way to immediate on-site support. In case of open source you should get an idea of the group developers that does updates, otherwise you may get stuck with the functionality you have.

What is the Business Case for Open Source
Functionality
+ Fast delivery of add-ons and new functionality by developers community
- No timelines for new functionality
Technology
+ Push from new technology (enables new functionality)
+/- Will it still exist at the expected end-of-lifetime of the needed functionality?
+/- Code set of Open Source can be changed by Development team, what is the risk for the future and ability to upgrade?
Pricing
+ No license fees
- Costs involved for keeping team of developers available for support
Support
+ A lot of independent developers are working on the source set
- In case of emergency, who to call
- Dependency on the developers after go-live (developers lock-in....)
- No official support

And the Business Case for Vendor based technology
Functionality
+ Track record of core functionality
+ Timelines for new functionality
+ Industry solutions implemented
- Marketing and sales strategy may have a higher priority than functional and technical developments
- Speed of development on new areas
Technology
+ Transactional stability (proven)
+ Track record of core functionality
+/- The role in the ecosystem as a provider of stable out-of-the-box systems is sometimes forgotten ?
+/- How well does the vendor do with regards to upgrades to newer versions/technologies?
Pricing
- License and support costs
+ A good fit between functionality and Vendor based technology / Industry solution can results in lower implementation costs
Support
+ High level of support possible

So depending on the needs of the system, the four 'fit' factors Functionality, Technology, Pricing and Support make the case for Open Source or Vendor Based technology. Any suggestions are always welcome!

Written with Joost van der Vlies, thanks to Ruben Spekle

Oracle Coherence experiences from the field

Last week I had a small conversation about Oracle Coherence on Twitter (#OraCohMDM). Here a more in depth description about the experience we've got with this product. Last year we came across a couple of projects which were an excellent case for the usage of a grid database based on the middle tier.

The first case was about processing bulk data in a small timeframe. Every minute some 150MB XML data was delivered (zipped) to our application by a Web Service. The requirement was that the data needed to be processed within 20 seconds and selected parts of the data needed to be send to to a bunch of subsribers. The architecture should be scalable for consumers and data load. For history reasons all the data needed to be stored in a database.

The second case could be described a having a high CIA constraint (Confidentiality, Integrity and Availablity). Multiple providers send data via Web Service, the messages were, contrary to the first case, small in size. The amount of messages however is very much higher, some 10-100 messages per second.

Since we know that (Oracle) databases are known to be stable, why can't we implement this with the help of a database? The answer is simple, there is a performance bottleneck between the middle tier and the database. In 'normal' situation, when dealing with for instance a web Based transaction application, the JDBC connection is able to do it's work properly. But when faced upon a situation with high data volume or high message volume, JDBC is clogging up. So how does a Grid database does a better job? Elements of success for a Grid database are the ability to scale up and down easiliy and minimize the risk of loosing data when for instance a server goes down. Preferrably there should be no master of the grid, all grid elements are equal. Simply put the Grid works as follows. A Grid is a series of interconnected nodes, which communicate via an optimized network protocol. Every Grid element has knowledge of the grid, what kind of work to do and the existence of their neighbours, just like bees in a beehive. When a new data object is entered in the Grid one Grid node takes responsibility and contacts another Grid node to be it's backup (preferably a node on another server). When a new node enters the grid, it communicates some sort of "Hello, I'm here", just like when you enter a party. All grid nodes start communicating with the new node, and (some) data is redistributed across the grid. This in order to minimize loss of data. When a node leaves the grid and again the (some) data is redistributed, this to avoid that a Data Object is only available on one node.
Oracle obtained Coherence (Tangosol) in 2007 because of it's excellent middle tier Grid capabilities. We've performed tests with Oracle Coherence in different hardware and network configurations. Our tests have shown that Coherence nearly scales linearly, with an optimum of around one node (or JVM) per CPU core. The addition of the JRockit JVM even removes further a smart part of the performance.

So how do you start with Oracle Coherence. Coherence is Java based, so a good knowledge of Java is essential. The data structure is stored as key/value pairs in maps (comparable to the Database tables). On each map a Listener can be placed, so an object change results into an event. By adding Object-Relational mapping (for instance with TopLink) you can percolate the newly added data into the database, or load the data from the database. By adding expiration time to the data within Coherence, you don't have to clean up your data.

Where to get more information

- SOA Patterns, deferred service state
- Oracle Coherence
- Oracle Coherence Incubator


Thanks to Oracle PTS (especially Flavius Sana & Kevin Li)