Ready, Steady, Stop - SOA gets in the way

Imagine your a supplier of software. You have invested in developing your software offerings over a decade or so and now you want to know how to respond to the move towards SOA among your customers. This is the problem facing a company I visited this week. They had two problems. The first is that they have several software offerings sold individually and in combination. Currently, these systems are not integrated and each time they sell a combination they hard wire them together afresh. The IT Department believes that the integration requirement is the same every time, but the implementers disagree.

The second problem is how to present themselves externally. If their customers are moving towards an SOA architecture, they are wondering if perhaps they should offer a loosely coupled integration option via web services.

The IT department is proposing that they adopt an SOA strategy and that they integrate their existing software systems using web services and offer web services as the means of integrating their products into their customers IT environments.

Unfortunately, this is a loose loose option. The first point is that adding a few web services to the existing software has nothing what ever to do with SOA. Adopting an SOA strategy would require them to re-build all their software from the ground up. Using web services simply as a means of integrating existing lumps of software is not only nothing to do with SOA's but also will not prove to be an advantage over using an EAI approach. Web services are slow and intended for inter company links where traffic volumes are low.

Unfortunately, the argument for using them for integration with the customers existing software systems is even weaker. With no advance knowledge of the system to be integrated with or how it has been implemented, it is unlikely that a pre-conceived web service will work any time let alone every time. Add to that that the traffic volumes involved are often very large and you have a recipe for disaster.

The Immediate problem for this companyis that the decision by IT adopt the use of web services and present it as an SOA strategy is blocking all attempts by the company's divisional management teams to address their immediate needs for a better way to meet their integration needs

Integration hub v ESB

I know several vendors of software products who are debating the same question: Should they go all out to use Web Services as the only primary interface for integration? Each of them sells software products that have to be integrated with other systems, often legacy systems in already in use at their customers.

This question raises two important issues plus one interesting technology question: The first is just how real is the movement towards SOA's i.e how long will it be before the big majority of their customer operates an ESB with which they can contract? The second is how important from a marketing perspective is it that they be seen to be ready to participate in an SOA?

The interesting technology question is: How good are Web Services for integration tasks anyway?

To give a view on the first question - there is a lot of activity in the marketplace. SOA's are the subject of the moment. That said, there are very few implemented examples of any meaning and the jury is definitely still out on whether or not the approach will give an adequate return on investment.

The answer to the second question - how important is the market perception of a products SOA credentials - paradoxically this seems to be very important. So here we have a conundrum. In practical terms there are very few ESB's in use. The majority of customers of these software's will need an alternative to Web Services for several years yet. But purchasers will want to be reassured that when and if they do get there their vendors will be ready for them

The more difficult question concerns Web Services themselves as the technology of choice. Being XML based they are slow - very possibly too slow for integration use where traffic volumes are high and/or each transaction is very broad.

Being practical, in the short term vendors will be forced to offer rules based EAI or, if their customers will wear it, individually hard coded integrations. Wouldn't it be good if there was a solution out there that offered its users the option to start will a rules based EAI approach and phase in an ESB solution when the customer was ready using the same suite of software?

Trend goes live with SOA solution

Trend Communications has gone live with a ground breaking on-line quotation and order entry system. Developed using EdenAgileIT (www.datadialogs.com).You can view it at

http://83.138.140.161:7734/TrendProto.html)

the solution consists of 150 re-usable services developed using Eden's codeless modeler and orchestrated with a combination of Eden's ESB and JAVA designer. The resulting Rich Internet Application revolutionises the way Trend does business through its distributors.

The approach Trend adopted was to leave their legacy ERP system alone rather than trying to expose any legacy functionality as services and the create the entire customer facing process set from scratch. Quotations under development are stored in a separate SQL Server database and only when an order is confirmed are the details of the assemblies required, with their bills of material passed into the ERP system (Infor's System21on an i-series) so limited are the options for using this ERP system within an SOA architecture that Trend decided not to try and literally Poke the data directly into the ERP systems database (This turned out to be much easier than the vendors scare stories would have Trend believe)

Web services were not used because the large volumes of data involved made them too slow. That's the third time this month I have encountered SOA developments electing to not use Web Services. It looks as though We Services are best suited to inter-company collaborative developments where high volumes are not an issue

Watching an SOA strategy take shape

Before I get on with the technical SOA stuff I promised, here is a diversion you might find interesting. I have come across a mid-sized company that packages and sells materials through supermarkets - Its a well known brand name with a turnover in the £60m range. Just the range that is going to have to deal with the issues I have discussed thus far in this blog.

They have already set up a strategy group to look at the feasibility of Service Oriented Architecture so I have the perfect opportunity to watch the process as it develops and write about it in this blog.

As soon as they give me the information, I will tell you about both the politics of the venture and the technical decisions they take with their reasoning. First decision will be how to start - Agility first i.e a quick kill by identifying a process or related set of processes or the purist, technology led approach through the creation of an ESB and an entirely new process built with all new services.

I can tell you the toolset they have chosen is IBM's Webspere and Weblogic. Watch this space!

SOA - What should I expect in return for my investment

The more people I talk to about Service Oriented Architecture, the harder I find it to present SOA as a one stop suits all solution. I can split the conversations into two groupings. The first set I would describe as managerial in that the questions centre around what the measurable benefits of implementing the technology should be. The second set are technical and centre around how the technology should be implemented. In this blog I am going to try and address the first group. The second group I will address in the next.

I guess the place to start is to re-state the driving objective behind the move to implement SOA's and that is to reduce the cost of delivering solutions to users. Of course, this needs further definition to be useful, since some managers might think that the most effective way to reduce such costs is not to attempt to deliver any. There are two sides to the cost equation. There is the cost incurred while waiting for a solution and there is the cost of delivering that solution. The balance between the two will form the core of any measurable ROI.

Service Oriented Technology is aimed squarely at both sides of this equation - the cost incurred while waiting will be less if the time taken to deliver the solution is less and the cost of delivering the solution will be less if it really is simpler and quicker to create than with any other technology as it is supposed to be.

Unfortunately, adopting this technology is not without an initial set up cost. How large this cost is and therefore how long it will take to reap a rewards for its introduction depends on the interpretation of the technology used and, that in turn, depends on the needs of the organisation proposing to adopt it.

Lets look at the options in turn. At the top of the pile we have the SOA purists. The approach proposed here will eschew any attempt to make a start by attempting to expose services from within traditional existing systems as "Sticking lipstick on a pig". This immediately requires the organisation to invest in the creation of the entire architecture from scratch and therefore the time taken to begin seeing any reward will be long - very long. For a technology that is presented as making IT more agile, this can be a dubious start. The same purist approach will demand that an Enterprise Service Bus (ESB) is implemented and that individual services have to contract with the bus. This is deemed necessary to ensure that no process requires any knowledge of the working of any of the services it needs. Instead it just needs to know what services exist in the registry and how to ask the bus for information from them at the right time. The bus knows content in what format the requesting service needs the information and it also knows what content and in what format it can get it from its register of services. In theory, I am now free to extend the capability of existing services or exchange them for new services at any time and only the ESB needs to understand the changes. Remember, any one service may be a constituent of many different processes.

If by now you are wondering what happens if a service is changed such that it no longer meets the need of every process that uses it this too has been addressed. Governance is the key to ensuring that services are managed so that this does not happen.

The problem I have with this is that there is a real danger that the very agility that is being sought can be negated by the need for tight governance restricting the development of services in line with the needs of the business. This in turn will lead to the development of ever more new services instead of encouraging the re-use of existing ones, thus undermining the whole theory. Whether this concern is well founded is still to be determined - there are not enough experiences around yet.

In terms of getting an acceptable return on the investment, the likelihood is that such a project will be unlikely to give a return on investment in a period of less than three to five years if current results remain the norm. Projecting the future that far is notoriously difficult and is the preserve of the very wealthy or foolhardy

The alternative to the purists are the incrementalists. Here the objective is make quick kills to convince the user community of the value of the approach. By definition, to make a quick kill corners have to be cut. Lipstick will be put on the pig so that existing investments in systems are preserved, An ESB may not be used and the overhead incurred when a service is altered swallowed, possibly inadvisedly, as an acceptable price to pay to make swift gains now.

This approach does, however, have something to be said for it, particularly because recent technology developments are riding to its assistance. The first development that helps concerns detecting and talking to services. The big argument for an ESB is that when a service is changed, only its contract with the ESB needs updating. This avoids the need to update every process with the change. However, there is now technology available that can discover and use a service without any need for re-programming. This makes this task much easier and undermines some of the advantages of an ESB and as ESB's are expensive swings the ROI balance in favour of the incrementalists.

The second development is one that makes the creation of services a code free exercise. This has several impacts on the argument. Its quick and easy to put lipstick on the pig, its cheap to create new services on demand and, most importantly of all, its very simple to re-engineer existing services. Taken together, these technologies when used incrementally put SOA technology within reach of businesses right down the the SME level.

Overview of Example of practical approach to SOA

Imagine a typical medium sized company. It has an ERP system it implemented two years ago, a CAD system and a PDM system plus an separate CRM package. The company has identified a requirement to upgrade its quotation and order entry system which currently is largely manual supported by multiple spreadsheets. This process is labour intensive, prone to error and seen by their customers as unresponsive.

In Service Oriented technology terms, the new process will be made up of the following identifiable compound services:

1. Handling customer data both existing and new
2. Creating a quotation/order header
3. Capturing order line details including Configuring the complex products
4. Creating bills of material as required
5. Printing Quotations/order confirmations

All of these services will be made up of several elemental services that can be re-used over and over again.

Each of the compound services needs to interact with the ERP system. If we start with the customer data, the ERP system needs to deliver customer name and addresses and delivery address information on request and also accept and process new customer data. Unfortunately, although the ERP system was purchased only two years ago, it is of traditional construction and does not offer any services. The first task, therefore, is to add service functionality to the ERP system. In fact, several services have to be created and exposed to get this job done.

1. Customer details existing and new
2. Quotation/Order header and line details input and recall
3. Parts data including costs and new bills of material input.

To accomplish this, I use a technique I call creating Application Service Managers. (ASM's) which I create with a code less modeling tool called EdenAgileIT. Essentially, ASM's are an extension to the Host ERP system that is integrated into it and exposes functionality as, in most cases, a web service. To build an ASM takes me between a day and three days, depending on the complexity of the service being created and the architecture of the host ERP system.

Once I have these ASM's in place, I can go on and create the new compound services required before orchestrating them into the new quotation process. These compound services can be very complex, especially those concerned with configuring products or services. It is not unusual for a configuration service to be made up of more than fifty elemental services. I use the same tool set to build all the services. We have accomplished this particular task for a number of customers now, and the whole process from start to roll out requires between 30 and 50 man days.