To ESB or not to ESB

Many of us have had to ponder this question. Technology selection is notoriously difficult in the enterprise space since the criteria and complexity of the problem is often not fully understood until later in the development process.
There is an interesting post from ThoughtWorker Erik Dörnenburg with the unfortunate title “Making the ESB pain Visible”. Erik provides a real-world example of when not to use an ESB citing that -

Based on conversations with the project sponsors I began to suspect that at least the introduction of the ESB was a case of RDD, ie. Resume-Driven Development, development in which key choices are made with only one question in mind: how good does it look on my CV? Talking to the developers I learned that the ESB had introduced “nothing but pain.” But how could something as simple as the architecture in the above diagram cause such pain to the developers? Was this really another case of architect’s dream, developer’s nightmare?

Later, Erik quite rightly points out that an ESB should not have been used in this scenario. This is a fairly common problem for ESBs and other enterprise technology like BPM/BPEL where the technology is not chosen for the right reasons and then the technology gets blamed when the project struggles. Given that much of the enterprise software disillusionment today stems from the incorrect usage of the technology I thought I'd offer some rough guidelines for selecting an ESB.

ESB selection checklist

Mule and other ESBs offer real value in scenarios where there are at least a few integration points or at least 3 applications to integrate. They are also well suited to scenarios where loose coupling, scalability and robustness are required.
Here is a quick ESB selection checklist –

1. Are you integrating 3 or more applications/services? If you only need to communicate between 2 applications, using point-to-point integration is going to be easier.


2. Will you really need to plug in more applications in the future? Try and avoid YNNI in your architecture. It’s better to keep things simple re-architect later if needed.


3. Do you need to use more than one type of communication protocol? If you are just using HTTP/Web Services or just JMS, you’re not going to get any of the benefits if cross protocol messaging and transformation that Mule provides.


4. Do you need message routing capabilities such as forking and aggregating message flows, or content-based routing? Many applications do not need these capabilities.


5. Do you need to publish services for consumption by other applications? This is a good fit for Mule as it provides a robust and scalable service container, but in Erik’s use case all they needed was an HTTP client from their front-end Struts application.


6. Do you have more than 10 applications to integrate? Avoid big-bang projects and consider breaking the project down in to smaller parts. Pilot your architecture on just 3 or 4 systems first and iron out any wrinkles before impacting other systems.


7. Do you really need the scalability of an ESB? It’s very easy to over-architect scalability requirements of an application. Mule scales down as well as up making it a popular choice for ‘building in’ scalability. However, there is a price to be paid for this since you are adding a new technology to the architecture.


8. Do you understand exactly what you want to achieve with your architecture? Vendors often draw an ESB as a box in the middle with lots of applications hanging off it. In reality, it does not work like that. There is a lot details that need to be understood first around the integration points, protocols, data formats, IT infrastructure, security etc. Starting small helps to keep the scope of the problem manageable and keep the fuckupery to a minimum. Until you understand your architecture and scope it properly you can't really make a decision as to whether an ESB is right for you.


9. Generally, always validate a product solution for your needs. Don’t choose an ESB or any other technology because –
   
   
   
   • It will look good on my resume
   
   
   • I don’t need the features today but there is a remote chance that I _might_ in future
   
   
   • I had a great golfing weekend with the head of sales
   
   

This checklist is not exhaustive, but will help clarify when not to use an ESB. Once you have decided that an ESB is a good fit for your project you’ll want to add additional selection criteria such as connectivity options, robustness, error management, service repository, performance, data support, etc. The important thing to remember is that there is no silver bullet for good architecture and you need to know your architecture before making a technology decision.

With this checklist in mind it’s easy to see that Erik’s example never needed an ESB in the first place.



However, if the architecture looked something more like this, then an ESB would have probably been a good fit.

Choosing Mule

Obviously, as the creator of Mule I have some bias for wanting everyone to use Mule. However, it is critical to the continued success of the Mule project and to MuleSource that users understand when not to use an ESB. Open source makes a lot of sense for enterprise software because projects need time to try out the technology and refine their proposed architecture. Having access to the product and source code helps a huge amount in this discovery process and allows the customer to make an informed decision.

In fact the MuleSource business model hinges on the ability of the user to self-select Mule and approach us only when they need development or production support. This has been working very well for everyone involved. Customers get to do a proof of concept (PoC) with our product knowing that if they end up using it for a mission critical application that they can get professional 24x7 support. For MuleSource it means that our customers buy from us rather than us selling to them, so they always get what they want – this is a far cry from the old proprietary upfront license model that used to hold the enterprise market hostage.

Fatherhood is fantastic, but not for blogging

My wife and I became parents for the first time a couple of months ago. It’s been an amazing experience so far and has changed our lives beyond recognition - mostly in a good way. Part of me wants to pour all over this, but I’ll leave that to the blogging dads that have the emotional maturity to describe their feelings above and beyond “a kinda fuzzy, cuddly sensation”. Instead I thought I’d share a couple of pictures. And since we’re starting to get the hang of parenting I am making a wild resolution to start blogging again.


Life changer


Evil Dad

Ten Tips for Technical Presentations

Today I attended the morning session of Sun’s open source day in Malta. I was pretty disappointed by the quality of the sessions. These sorts of events cost time and money, doing them badly can do more harm than good. I don’t think Sun did themselves any favours today.

I thought I’d share some tips that I have picked up over the years that would have made today a better experience for the audience.

1. Always introduce yourself and what you are presenting. I watched 4 presenters today where only one did this. I want to know who is talking and more importantly what the talk is about.


2. Understand the single message you want to get across to your audience and make sure the presentation supports it. It’s very easy to try and include the kitchen sink when you have an hour with a captive audience. However, it’s better to use that time to get a single message across that trying to cover many different facets of the subject.


3. Articulate the goals of the presentation and conclude with a recap. This seems obvious but a presentation should have an agenda. This clearly defines what you will cover and lets the audience know what to expect. The content should circle back on these goals and ultimately leave the audience feeling they learned something useful.


4. Tailor your material to run in the time allotted. Two of the presenters apologized for having too many slides then proceeded to rush through the content. It renders the session pointless.


5. Understand your audience. Most presentations I see the audience is usually curious but not an actual user. There is an art to presenting information to new users and I confess I haven’t quite figured this one out yet, but I am conscious of it and do seek other people’s opinions.


6. Always provide the 'Why' in your material, all I heard today was the 'What'. Try and answer some fundamental audience questions such as-
   
   
   
   • Why should I give you an hour of my time?
   
   
   • Why should I use this project/approach/pattern?
   
   
   • Why is this method better than others?
   
   
   • Why will I remember this talk?
   
   
   
   


7. It is almost inevitable in a technical presentation that you’ll want to look at code. Code and slides don’t usually work well together and IDE resolution is usually too small. Make sure that any code will be readable 30 feet away and ensure you don’t bombard the audience with reams of code that they cannot absorb.


8. Always solicit feedback. You have a captive audience, use them to ask a few yes/no questions that will help you understand the level of knowledge of your audience. The feedback can be used when delivering the presentation right away and in future.


9. Reduce technical jargon. You can only get across a couple concepts in an hour. Make sure you describe and reiterate these. Remember these should tie into the goals and overall message of the presentation.


10. Learn from others. Use resources such as Parleys.com to watch other presentations. Note the ones you like and figure out why. Doing good presentations is a skill and like any other skill you need to practice and learn from others.

Also see: Giving effective product demonstrations.

UPDATE: Tadayoshi Sato has translated this post in Japanese. Thanks Tadayoshi!

Gmail Fails, Cloud Computing Fails

While I don't go to the MotleyFool for my tech news, they ran an article this week about the Gmail Fail (or #gfail on Twitter). The Fool was in praise of Gmail because even though the service was down for a few hours some interfaces such as IMAP for the iPhone kept working (though this wasn't true for me). To me the fact that Gmail went down and affected people in Europe and the US was a big blow for the cloud exuberance we're currently experiencing. One of the benefits around cloud computing is being able to scale and utilise redundant machines if other machines fail. In this instance we had millions of users across continents lose email. What's worse it wasn't just the free Gmail service that failed but also their premium service. This means that the premium Google Apps service missed their SLA and Google have had to compensate (we got an email from them yesterday).

One can't help but see this as a blow for the Cloud since it shows how vulnerable we can be when we lose service. It also tells us that the one company who showed the world how to scale is not immune to severe outages. This incident didn't cost Google much, but most mission critical architectures that I have worked on would have suffered a 3 hour outage in a crippling fashion.

Does SalesForce go down all at once too? What assurances do we have in the cloud other than SLAs that can be easily broken? It is going to take a long time before companies will trust public SaaS services for anything remotely mission critical, they may never overcome that hurdle.

How SOA paved the way for Cloud

Though some think SOA failed and others see it only as a partial success, this has been because of the approach to SOA not its underlying principles. SOA (or what I’d rather call Service Orientation, but that doesn’t fit our obsession with TLAs) is responsible for beginning a new evolution in the way we build software. This change is as fundamental as the shift to OO programming but essentially a much bigger jump - OO was adopted one developer at a time, but SOA requires teams and departments to make the culture, technology and mind shift together. Where SOA has been most successful is evolving the way we build applications to the point where new deployment models become possible; enter the cloud.

There isn’t anything terribly new about the cloud. Virtualization (VMWare, XEN, Parallels, Hyper-V) has been around for a while, application scale-out solutions have existed in the forms of Grid computing (GigaSpaces, GemStone, Globus, GridGain, Paremus), Map Reduce (Hadoop) and distributed caching (Coherence, GigaSpaces). We could build clouds in the past, but there was a lot of work involved. Infrastructure as a Service (IaaS) was coined in 2006 and Amazon has been at the forefront of this movement. What changed recently was the ability to pull all of this stuff together in an easy way and the utility pricing model, driven by Amazon, has put all this technology in the reach of a far wider user community that couldn’t afford to play before.

In order for cloud to work, we also needed to change the way we built applications to run in the cloud. SOA has paved the way here, shifting our focus from building isolated applications that communicate using point-to-point integration to thinking of these applications in terms of services. Services are discrete pieces of logic that have a well-known contract and well understood function. This enables consumers to invoke services and execute their functions. The fundamental shift here is that you must think about how others will interact with your code. This is quite a big jump since most developers can happily develop in a vacuum and just get things working. In a service oriented application, your consumers may be your co-workers, other departments, the whole company, business partners or the whole world. The fact that other people will be using your services adds additional burden since interfaces need to be defined with consideration to hiding sensitive data, Security, versioning, data types, interface languages and a consistent API across services. SOA has been conditioning us to do this. For many organizations moving to the cloud is much easier because the heavy lifting has already been done. Applications that have been componentized into services are much better suited to the cloud, by their nature components are loosely coupled and can be hosted with location independence.

However, Moving your services to the cloud isn't necessarily a plug-and-play task. The cloud changes some of the rules for applications, I'll cover this in a follow up blog.