Wednesday, February 12, 2014

Enterprise Service Bus implementation patterns

This article describes technical criteria for selecting an ESB, shows how IBM products can implement an ESB, and then surveys common ESB implementation patterns. It covers the three primary IBM ESB products (WebSphere Message Broker, WebSphere ESB, and WebSphere DataPower SOA Appliances) as well as products that support or extend the ESB pattern (WebSphere MQ, WebSphere Service Registry and Repository, WebSphere Transformation Extender, WebSphere Adapters, WebSphere Process Server, WebSphere Business Services Fabric, and IBM Tivoli Composite Application Management for SOA. The article also describes two detailed ESB implementation case studies.
0 Comments
Victor Grund (vgrund@us.ibm.com), WebSphere Technical Sales Manager, IBM
Chuck Rexroad (crexroad@us.ibm.com), Executive Software IT Architect, IBM
05 December 2007
Also available in Japanese

Role and value of an ESB in an enterprise architecture

Service Oriented Architectures (SOAs) offer a flexible, extensible, and composable approach to reusing and extending existing applications as well as constructing new ones. The most important characteristic of an SOA is the flexibility to treat elements of business processes and the underlying IT infrastructure as secure, standardized services that you can reuse and assemble to address changing business requirements. Services in an SOA have well-defined interfaces defined by a set of messages that the service receives and sends, plus an implementation of the interface, and once deployed, a binding to a documented service endpoint.
An Enterprise Service Bus (ESB) is an architectural pattern that supports virtualization and management of service interactions between communicating participants. It acts as an intermediary for connectivity services between service providers and requesters in an SOA. It is a flexible connectivity framework that facilitates reliable and secure system integration while reducing the number, size, and complexity of application interfaces.
Rather than interacting directly, services communicate through a service connectivity framework (the ESB) that provides virtualization and management features that implement and extend the core definition of SOA. The ESB pattern provides virtualization of:
  • Location and Identity: The ESB identifies messages and routes them between interacting services. These services need not know the location or identity of other participants in the communication. For example, requesters need not be aware that a request could be serviced by any of several providers. You can add additional service providers to a functioning ESB, allowing messages to be routed to them without disrupting the requesters.
  • Communication protocol: The ESB enables messages to flow across different transport protocols or interaction styles as they move from service requestors to service providers and back again. For example a request expressed as SOAP/HTTP may be serviced by a provider that only accepts JMS input.
  • Interface: Requesters and providers of a service need not agree on a single interface. The ESB can reconcile differences between interfaces by transforming and augmenting the messages from the requester to the form expected by the provider.
You can implement the ESB pattern using a variety of middleware products (hardware and software), programming models, and interaction styles. An ESB:
  • Identifies messages and routes them between applications and services
  • Enables messages to flow across different transport protocols as they move from service requestor to service provider and back again
  • Transforms message formats between requestor and service
  • Recognizes and distributes business events to and from different sources.
  • Provides robust and secure communications
  • Creates an extensible architecture based on pluggable components
  • Provides intelligent routing and location-independent processing
  • Manages descriptions and definitions of messages and their formats through metadata
  • Integrates all types of assets to match the needs of your enterprise
Many ESB implementations leverage standards for service connectivity such as XML and Web services, which encapsulates the industry consensus view of the definition of an ESB. However, there is no need to limit the ESB to these standards. The ESB architectural pattern can be extended to support existing IT investments such as mainframe assets, packaged applications, sensors and devices, and files and custom protocols not based on any common standards. The ESB should support all of the connectivity needs of the enterprise.
An ESB should support different types of service interaction: one-way messages as well as request/response, asynchronous invocation as well as synchronous, and the publish/subscribe model as well as complex event processing in which a series of events may be observed or consumed to produce one event as a consequence of the relationships in the series.
Establishing an ESB is a key foundational investment for establishing SOA in an organization. It lets organizations:
  • Simplify: Reduce the number, size, and complexity of interfaces.
  • Reduce risk and costs: Enable IT to be more responsive to changes in the needs of the business.
  • Promote reuse: Make data and business logic more usable, and make applications easier to service-enable.
  • Support dynamic, real-time, event-driven SOA: Replace rigid, unresponsive, or batch-updating IT systems.

User roles and tasks

SOA, like any IT initiative, is dependent on technology, people, and process. A critical success factor in ESB adoption is that ESB technology must be manageable within a governance framework in order to bring value to the organization.
Many organizations find that an SOA Center of Excellence (COE) provides great value and speeds adoption of SOA technologies. The COE must reach out to teams across the organization to share experiences, provide education, suggest improved processes, and ensure the maturity of the SOA environment. The COE needs to assist the SOA Governance Board with issues that SOA raises, such as versioning, sharing of services, and security. The SOA COE is a cross-organization IT team that guides IT investment, design decisions, and implementation towards the strategic shared IT solutions targeted by the SOA vision and strategy. The COE assists with numerous issues, such as:
  • Governance
    • Serving as the information dissemination vehicle for SOA in the organization
    • Proving support for the SOA Governance Board in defining SOA governance and management processes
    • Implementing the SOA governance and management process
  • Thought leadership and visioning: Acting as the thought leader for SOA governance and management processes
  • Providing expert SOA skills and resources
  • Demonstrating knowledge management in the harvesting of assets
  • Communicating throughout the IT group and with the rest of the organization
The many roles of the COE are portrayed graphically below.

Technology selection criteria

Interaction style

In order to fully support the variety of interaction patterns that are required in a comprehensive SOA (such as request/response, publish/subscribe, and events), the ESB should ideally support in one infrastructure the three major styles of enterprise integration:
  • SOAs in which applications communicate through reusable services with well-defined, explicit interfaces. Service-oriented interactions leverage underlying messaging and event communication models.
  • Message-driven architectures in which applications send messages through the ESB to receiving applications.
  • Event-driven architectures in which applications generate and consume messages independently of one another.
A consumer can implement a service invocation in different ways. From the consumer's point of view, the difference is:
  • Synchronous: The consumer uses a single thread to invoke the service; the thread sends the request, blocks while the service is running, and waits for the response.
  • Asynchronous: The consumer uses a pair of threads to invoke the service; one thread sends the request, then a separate thread listens for and receives the response.
  • Publish/subscribe: A service publishes messages on a specific topic. Multiple services (subscribers) can subscribe to this topic and receive the published messages.
An ESB may offer any combination of these invocation models for a single service, letting service consumers select a preferred invocation model.
Interaction style can influence the ESB implementation. IBM® WebSphere® Message Broker is especially suited to architectures in which the basic flow paradigm is asynchronous or pseudo-synchronous.

Statefulness

Certain situations may require an ESB to maintain state as messages traverse it. A simple example involves selecting a particular service endpoint from the context information of the last applicable message. A more complex example extends the ESB paradigm by detecting complex situations that involve a context-sensitive composition of messages and events (semantic context, temporal context, spacious-temporal context). This functionality is extremely powerful when applied to the processing of input from multiple event sources: from the perspective of a business, application, or infrastructure within different contexts. For example, such scenarios may involve Service Level Agreement alerts and compliance checking, which is applicable to the security, finance, banking, and insurance industries.
Hardware ESB implementations do not generally support stateful interactions, making statefulness a criterion for software ESB implementations. WebSphere Message Broker, uniquely, implements complex message processing directly in the product, making it the most suitable for use cases requiring this function.

Endpoints, standards, and protocols

Endpoints are consumers and services that interact through the ESB.
Endpoints that employ standard technology and protocols (such as web services) allow the most flexibility in ESB technology selection. However, it is often not practical or cost-effective to restrict endpoints to these standards where large legacy investments exist. Similarly, while integrating packaged applications is possible using their proprietary APIs, it is often useful to delegate this integration to a middleware vendor. To address these concerns, adapters exist that can greatly simplify integration.
The use of adapters influences ESB product selection because adapters require a software runtime (they are not supported on hardware ESB implementations) and prerequisites for adapters vary.

Policy-based interaction management and dynamic service selection

An emerging ESB technology involves dynamic endpoint selection that exploits agent-based endpoint statistics such as server utilization and health. This is supported out of the box in certain IBM ESB products through integration with IBM Tivoli Composite Application Manager.

Message volume, size, and type

ESB products have different scalability characteristics. In general, hardware-based implementations scale quite well and have excellent support for industry standard data formats. However they do not provide the versatility of software implementations. WebSphere Message Broker performs very well, offers a very rich set of ESB functionality, and can be extended with custom components and adapters. WebSphere ESB has a very rich set of support for industry standard formats, Java and Web Services standards, and can be extended with custom components and adapters.

Reliability, availability, and serviceability (RAS)

ESB products have different operational characteristics and they achieve scalability and high availability through different mechanisms and architectural footprints (one example is application server clustering versus OS-level clustering). This must be taken into consideration and the resulting solution architecture must balance RAS considerations against existing budget, skills within the organization and operational complexity.

Required mediations

ESB products differ in their ability to handle mediations without custom programming. Notably, WebSphere Message Broker has extensive support for standard data formats such as ACORD, SWIFT and COBOL copybooks. If data formats are XML and Web Services, Websphere DataPower® SOA Appliances are an ideal fit, offering very high performance.

IBM ESB products

There are many products that can be used to create an Enterprise Service Bus (ESB). IBM provides several options for creating an ESB. This allows customers to select ESB technology based on their specific needs. In many large organizations more than one ESB technology may be used to create a hybrid ESB as required by differences in geography, technology, or for other reasons. Hybrid ESBs are especially likely when two large departments each embark on SOA separately and then find they need to interoperate with each other.
IBM's three main ESB products are WebSphere Message Broker, WebSphere ESB, and WebSphere DataPower SOA Appliances.

WebSphere Message Broker

WebSphere Message Broker offers capabilities that many customers use to create their ESB (or a central ESB in a hybrid solution). WebSphere Message Broker has its heritage in the WebSphere MQ messaging space, and is particularly well suited for environments heavy in MQ messaging. It offers interface definitions for WSDL and other message formats, mediation capabilities through message flows, and support for a range of communication formats, including WMQ and HTTP. WebSphere Message Broker also offers content based publish / subscribe interactions and managed topic spaces.
WebSphere Message Broker supports implementation of mediation patterns for a broad spectrum of service interaction endpoints. It supports a large number of industry standard message sets (some, but certainly not all using XML encodings) and enables support for additional, user-defined message formats. The following typical customer requirements are well suited to a WebSphere Message Broker solution:
  • Transactional
  • Publish/subscribe
  • ACORD, SWIFT, or COBOL copybook standard formats
  • XML formatted data
  • WS-* standard compliant
  • WebSphere MQ messaging
  • Complex transformations
  • Complex Event Processing
WebSphere Message Broker provides numerous ESB connectivity options and any-to-any data transformation. This enables legacy applications and those that do not conform to standards to connect to an ESB.

WebSphere ESB

WebSphere ESB is a J2EE instantiation of an ESB. WebSphere ESB runs within a J2EE container in the IBM WebSphere Application Server. It is well suited to J2EE and WS-* standards-based applications, especially those applications that only need to communicate with other application servers. It also provides an excellent entry point into the SOA world, allowing basic web services to be utilized as a stepping-stone into a more robust SOA environment.
WebSphere ESB provides standards-based Web Services connectivity, JMS messaging, and service oriented integration. JMS applications for point-to-point and pub/sub messaging, and JAX-RPC service oriented applications can connect directly to WebSphere ESB, or messages can be delivered to WebSphere ESB across a variety of transports including WMQ, SOAP/HTTP and SOAP/JMS. WebSphere ESB implements a web services gateway which can mediate between SOAP/HTTP and SOAP/JMS based applications. Finally, it also provides an implementation of Universal Description Discovery and Integration (UDDI). The following typical customer requirements are well suited to a WebSphere ESB solution:
  • J2EE implementations
  • Web Services interfaces
  • SOAP/HTTP

WebSphere DataPower SOA Appliance

WebSphere DataPower SOA Appliance is a hardware and software offering that provides a number of important functions: XML acceleration, security enforcement, and ESB functionality. DataPower has several important characteristics:
  • Optimized hardware, firmware, and imbedded operating system
  • A high level of assurance that the configuration is locked-down
  • Reduced security vulnerabilities
  • Hardware storage of encryption keys and a locked audit log
  • No hard disks, CD ROMs, or USB ports
  • Tamper-proof case that renders the machine unusable if opened
  • Reduced operational complexity, as it is truly an SOA appliance. DataPower provides the fastest entry into the SOA world, while at the same time providing an enhanced security environment.
The following typical customer requirements are well suited to a DataPower solution:
  • Security Gateway
  • XML firewall, parsing and validation
  • Basic routing
  • Content-based routing
  • Protocol bridging (HTTP, WebSphere MQ clients, FTP, ODBC, etc.)

Ancillary technology

In addition to the ESB technologies discussed above, the following complementary products can further accelerate or extend an ESB implementation:
  • WebSphere MQ provides reliable messages to/from many diverse platforms. WebSphere MQ provides the messaging backbone for many corporations.
  • WebSphere Service Registry and Repository provides an integrated service metadata repository to govern services and manage service lifecycle. It promotes service visibility, consistency, and decreases service redundancy in an SOA.
  • WebSphere Transformation Extender offers a universal transformation capability that is well suited to extremely complex requirements or rapidly changing requirements, such as EDI.
  • WebSphere Adapters are add-ons that help service-enable packaged applications or other legacy assets so they can participate in an SOA. Adapters exist that allow a wide variety of legacy information systems and technologies to be incorporated into the ESB.
  • WebSphere Process Server offers process workflow capabilities and includes WebSphere ESB. This solution provides the ability to choreograph complex ESB interactions that require coordinating service calls and replies from many systems.
  • IBM Tivoli Composite Application Management for SOA is the Tivoli monitoring offering specifically for SOA environments. IBM Tivoli Composite Application Manager for SOA leverages existing system management tooling and provides a holistic operational view of the SOA environment
  • WebSphere Business Services Fabric is an end-to-end SOA platform for the modeling, assembly, deployment, management, and governance of composite business services. WebSphere Business Services Fabric assists in the creation of business-level services that can be assembled into extended, cross-enterprise business processes and solutions that are dynamically personalized and delivered based on the business context of the service request.

WebSphere Service Registry and Repository

WebSphere Service Registry and Repository provides many capabilities beyond a basic UDDI service directory. In fact, many organizations have found that the UDDI specification is implemented differently by different vendors as the specification is not completely clear. WebSphere Service Registry and Repository provides not only a registry of what services exit, but also a facility for providing those services to an ESB as well as for developers searching for existing services to leverage. WebSphere Service Registry and Repository includes:
  • A service registry that contains information about services, such as the service interfaces, its operations, and parameters
  • A metadata repository that has the robust framework and extensibility needed to suit the diverse nature of service usage
WebSphere Service Registry and Repository interacts with almost all parts of the SOA life cycle:
  • Interacting with the service development life cycle with standard asset management solutions
  • Tracking the service life-cycle stages as the service goes through the change and release management processes
  • Providing optimized access to run-time facilities
  • Sharing information on services and service metadata with operational system management utilities.
WebSphere Service Registry and Repository provides information on who did what to which service, and when. This capability is discussed in a case study below.

Basic ESB patterns

The following section introduces a common language for core ESB mediation functions. These patterns are useful to show ESB mediation in context and can be used to express repeatable mediation patterns that are compliant with best practices and lessons learned. ESB Use Cases can be expressed as assemblies of these core patterns.
Protocol switch
  • Enables service requesters to dispatch their messages using a variety of interaction protocols or APIs, such as SOAP/HTTP, JMS, and MQ Integrator (MQI).
  • Transforms requests into the targeted service provider's format.
  • Can be applied at the requester or the provider end of an interaction, at both ends, or anywhere in between.
  • Mediations of this pattern can often be created automatically when there is a clear definition of formats and their relationship to each other
Modifier
  • Updates the payload of the message without any change to the context information
  • Transformation subpattern
    • Message payload is transformed from one format (schema) to another to match definition of a message of the requestor to that of a provider
    • Includes "enveloping and de-enveloping," which is the process of putting a message in one network format inside the format envelope needed for transmission over another network or corresponding removal of an envelope
    • Includes encryption
  • Enrichment subpattern
    • Message payload is updated by adding information from external data sources such as custom parameters of the mediation or database queries
Router
  • Changes the intended route of a message, selecting between the service providers associated with the mediation
  • Examples: gold customers to alternate, higher-SLA providers
Discovery
  • Queries ESB registry to discover a set of service providers that match requirements of the requestor, selects one, and routes the message to it
  • An enhancement of the routing pattern:
    • Set of possible service providers is not preconfigured at the mediation
    • Providers match requestor's message format, QOS required, or the protocol supported from the mediation to the possible providers
  • Example: Data center failover scenario. Bring data center online, register services, route traffic to it without having to updating the configuration of every mediation
Distribute
  • Distributes the message to a set of interested parties and is usually driven by the subscribers' interest profiles.
Monitor
  • Used to observe messages as they pass through the ESB without changing them in any way. Examples:
    • Monitor service levels
    • Provide data for problem determination
    • Chargeback Metering
    • Record business-level events (transactions exceeding a certain revenue)
    • Log messages for audits
Correlate (complex event processing)
  • Derives complex events from message or event streams. Includes rules for pattern identification and rules that react to pattern discovery, for example, by generating a complex event derived from content of the triggering event stream.
These basic patterns can be aggregated to produce more complex patterns.
Canonical adapter
  • Set of messages and business objects used by all parties is normalized to a canonical format. The canonical adapter pattern converts endpoints' native bus-attachment protocols to a standard protocol, normalizes payload, and reverses these transformations upon delivery. This is an aggregate of Protocol Switch and Transform patterns.
Transform, log, route
  • A common ESB aggregate pattern. Transform incoming messages (Transform), log them (Monitor), and Route to endpoints.
Proxy or gateway pattern
  • Variant of routing or protocol switch pattern which maps service endpoints, possibly providing security functions (authorization and access control) and logging or auditing capabilities.
  • May incorporate transform and monitor mediations to provide encryption and logging, or auditing. It may also aggregate and disaggregate messages in a one-to-many relationship.
Example: Service portals which act as a single point of contact for multiple services and hide the details of "internal" services

Complex ESB patterns

These patterns provide a more real-world perspective on how to use ESBs to achieve the rich service virtualization desired. In many large organizations or in organizations working with outside SOA partners, there is often a need to use more than one ESB. This need occurs for many reasons, including:
  • Security
  • Performance
  • Accountability/Auditability
  • Control of resources
The following patterns are illustrative but are by no means an exhaustive view. Many of these patterns can be combined with each other, depending on the needs of the particular project.

Global ESB

The Global ESB provides all service requestors access to one registry, with each service being visible to every service requestor. This pattern may not be appropriate for a production environment, although it may be appropriate for small system testing environments. This discussion of suitability refers to using a Global ESB without other ESBs, in other words a non-hybrid approach. Hybrid approaches are discussed later in this section.
  • All services share one namespace, and each service provider is visible to every service requester across a heterogeneous, centrally administered, geographically distributed environment.
  • Used by departments or small enterprises where all the services are likely to be applicable throughout the organization.
In the case of a global ESB, there are two major options for ESB technology: WebSphere Message Broker and WebSphere ESB. WebSphere ESB is the preferred choice where the entire environment is J2EE and standards-based. Where there are complex transformations and interactions with non-J2EE Systems WebSphere Message Broker is likely more appropriate. Large organizations will likely find it appropriate to run both products, routing messages appropriately based on message type and destination as discussed in the next example, the Federated ESB.

ESB gateway

The ESB gateway pattern provides a controlled and secure service interaction between internal or external domain boundaries.
The ESB gateway pattern is often instantiated using the DataPower appliance, as this provides an XML firewall in addition to the desired gateway functions. The DataPower is well suited to serving on network boundaries as it uses an embedded operating system and software implementation, thus reducing the risks inherent in setting up an operating system environment.

Federated ESB

The federated ESB provides the ability to use multiple service registries and administration domains while mapping the disparate registries into a federated set of services. The federated ESB facilitates service interactions with multiple ESB implementations. The federated ESB is often used to provide a subset of services that are applicable throughout the enterprise.
  • One master ESB into which several dependent ESBs are federated.
  • Service consumers and providers connect to the master or to a dependent ESB to access services throughout the network.
  • Used by organizations that want to federate a set of moderately autonomous departments under the umbrella of a supervising department.
In the case of the Federated ESB the "core" ESB is likely to be WebSphere Message Broker, while the non-core ESBs could be WebSphere ESB or in the case of small remote offices, could be DataPower appliances. In this pattern each site has its own Service registry. In the Federated ESB pattern the best practice would be to use DataPower appliances at network boundaries to enforce authorization and authentication, as well as to parse and validate XML before it is presented to internal systems. In this way, the DataPower appliance can accelerate the speed of the transaction and reduce CPU utilization of the ESB servers.

Brokered ESB

The brokered ESB is used to support multiple registries where a subset of services is applicable throughout the organization, but without there being multiple implementations of the service.
  • Bridge services that selectively expose requesters or providers to partners in other ESB domains
  • Each ESB manages its own namespace.
  • Service interactions between ESBs are facilitated through a common broker that implements the bridge services.
  • Used by departments that develop and manage their own services, but share a few of them, or selectively access services provided across the enterprise
The brokered ESB pattern is very similar to the Federated ESB pattern, with the exception that the brokering ESB is an enforcement point as well as a router for services.
The JBI standard (JSR 208) is often discussed in reference to brokered ESBs. To date, IBM has abstained from the JBI specification because it doesn't represent a sufficient step forward above what customers can already do. Many technologies and open specifications are available with more compelling interoperability and better mechanisms for component composition. IBM's priority is to enable integration with the broadest range of products, application, and existing business assets.

ESB scenario #1: ABC Hotel

ABC Hotel is a global hotel and leisure company. It offers luxury and upscale full-service hotels under several brand names. This includes over 900 owned, managed or franchised hotels with over 230,000 rooms in inventory.
ABC initiated a large, multi-year IT project to replace their large, monolithic mainframe-based Central Reservations System (CRS) with one implemented as a set of distributed J2EE services. Drivers for this project were cost reduction (mainframe replacement) and increased business agility. The project was conceived largely as one that would employ concepts and doctrine of SOA. Services are to be deployed in two geographically distributed datacenters. Multiple instances of individual services will be deployed to facilitate system throughput and availability.
Notably, the CRS project was originally conceived without a general purpose ESB framework. As the project matured, the complexity of service integration rose to unacceptable levels. It imposed a large burden on architects, developers and operations and was an inhibitor to successful completion of the CRS project. Thus a decision was made to refactor to incorporate an ESB. ABC undertook an evaluation effort to select the proper course of action.
Summary of ESB requirements presented to IBM:
  • Request routing and workload management: Multiple copies of individual services were deployed within the infrastructure to facilitate reliability, serviceability, and availability. The ESB should route client requests to appropriate versions and instances of requested services. The ESB should also make some provisions for workload management, distributing requests to identical service instances. This could be a simple load distribution algorithm but there was strong interest in more active, endpoint-driven load workload management.
  • Registry integration: ABC employed a UDDI-based service registry for a build-time catalog of services. There was no provision for more active, runtime service registry but the requirement to support such a registry was understood.
  • Mediation: ABC expected the ESB to mediate between clients and services. Service data is expressed in WSDL / XML so there was no need to handle more esoteric data formats. Message validation (WS-I and XSD) as well as support for performance / transformation acceleration were key drivers.
  • Transaction management: ABC had certain use cases which would require compensation and rollback of committed service requests (distributed transaction support in composite services). They desired support for rule-based, atomic and process-driven transactions.
  • Service choreography and workflow: Complex workflow support was a requirement for a small subset of service interactions. BPEL support was highly desirable for these cases. Support for business state machines, scheduling and timing of interactions, event correlation and workflow involving human interaction was also preferred.
  • Ability to facilitate integration with backend systems: Ability to integrate directly with Siebel and other backend systems was a consideration.
  • High performance / low latency: The ESB should have a minimal negative impact on overall performance and latency.
  • Monitoring, managing, maintaining: Collectively, there were broad requirements for service level monitoring, collecting standard and custom metrics, fault detection, root cause analysis, reporting and historical analysis. Automated operator alert and notification facilities were critical.
  • Security: Integration with Tivoli Access Manager, LDAP and other third party security products was a requirement. Credential and key management capability, standards-based authentication, access-control, encryption / decryption, SSL acceleration, Sarbanes-Oxley (SOX) controls, and suitability for DMZ deployment were all considerations.
  • SDLC: ABC uses Eclipse-based Rational Application Developer and Rational Software Architect. ESB development tools that fit well within their existing environment were preferred.
  • Fast deployment: ABC had already invested considerably before choosing to refactor to include an ESB. Cost and speed of deployment were critical considerations.
  • Reduced operational complexity: Reduction of risk and complexity was what drove ABC to refactor to an ESB-based architecture. ABC had a strong preference for a turnkey, low-complexity solution.

Sample use cases

Business partner invoking backend services
ABC regularly communications with business partners for room availability and booking requests. There is a requirement to virtualize calls to backend services using a DMZ deployment pattern. Clients must be authenticated and authorized. Also, all activity must be monitored and logged.
Segmenting traffic by client, request type or other information is a possible future requirement.
Hotel property invoking backend service
Individual properties have local property management systems (PMS) that must interface with the CRS. One scenario is a property updating rooms and closing out inventory. The CRS must be made aware of the changes in room inventory. Properties interface with the CRS using XML over MQ, while CRS services may be deployed using another wire protocol such as SOAP/HTTP.
Similarly, augmenting this pattern with and ability to route by endpoint utilization statistics is a possible scenario.
Together, the above two use cases represent a large percentage of all ESB traffic.
Fault with complex recovery scenario
In the scenario, envision a fault in a composite service interaction while fulfilling a client request from a property. Recovery from the fault is deemed significant enough to require a workflow that possibly includes human recovery tasks. The core ESB patterns do not change although the fulfilling architecture will include additional components to enable this function.
Complex workflow with backend integration
In this scenario, a create customer request is entered into a preferred guest tracking system. Business users want portions of this information entered in Siebel as part of this mediation. The ESB fulfills standard mediation of existing services while also including an update to an ISV backend system. The transform-monitor-route ESB aggregate pattern applies here but the fulfilling architecture utilizes multiple IT components to fulfill the service request.

Solution architecture overview

IBM proposed a hybrid ESB architecture consisting of both hardware and software ESB implementations. Each has different strengths in ESB functionality.
Appliance
  • Optimized for very high performance, very low latency processing
  • Simplest configuration, deployment and operation
  • Secure, tamper-proof
  • Offloads expensive processing from general purpose platforms
Software
  • Support application hosting, workflow support, and complex mediation
  • Application integration via Adapters
  • Enables Business Activity Monitoring
IBM expects many customers to deploy hybrid architectures that leverage strengths of both approaches. The hybrid architecture is augmented with SOA Service Registry and Management capabilities.

Solution components

  • DataPower XI50 (hardware ESB)
  • WebSphere Process Server (includes WebSphere ESB)
  • WebSphere Business Integration Adapter(s) (integration with backend systems)
  • IBM Tivoli Composite Application Management for SOA (service monitoring and management)
  • WebSphere Service Registry and Repository (service registry)
The following system context diagram illustrates the interaction of these components.
The DataPower XI50 fulfills two roles in this architecture: Service Gateway (in the DMZ) and general Hardware ESB (in the Trusted Zone). Tasks that require a software ESB implementation (such as using a Siebel Adapter or complex error recovery with human interaction) are offloaded to WebSphere Process Server includes as a core component WebSphere ESB.
DataPower fulfills the majority of ESB traffic requests at ABC, enabling very high performance. WebSphere Process Server and WebSphere ESB augment DataPower to allow maximum flexibility in the architecture.
IBM Tivoli Composite Application Management products enable monitoring and management of the ESB and components that interact with it. Information collected monitoring agents can enable complex ESB Enrichment and Routing patterns, such as dynamic service selection (selecting a target service based on information collected by monitoring agents, such as hosting server load).
WebSphere Service Registry and Repository enables full lifecycle governance of services and plays a role in enabling dynamic service selection in the ESB. As noted above, UDDI alone is inadequate for these tasks.
This architecture is intended to strike a good balance between performance and flexibility.

ESB scenario #2: XYZ Insurance

XYZ Insurance is a medical insurance provider. The company insures millions of customers and must interact with the customers, providers, third-party agencies, and others. This example is a composite of a number of real-world insurance projects.
XYZ decided to embark on an SOA project as a way to modernize their Information Technology while leveraging a substantial investment in mainframe (z/OS) applications written in COBOL, Assembler, and PL/I. These systems use DB2 and IMS databases and much of their communication with end-users is through CICS (Customer Information Control System). IBM 's WebSphere MQ provides much of the intra and inter-application messaging backbone. There is a main datacenter which is connected to several satellite datacenters and to third-party systems.
XYZ decided to use a Service Oriented Architecture (SOA) to build a new Call Center Environment (CCE) and to build a new claims processing Application (CPA). Many other applications and changes are taking place, but we will focus on these two major systems.
The overall project requirements that IBM and XYZ used to define the architecture included:
  • Leverage legacy applications without requiring re-write of code,
  • Leverage mainframe systems, while creating new Services on the platform of choice for that application,
  • Support multiple communications protocols (SOAP/HTTM, WebSphere MQ, etc.)
  • Securely support SOA interactions with outside organizations
  • Operate in an environment with several data centers, geographically dispersed
  • Provide the ability for real-time monitoring and routing of transactions based upon desired level of service
The ESB requirements are similar to the ABC case study, except that more complex mediations are required. Also, due to many regulations, security and auditability must be more robust than in the ABC case study.

Sample use cases

Remote datacenter invoking main datacenter services
One of the remote datacenters makes frequent calls to the main datacenter in support of the Cal Center Environment (CCE). While this is considered to be within the organization, this case still uses a gateway to ensure application security and audit requirements are met.
In this case, a single service call to the main datacenter can result in multiple service calls within the legacy applications, thus requiring service choreography and workflow components to assemble the resulting information into a single service response.
The remote datacenter side of the service interaction is represented below:
The main datacenter side of the service interaction is represented below. Please note that the gateway and monitoring components are not shown in the diagram for clarity, but are certainly vital components of the solution:
Main datacenter invoking remote datacenter services
The main datacenter will frequently call one of the remote datacenters in order to ascertain the latest Call Center Environment (CCE) activity. This requires the remote datacenter to have a complete SOA environment and to replicate the security and audit functionality of the main datacenter.
The main datacenter side of the service interaction is represented below:
Main datacenter invoking a third-party service for address verification
The main datacenter calls a third-party service to verify that addresses are valid before those addresses are accepted and stored in XYZ's databases.
Outside agency making services calls on the legacy application services
While an outside third-party application would have no reason to know that the service being called is a legacy application, the use case is architecturally interesting.
The remote datacenter side of the service interaction is represented below:
The main datacenter side of the service interaction is represented below. Please note that the gateway and monitoring components are not shown in the diagram for clarity, but are certainly vital components of the solution:
Fraud detection based on service call patterns
Fraud detection based on service call patterns is a complex case requiring monitoring of not only the number of calls, but also what entity made the calls, and in what timeframe. In this case, the information on a particular user making an abnormally high number of calls involving one customer or provider temporarily disables the user's access and provides notification so that XYZ can investigate the abnormal behavior. This monitoring, logging, correlation and security system integration presents a significant challenge, but can be met with existing ESB technology, along with other SOA technologies.

Solution architecture overview

IBM and XYZ worked together to create a hybrid ESB architecture consisting of hardware and software ESB components. In this case all three of IBM's ESB technologies were brought into play to provide the required functionality. The need for legacy integration, service choreography, and basic workflow drove the architecture decisions in using the technology set.
The following system context diagram illustrates the interaction of these components.
As in the ABC example, the DataPower XI50 plays two roles: security firewall in the DMZ and hardware ESB in the trusted zone. WebSphere Process Server/ESB is used to choreograph complex service interactions, taking data from the data fabric and combining it into more complex business service replies. WebSphere Message Broker is used to handle complex message transformation needed to interact with legacy systems. This example shows the value of using all three IBM ESB solutions in one design, as each is leveraged to take advantage of its most significant capabilities.
IBM realizes that many organizations will have similar complex needs as XYZ. These requirements will drive the use of several technologies in order to provide the most appropriate capabilities and meet availability, performance, and security requirements

Solution components

The solution used by XYZ includes:
  • DataPower XI50 (hardware ESB, XML security gateway)
  • WebSphere Process Server (includes WebSphere ESB and process choreography of complex service interactions)
  • WebSphere Message Broker (software ESB supporting complex message transformation and non-standard messaging)

Conclusion

This paper provided conceptual and real-world examples of ESB usage. It discussed technology options for creating an ESB, and which products are best suited for particular situations. As with any technical topic, this guidance should be used in concert with experienced practitioners in order to design the most appropriate solution for a particular situation.

Acknowledgments

The authors wish to acknowledge the efforts of Marc-Thomas Schmidt, Kyle Brown, and Rachel Reinitz. Their work provided the basis for much of the IBM SOA initiative and is particularly important in the area of ESBs.

Resources

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)