Open Source Integration with Apache Camel and How Fuse IDE Can Help
Take any integration project and you have multiple applications talking over multiple transports on multiple platforms. As you can imagine, in large enterprise applications this can get complex very fast. Much of the complexity stems from two issues:
Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and DSLs. There is of course a lot more going on here under the hood but we'll leave that for another discussion.
Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 13 essential components. There are over 80 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queueaQueue and send them to a file system directory "/tmp", you could use URIs like "jms:aQueue" and "file:/tmp".
Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports over 40 patterns from the EIP book and many other useful Processors.
To wire Processors and Endpoints together, Camel defines multiple DSLs in regular programming languages such as Java, Scala and Groovy. It also allows routing rules to be specified in XML. Here are some DSL examples using different languages and staying functionally equivalent:
In all the above examples we define a routing rule that will load files in the “/tmp” directory into memory, create a new JMS message with the file contents, and send that message to a JMS queue namedaQueue.
So we have several patterns in use here.
1. There are two Message Endpoints; one for FTP connectivity and another for HTTP.
2. Messages from these endpoints are fed into the incomingOrders Message Channel
3. The messages are consumed from the incomingOrders Message Channel and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file.
4. Both Message Translators convert the message content into a POJO, which is fed into the ordersMessage Channel.
The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer.
While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel entirely from Spring. In fact, the entire solution (except for the domain POJO and Maven build script) fits neatly into a single Spring XML file. This is because we are implementing our routing rules using the Spring XML configuration rather than one of Camel’s other DSLs. There are pros and cons to each of course but in this case I mainly wanted to show off some cool tooling that currently works with the XML based routing rules only. The Spring XML file is shown in Listing 2 below.
Listing 2: Complete Spring XML file that configures an embedded ActiveMQ broker and initializes a Camel Context with three routes.
Listing 3: Incoming message formats; XML on the left, CSV on the right.
1. dealing with the specifics of applications and transports, and
2. coming up with good solutions to integration problems.
Making your applications speak transports and APIs is relatively easy on its own. I'm sure everyone knows how to send JMS messages to their broker of choice; though it still requires in depth knowledge of the JMS specification, which many developers may not have. On top of that, what happens when you want to route that JMS message to another application? You then have to take care of mapping the JMS message to the application plus handle any new concepts related to the application. Add a dozen other applications into the mix and you've got quite a headache on your hands.
Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations.
Apache Camel was created with the intention of addressing these two issues. In this article I'll first give you a quick overview of what Camel is. I’ll then guide you through how Camel solves a sample integration problem. Finally I’ll show off some new graphical tooling that makes it even easier to solve integration problems with Camel.
Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations.
Apache Camel was created with the intention of addressing these two issues. In this article I'll first give you a quick overview of what Camel is. I’ll then guide you through how Camel solves a sample integration problem. Finally I’ll show off some new graphical tooling that makes it even easier to solve integration problems with Camel.
What is Camel?
Apache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers. It does this by providing:
• concrete implementations of all the widely used EIPs
• connectivity to a great variety of transports and APIs
• easy to use Domain Specific Languages (DSLs) to wire EIPs and transports together
Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and DSLs. There is of course a lot more going on here under the hood but we'll leave that for another discussion.
Figure 1: High level view of Camel's architecture (from Camel in Action).
Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 13 essential components. There are over 80 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queueaQueue and send them to a file system directory "/tmp", you could use URIs like "jms:aQueue" and "file:/tmp".
Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports over 40 patterns from the EIP book and many other useful Processors.
To wire Processors and Endpoints together, Camel defines multiple DSLs in regular programming languages such as Java, Scala and Groovy. It also allows routing rules to be specified in XML. Here are some DSL examples using different languages and staying functionally equivalent:
• Java DSL
1.from ("file:/tmp").to("jms:aQueue");
• Spring DSL
1.<route>2.<from uri="file:/tmp"/>3.<to uri="jms:aQueue"/>4.</route>
• Scala DSL
1.from "file:/tmp" -> "jms:aQueue"In all the above examples we define a routing rule that will load files in the “/tmp” directory into memory, create a new JMS message with the file contents, and send that message to a JMS queue namedaQueue.
These are the concepts that Camel was built upon. Since then many other interesting features have been added. I recommend reading Camel in Action, by Claus Ibsen and myself to really get the full picture of what Camel can do. But, to get you started, some of these extra features include:
• Pluggable data formats and type converters for easy message transformation between CSV, EDI, Flatpack, HL7, JAXB, JSON, XmlBeans, XStream, Zip, etc.
• Pluggable languages to create expressions or predicates for use in the DSL. Some of these languages include: EL, JXPath, Mvel, OGNL, BeanShell, JavaScript, Groovy, Python, PHP, Ruby, SQL, XPath, XQuery, etc.
• Support for the integration of beans and POJOs in various places in Camel.
• Excellent support for testing distributed and asynchronous systems using a messaging approach
• and much more...
Introducing Rider Auto Parts
The example in this article is based on a fictional motorcycle parts business used throughout theCamel in Action book. The company, named Rider Auto Parts, supplies parts to motorcycle manufacturers. Over the years, they’ve changed the way they receive orders several times. Initially, orders were placed by uploading comma-separated value (CSV) files to an FTP server. The message format was later changed to XML. Currently they provide a website through which orders are submitted as XML messages over HTTP.
Rider Auto Parts asks new customers to use the web interface to place orders, but because of service level agreements (SLAs) with existing customers, they must keep all the old message formats and interfaces up and running. All of these messages are converted to an internal Plain Old Java Object (POJO) format before processing. A high level view of the order processing system is shown in Figure 2.
Figure 2: High level view of order processing at Rider Auto Parts (from Camel in Action).
We need to find out the best way to implement the “Rider order frontend” in Figure 2 above. Let’s first see how this looks using notation from the Enterprise Integration Patterns book.
Solution using EIPs
Rider Auto Parts faces a pretty common problem; over years of operation businesses acquire software baggage in the form of transports/data formats that are popular at the time. Using patterns from the EIP book we can envision the solution as something like Figure 3.
Figure 3: This shows the solution to Rider Auto Parts integration problem using notation from the Enterprise Integration Patterns book.
So we have several patterns in use here.
1. There are two Message Endpoints; one for FTP connectivity and another for HTTP.
2. Messages from these endpoints are fed into the incomingOrders Message Channel
3. The messages are consumed from the incomingOrders Message Channel and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file.
4. Both Message Translators convert the message content into a POJO, which is fed into the ordersMessage Channel.
The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer.
Implementation using Camel
As mentioned before, Camel has a small core set of components included by default. The rest of the components exist as separate modules. In applications that require many types of connectivity it is useful to figure out what Camel modules to include. Listing 1 shows the dependencies using Apache Maven for the Camel implementation of the Rider Auto Parts example. Of course, you don't need to use Apache Maven for dependencies - it is just the easiest way to rapidly add new dependencies to your applications. The list of dependencies includes support for core Camel, ActiveMQ, JAXB marshaling, CSV marshaling, and HTTP. To make the example easier to try out, I've opted to use the File endpoint instead of the FTP. If we were using the FTP endpoint we would need to add a dependency on the camel-ftp module as well.
Listing 1: Maven dependencies for the Camel implementation
Listing 1: Maven dependencies for the Camel implementation
01.<dependencies>02.03.<dependency>04.<groupId>org.apache.camel</groupId>05.<artifactId>camel-core</artifactId>06.<version>${camel-version}</version>07.</dependency>08.<dependency>09.<groupId>org.apache.camel</groupId>10.<artifactId>camel-spring</artifactId>11.<version>${camel-version}</version>12.</dependency>13. 14.15.<dependency>16.<groupId>org.apache.activemq</groupId>17.<artifactId>activemq-core</artifactId>18.<version>${activemq-version}</version>19.</dependency>20.<dependency>21.<groupId>org.apache.xbean</groupId>22.<artifactId>xbean-spring</artifactId>23.<version>${xbean-spring-version}</version>24.</dependency>25. 26.27.<dependency>28.<groupId>org.apache.activemq</groupId>29.<artifactId>activemq-camel</artifactId>30.<version>${activemq-version}</version>31.</dependency>32. 33.34.<dependency>35.<groupId>org.apache.camel</groupId>36.<artifactId>camel-jaxb</artifactId>37.<version>${camel-version}</version>38.</dependency>39. 40.41.<dependency>42.<groupId>org.apache.camel</groupId>43.<artifactId>camel-bindy</artifactId>44.<version>${camel-version}</version>45.</dependency> 46. 47.48.<dependency>49.<groupId>org.apache.camel</groupId>50.<artifactId>camel-jetty</artifactId>51.<version>${camel-version}</version>52.</dependency>53. 54.</dependencies>While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel entirely from Spring. In fact, the entire solution (except for the domain POJO and Maven build script) fits neatly into a single Spring XML file. This is because we are implementing our routing rules using the Spring XML configuration rather than one of Camel’s other DSLs. There are pros and cons to each of course but in this case I mainly wanted to show off some cool tooling that currently works with the XML based routing rules only. The Spring XML file is shown in Listing 2 below.
Listing 2: Complete Spring XML file that configures an embedded ActiveMQ broker and initializes a Camel Context with three routes.
01.<beans xmlns="http://www.springframework.org/schema/beans"02.xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"03.xmlns:amq="http://activemq.apache.org/schema/core"04.xsi:schemaLocation="11. 12.<amq:broker brokerName="localhost" persistent="false" useJmx="false"/>13. 14.<bean id="jms"class="org.apache.activemq.camel.component.ActiveMQComponent">15.<property name="brokerURL" value="vm://localhost" />16.</bean>17. 18.<camelContext xmlns="http://camel.apache.org/schema/spring">19.<route id="FileToJMS">20.<from uri="file:target/placeorder" />21.<to uri="jms:incomingOrders" />22.</route>23. 24.<route id="HTTPtoJMS">25.<from uri="jetty:http://0.0.0.0:8888/placeorder" />26.<inOnly uri="jms:incomingOrders" />27.<transform>28.<constant>OK</constant>29.</transform>30.</route>31. 32.<route id="NormalizeMessageData">33.<from uri="jms:incomingOrders" />34.<convertBodyTo type="java.lang.String" />35.<choice>36.<when>37.<simple>${body} contains '?xml'</simple>38.<unmarshal>39.<jaxb contextPath="org.fusesource.camel" />40.</unmarshal>41.<to uri="jms:orders" />42.</when>43.<otherwise>44.<unmarshal>45.<bindy packages="org.fusesource.camel" type="Csv" />46.</unmarshal>47.<to uri="jms:orders" />48.</otherwise>49.</choice> 50.</route>51.</camelContext>52.</beans>
In this file we first start an embedded Apache ActiveMQ broker and connect Camel to it. Next comes the camelContext element where we define our routing rules. Looking back at Figure 3, we need to receive orders from an FTP (substituted with File in this example) and HTTP endpoint, formatted as shown in Listing 3. In the Spring XML configuration we can specify these incoming endpoints with twofrom elements. The “FileToJMS” and “HTTPtoJMS” routes start off with these from elements. Bothfrom elements are connected to a producer (to and inOnly elements) using the “jms:incomingOrders"URI, which will send the messages to a queue named incomingOrders on the ActiveMQ broker.
Listing 3: Incoming message formats; XML on the left, CSV on the right.
In the case of the HTTP endpoint, there are a couple of extra things to mention. First off the HTTP client will be expecting a response from the application so we have to handle that. In Camel, we have full control over what the client gets back from the HTTP endpoint. Each response is determined by the last method in our current route definition. In our case we use the transform method to set the response to the constant string "OK". Since we handle the response ourselves, we don’t want any response to come from the JMS incomingOrders queue. To send to this queue in a fire-and-forget fashion we use the inOnly element rather than the to element.
The “NormalizeMessageData” route in Listing 2 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First we specify that we want to consume messages from the incomingOrders queue on the ActiveMQ broker. The content based routing of the messages is done with the choice, when, and otherwise elements. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message data format we have we use a Simple expression which checks the message body for the “Simple
The “NormalizeMessageData” route in Listing 2 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First we specify that we want to consume messages from the incomingOrders queue on the ActiveMQ broker. The content based routing of the messages is done with the choice, when, and otherwise elements. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message data format we have we use a Simple expression which checks the message body for the “Simple
To unmarshal the XML and CSV payloads into an Order object, we use Camel’s data formats. Listing 2 shows how these data formats are defined within the unmarshal elements. As shown in Listing 5, theOrder object has JAXB and Camel Bindy annotations to describe the mapping to XML and CSV.
Listing 5: The Order domain class with JAXB/Bindy annotations for easy mapping to and from XML/CSV.
01.@XmlRootElement02.@XmlAccessorType(XmlAccessType.FIELD)03.@CsvRecord(separator = ",", skipFirstLine = true)04.public class Order implements Serializable {05.@XmlAttribute06.@DataField(pos = 1)07.private String name;08. 09.@XmlAttribute10.@DataField(pos = 2)11.private int amount;12. 13.public Order() {14.}15. 16.public Order(String name, int amount) {17.this.name = name;18.this.amount = amount;19.}20.}At this point, successfully normalized messages are sent to the orders queue for processing by some other application at the Rider Auto Parts business.
Implementation using Fuse IDE for Camel
As we’ve seen, Apache Camel certainly makes it easy to create integrations. The DSLs in various languages are designed to be concise, powerful and easy to read – all with the goal of making you, the developer, more productive. With improving productivity in mind, FuseSource has created an IDE for Camel development called Fuse IDE for Camel.
Fuse IDE is an Eclipse-based tool that allows you to drag and drop enterprise integration pattern (EIP) icons onto a canvas and connect them together to form routing rules. The backend of the IDE generates Camel routing rules in Spring XML for you so you don’t ever have to learn the specifics of Camel’s DSLs. Essentially you can create a picture like Figure 3 and have Fuse IDE generate the Camel application for you.
Figure 4 shows what the Fuse IDE designer looks like when we loaded our example configuration from Listing 2.
Figure 4 shows what the Fuse IDE designer looks like when we loaded our example configuration from Listing 2.
Figure 4: Spring XML file in Listing 2 loaded into Fuse IDE’s designer.
From the designer view we can see the graphical representation of each of the routes we created in Spring XML beforehand. From this point we can change how the EIP icons are connected, endpoint URI properties, add additional steps to the routes, and so on. Also, as of version 1.0, Fuse IDE will support all EIPs and components available in Apache Camel – so there is no waiting for version 2.0 to use your favorite Camel features.
In this article we started first by creating the Spring XML configuration by hand. This should not mislead you however - there is no requirement to do this. In fact, since Fuse IDE supports 1-to-1 round tripping between Camel Spring XML configuration and the designer view, you can use whichever you prefer. Figure 5 shows the source view of the Spring XML file. Any changes made to the routes in the source view will be matched when you switch back to the designer view.
Figure 5: Source view in Fuse IDE.
When you have finished constructing your routes, you can then take advantage of another neat feature included in Fuse IDE: that is the ability to generate a JUnit test case from any Camel Spring XML file. Once you are satisfied with your testing, you can then debug in a local Spring container or deploy to your container of choice – all from within the IDE. Of course, much more is planned for Fuse IDE in the future so Camel development will only get easier.
Summary
In this article I've shown two common problems that an integration developer may face: dealing with the specifics of applications and transports, and coming up with good solutions to integration problems. The Apache Camel project provides a nice answer to both of these problems. As the example has shown, solving integration problems with Camel is straight forward and results in very concise route definitions. Building on that we saw how Fuse IDE for Camel makes Camel development even easier.
Links
1. Apache Camel – http://camel.apache.org
2. FuseSource distribution of Apache Camel – http://fusesource.com/products/enterprise-camel
3. Fuse IDE for Camel - http://fusesource.com/ide
4. Camel in Action book - http://www.manning.com/ibsen
5. Jon’s Blog – http://janstey.blogspot.com
6. Article source code - http://repo.fusesource.com/maven2/org/fusesource/examples/rider-auto-spring/2.0/rider-auto-spring-2.0.zip
1 comment:
ok thanks for this post it's quite informative and I have learned new things.
Post a Comment