Why asynchronous processing is the new foundation of Web 2.0

By Dr. Xinyu Liu, JavaWorld.com, 02/19/09

http://www.javaworld.com/javaworld/jw-02-2009/jw-02-servlet3.html

Even as a mid-level API ensconced in modern UI component-based Web frameworks and Web services technologies, the incoming Servlet 3.0 specification (JSR 315) will have groundbreaking impact on Java Web application development. Author Xinyu Liu explains in detail why asynchronous processing is foundational for the collaborative, multi-user applications that define Web 2.0. He also summarizes Servlet 3.0's other enhancements such as ease of configuration and pluggability. Level: Intermediate

The Java Servlet specification is the common denominator for most server-side Java Web technologies, including JavaServer Pages (JSP), JavaServer Faces (JSF), numerous Web frameworks, SOAP and RESTful Web services APIs, and newsfeeds. The servlets running underneath these technologies make them portable across all Java Web servers (servlet containers). Any proposed changes to this widely accepted API for handling HTTP communications will potentially affect all affiliated server-side Web technologies.
The upcoming Servlet 3.0 specification, which passed public review in January 2009, is a major release with important new features that will change the lives of Java Web developers for the better. Here's a list of what you can expect in Servlet 3.0:

• Asynchronous support
• Ease of configuration
• Pluggability
• Enhancements to existing APIs

Asynchronous support is Servlet 3.0's most significant enhancement, intended to make the server-side processing of Ajax applications much more efficient. In this article I'll focus on asynchronous support in Servlet 3.0, starting by explaining the connection and thread-consumption issues that underlie the need for asynchronous support. I'll then explain how real-world applications today utilize asynchronous processing in server push implementations like Comet, or reverse Ajax. Finally, I'll touch on Servlet 3.0's other enhancements such as pluggability and ease of configuration, leaving you with a good impression of Servlet 3.0 and its impact on Java Web development.

Asynchronous support: Background concepts

Web 2.0 technologies drastically change the traffic profile between Web clients (such as browsers) and Web servers. Asynchronous support introduced in Servlet 3.0 is designed to respond to this new challenge. In order to understand the importance of asynchronous processing, let's first consider the evolution of HTTP communications.

HTTP 1.0 to HTTP 1.1

A major improvement in the HTTP 1.1 standard is persistent connections. In HTTP 1.0, a connection between a Web client and server is closed after a single request/response cycle. In HTTP 1.1, a connection is kept alive and reused for multiple requests. Persistent connections reduce communication lag perceptibly, because the client doesn't need to renegotiate the TCP connection after each request.

Thread per connection

Figuring out how to make Web servers more scalable is an ongoing challenge for vendors. Thread per HTTP connection, which is based on HTTP 1.1's persistent connections, is a common solution vendors have adopted. Under this strategy, each HTTP connection between client and server is associated with one thread on the server side. Threads are allocated from a server-managed thread pool. Once a connection is closed, the dedicated thread is recycled back to the pool and is ready to serve other tasks. Depending on the hardware configuration, this approach can scale to a high number of concurrent connections. Experiments with high-profile Web servers have yielded numerical results revealing that memory consumption increases almost in direct proportion with the number of HTTP connections. The reason is that threads are relatively expensive in terms of memory use. Servers configured with a fixed number of threads can suffer the thread starvation problem, whereby requests from new clients are rejected once all the threads in the pool are taken.
On the other hand, for many Web sites, users request pages from the server only sporadically. This is known as a page-by-page model. The connection threads are idling most of the time, which is a waste of resources.

Thread per request

Thanks to the non-blocking I/O capability introduced in Java 4's New I/O APIs for the Java Platform (NIO) package, a persistent HTTP connection doesn't require that a thread be constantly attached to it. Threads can be allocated to connections only when requests are being processed. When a connection is idle between requests, the thread can be recycled, and the connection is placed in a centralized NIO select set to detect new requests without consuming a separate thread. This model, called thread per request, potentially allows Web servers to handle a growing number of user connections with a fixed number of threads. With the same hardware configuration, Web servers running in this mode scale much better than in the thread-per-connection mode. Today, popular Web servers -- including Tomcat, Jetty, GlassFish (Grizzly), WebLogic, and WebSphere -- all use thread per request through Java NIO. For application developers, the good news is that Web servers implement non-blocking I/O in a hidden manner, with no exposure whatsoever to applications through servlet APIs.

Meeting Ajax challenges

To offer a richer user experience with more-responsive interfaces, more and more Web applications use Ajax. Users of Ajax applications interact with the Web server much more frequently than in the page-by-page model. Unlike ordinary user requests, Ajax requests can be sent frequently by one client to the server. In addition, both the client and scripts running on the client side can poll the Web server regularly for updates. More simultaneous requests cause more threads to be consumed, which cancels out the benefit of the thread-per-request approach to a high degree.

Slow running, limited resources

Some slow-running back-end routines worsen the situation. For example, a request could be blocked by a depleted JDBC connection pool, or a low-throughput Web service endpoint. Until the resource becomes available, the thread could be stuck with the pending request for a long time. It would be better to place the request in a centralized queue waiting for available resources and recycle that thread. This effectively throttles the number of request threads to match the capacity of the slow-running back-end routines. It also suggests that at a certain point during request processing (when the request is stored in the queue), no threads are consumed for the request at all. Asynchronous support in Servlet 3.0 is designed to achieve this scenario through a universal and portable approach, whether Ajax is used or not. Listing 1 shows you how it works.

Listing 1. Throttling access to resources

@WebServlet(name="myServlet", urlPatterns={"/slowprocess"}, asyncSupported=true)
public class MyServlet extends HttpServlet {
   
    public void doGet(HttpServletRequest request, HttpServletResponse response) {
        AsyncContext aCtx = request.startAsync(request, response); 
        ServletContext appScope = request.getServletContext();
        ((Queue)appScope.getAttribute("slowWebServiceJobQueue")).add(aCtx);
    }
}

@WebServletContextListener
public class SlowWebService implements ServletContextListener {

    public void contextInitialized(ServletContextEvent sce) {
        Queue jobQueue = new ConcurrentLinkedQueue();
        sce.getServletContext().setAttribute("slowWebServiceJobQueue", jobQueue);
        // pool size matching Web services capacity
        Executor executor = Executors.newFixedThreadPool(10);
        while(true)
        {
            if(!jobQueue.isEmpty())
            {
                final AsyncContext aCtx = jobQueue.poll();
                executor.execute(new Runnable(){
                    public void run() {
                        ServletRequest request = aCtx.getRequest();
                        // get parameteres
                        // invoke a Web service endpoint
                        // set results
                        aCtx.forward("/result.jsp");
                    }                    
                });             
            }
        }
    }
    
    public void contextDestroyed(ServletContextEvent sce) {
    }
}

When the asyncSupported attribute is set to true, the response object is not committed on method exit. CallingstartAsync() returns an AsyncContext object that caches the request/response object pair. The AsyncContext object is then stored in an application-scoped queue. Without any delay, the doGet() method returns, and the original request thread is recycled. In the ServletContextListener object, separate threads initiated during application launch monitor the queue and resume request processing whenever the resources become available. After a request is processed, you have the option of calling ServletResponse.getWriter().print(...), and then complete() to commit the response, or callingforward() to direct the flow to a JSP page to be displayed as the result. Note that JSP pages are servlets with anasyncSupported attribute that defaults to false.
In addition, the AsyncEvent and AsynListener classes in Servlet 3.0 give developers elaborate control of asynchronous lifecycle events. You can register an AsynListener through the ServletRequest.addAsyncListener() method. After thestartAsync() method is called on the request, an AsyncEvent is sent to the registered AsyncListener as soon as the asynchronous operation has completed or timed out. The AsyncEvent also contains the same request and response objects as in the AsyncContext object.

Server push

A more interesting and vital use case for the Servlet 3.0 asynchronous feature is server push. GTalk, a widget that lets GMail users chat online, is an example of server push. GTalk doesn't poll the server frequently to check if a new message is available to display. Instead it waits for the server to push back new messages. This approach has two obvious advantages: low-lag communication without requests being sent, and no waste of server resources and network bandwidth.
Ajax allows a user to interact with a page even if other requests from the same user are being processed at the same time. A common use case is to have a browser regularly poll the server for updates of state changes without interrupting the user. However, high polling frequencies waste server resources and network bandwidth. If the server could actively push data to browsers -- in other words, deliver asynchronous messages to clients on events (state changes) -- Ajax applications would perform better and save precious server and network resources.
The HTTP protocol is a request/response protocol. A client sends a request message to a server, and the server replies with a response message. The server can't initiate a connection with a client or send an unexpected message to the client. This aspect of the HTTP protocol seemingly makes server push impossible. But several ingenious techniques have been devised to circumvent this constraint:

• Service streaming (streaming) allows a server to send a message to a client when an event occurs, without an explicit request from the client. In real-world implementations, the client initiates a connection to the server through a request, and the response returns bits and pieces each time a server-side event occurs; the response lasts (theoretically) forever. Those bits and pieces can be interpreted by client-side JavaScript and displayed through the browser's incremental rendering ability.
• Long polling, also known as asynchronous polling, is a hybrid of pure server push and client pull. It is based on the Bayeux protocol, which uses a topic-based publish-subscribe scheme. As in streaming, a client subscribes to a connection channel on the server by sending a request. The server holds the request and waits for an event to happen. Once the event occurs (or after a predefined timeout), a complete response message is sent to the client. Upon receiving the response, the client immediately sends a new request. The server, then, almost always has an outstanding request that it can use to deliver data in response to a server-side event. Long polling is relatively easier to implement on the browser side than streaming.
• Passive piggyback: When the server has an update to send, it waits for the next time the browser makes a request and then sends its update along with the response that the browser was expecting.

Service streaming and long polling, implemented with Ajax, are known as Comet, or reverse Ajax. (Some developers call all interactive techniques reverse Ajax, including regular polling, Comet, and piggyback.)
Ajax improves single-user responsiveness. Server-push technologies like Comet improve application responsiveness for collaborative, multi-user applications without the overhead of regular polling.
The client aspect of the server push techniques -- such as hidden iframes, XMLHttpRequest streaming, and some Dojo and jQuery libraries that facilitate asynchronous communication -- are outside this article's scope. Instead, our interest is on the server side, specifically how the Servlet 3.0 specification helps implement interactive applications with server push.

Efficient thread use: A universal solution

Comet (streaming and long polling) always occupies a channel waiting on the server to push back state changes. The same channel can't be reused by the client to send ordinary requests. So Comet applications typically work with two connections per client. If the application stays in thread-per-request mode, an outstanding thread will be associated with each Comet connection, resulting in more threads than the number of users. Comet applications that use the thread-per-request approach can scale only at the prohibitive expense of huge thread consumption.
Lack of asynchronous support in the Servlet 2.5 specification has caused server vendors to devise workarounds by weaving proprietary classes and interfaces that promote scalable Comet programming into the Servlet 2.5 API. For example, Tomcat has a CometProcessor class, Jetty 6 has Continuations, and Grizzly has a CometEngine class. With Servlet 3.0, you no longer need to sacrifice portability when programming scalable Comet applications. The asynchronous feature shown in Listing 1 consolidates great design ideas from multiple vendors and presents a universal, portable solution for efficient thread usage in Comet applications.
Let's look at a real-world example -- an online auction Web site. When buyers bid for an item online, it annoys them that they must keep refreshing the page to see the latest/highest bid. This is a hassle especially when the bidding is about to end. With server-push technologies, no page refresh or high-frequency Ajax polling is needed. The server simply pushes the item's bid price to the registered clients whenever a new bid is placed, as shown in Listing 2.

Listing 2. Auction watching

@WebServlet(name="myServlet", urlPatterns={"/auctionservice"}, asyncSupported=true)
public class MyServlet extends HttpServlet {
   
   // track bid prices
   public void doGet(HttpServletRequest request, HttpServletResponse response) {
      AsyncContext aCtx = request.startAsync(request, response); 
      // This could be a cluser-wide cache.
      ServletContext appScope = request.getServletContext();    
      Map> aucWatchers = (Map>)appScope.getAttribute("aucWatchers");
      List watchers = (List)aucWatchers.get(request.getParameter("auctionId"));
      watchers.add(aCtx); // register a watcher
   }

   // place a bid
   public void doPost(HttpServletRequest request, HttpServletResponse response) {
      // run in a transactional context 
      // save a new bid
      AsyncContext aCtx = request.startAsync(request, response); 
      ServletContext appScope = request.getServletContext(); 
      Queue aucBids = (Queue)appScope.getAttribute("aucBids");
      aucBids.add((Bid)request.getAttribute("bid"));  // a new bid event is placed queued.  
   }
}

@WebServletContextListener
public class BidPushService implements ServletContextListener{

   public void contextInitialized(ServletContextEvent sce) {   
      Map> aucWatchers = new HashMap>();
      sce.getServletContext().setAttribute("aucWatchers", aucWatchers);
      // store new bids not published yet
      Queue aucBids = new ConcurrentLinkedQueue();
      sce.getServletContext().setAttribute("aucBids", aucBids);
        
      Executor bidExecutor = Executors.newCachedThreadPool(); 
      final Executor watcherExecutor = Executors.newCachedThreadPool();
      while(true)
      {        
         if(!aucBids.isEmpty()) // There are unpublished new bid events.
         {
            final Bid bid = aucBids.poll();
            bidExecutor.execute(new Runnable(){
               public void run() {
                  List watchers = aucWatchers.get(bid.getAuctionId()); 
                  for(final AsyncContext aCtx : watchers)
                  {
                     watcherExecutor.execute(new Runnable(){
                        public void run() {
                           // publish a new bid event to a watcher
                           aCtx.getResponse().getWriter().print("A new bid on the item was placed. The current price ..., next bid price is ...");
                        };
                     });
                  }                           
               }
            });
         }
      }
   }
    
   public void contextDestroyed(ServletContextEvent sce) {
   }
}

Listing 2 is quite self-explanatory. For all the auction watchers, the uncommitted, long-lived response objects are stored in theAsynchContext objects, which are cached in an application-scoped Map. The original threads initiating the request/response loop are then returned to the thread pool and ready to serve other tasks. Whenever a new bid is placed, threads initiated in theServletContextListener object pick it up from a queue, and then push the event to all the watchers (stored response objects) registered for the same auction. The new bid data is streamed to the connected browsers without being explicitly requested by the clients. These response objects are either uncommitted for streaming, or committed for long polling. As you can see, no threads are consumed for watchers when no bid event is happening on the sever side.
New portable custom UI components (widgets) with Comet ability can be delivered under frameworks such as JSF and Wicket. With annotations and pluggability in Servlet 3.0 -- features I'll discuss in the next section -- these widgets can be used off the shelf without any configuration. For application developers, this surely is great news.

Other enhancements in Servlet 3.0

In addition to asynchronous support, the Servlet 3.0 specification includes enhancements designed to make application configuration and development easier.

Ease of configuration

With Servlet 3.0, you now have three options for configuring a Java Web application: annotations, the API, and XML. Annotations, introduced in Java 5, are widely accepted as a metadata facility at the Java source-code level. With annotations, the need for XML configuration is drastically lowered. The new specification introduces several handy annotations. The@WebServlet, @ServletFilter, and @WebServletContextListener annotations are equivalent to their corresponding tags in the web.xml deployment descriptor file.

Complementary techniques

A general rule that applies to various Java technologies is that configurations set programmatically take precedence over values in XML, and that XML always overrides annotations. I think of XML and annotations as complementary techniques. For relatively static settings closely related to specific Java classes, methods, properties, I would use source-level annotations. I prefer to use XML for more volatile, global settings not tied to Java source code to avoid recompilation on changes, or when I don't have access to the Java source code (because classes are wrapped in a JAR file), or when I need to override the settings defined in annotations.

At the same time, the specification augments the ServletContext class with a few methods --ServletContext.addServletMapping(...) and ServletContext.addFilter(...), for example -- that allow you to alter configurations programmatically. The availability of annotations and these new methods as configuration options means aweb.xml file is no longer required in a Java Web archive (WAR) file.

Pluggability

Regardless of the framework you use now for Web development, you're always required to add several technology-mandated servlets or filters with specific settings in your Web application's web.xml file. Even worse, you must add yet more servlets, filters, and parameters if you want to incorporate technology extension packages offering custom GUI widgets or security features (such as Spring Security). Your web.xml file keeps growing and gradually becomes a maintenance hassle.
Although annotations in Servlet 3.0 make a web.xml file optional, XML sometimes is still desirable for cases such as incremental upgrades, to override either the default attribute values or the values set via annotations. To improve pluggability for frameworks and component libraries, Servlet 3.0 introduces a new XML  tag for specifying a Web fragment (a list of components described as a partial web.xml file ) that can be included in the META-INF directory of a library or framework's JAR file. Servlet containers will pick up and merge the fragment declarations when the JAR file is loaded and scanned as part of the entire Web application.

Enhancements to existing APIs

The Servlet 3.0 specification makes a number of adjustments to servlet APIs. For example, a getServletContext method is added to the ServletRequest class as a convenience method. The new version of the Cookie class now supports HTTPOnlycookies, which are not exposed to client-side scripting code, to mitigate cross-site scripting attacks.
A few methods for programmatic login/logout proposed as a security enhancement in an early draft of the Servlet 3.0 specification were removed. It's also worth mentioning that the JSR expert group tried to make the new servlet APIs more POJO-like in an early draft, but they ultimately gave up. After all, making mid-to-low level APIs POJO-like is unimportant because the API has minimal exposure to application developers.

In conclusion

Servlet 3.0 is a major specification release. The final version will be included in Java EE 6, which is likely to debut in the spring of 2009. The API enhancements, annotation-based configuration, and pluggability features in Servlet 3.0 will certainly make Web application development much easier. Most important, the long-desired asynchronous feature unifies the APIs for implementing server-push technologies such as Comet, as well as resource throttling. Comet applications built on top of the Servlet 3.0 specification will be more portable and scalable than ever.

About the author

Dr. Xinyu Liu is a Sun Microsystems certified enterprise architect working in a healthcare corporation and an IT consulting firm.
Read more about Enterprise Java in JavaWorld's Enterprise Java section.
All contents copyright 1995-2012 Java World, Inc. http://www.javaworld.com