Open Source Java and Web Testing Tools
Just like writing an article, testing is a skill. While this may come as a surprise to some people, it is a simple fact. For any system, there are several possible test cases, and yet, invariably, we have time to run only a few of them, which in turn are expected to find most of the defects in the software. We know from experience that this is simply not possible. Therefore, in order to increase the chances of finding defects, testing tools may be used. There are numerous testing tools, each with specific capabilities and test objectives.
The selection of the best testing tool for a particular development environment is a critical success factor for the testing activities. A testing tool should be considered based on the test objectives. As a general guideline, one must investigate the appropriateness of a testing tool when the manual process is inadequate. For example, if a system needs to be stress tested, a group of testers could simultaneously log on to the system and attempt to simulate peak loads, but this approach is not a practical solution, as it has many limitations. The performance cannot be measured precisely or repeatedly. Hence, a suitable load testing tool can be used to simulate several virtual users under controlled stress conditions to measure the performance more precisely.
Using testing tools is also a skill, but a very different skill from testing. Many organizations are surprised to find that it is more expensive to do testing using tools. In order to gain benefits from testing tools, careful thought must be given for which tests you want to use tools and to the tool being chosen. A guideline on when and how to consider using a testing tool is beyond the scope of this article, but I am considering taking this up in a future article.
One of the chief concerns while choosing a testing tool is the cost of the tool itself. Since software testing usually attracts the least budget in an organization, many people look to the Open Source community for good solutions.
The objective of this article is to provide an overview of some test tools hosted by SourceForge and demonstrate how they can improve the quality and productivity of a development effort. This article will focus on testing tools that can be used for applications written in Java and Web-based applications.
Java Application Testing Tools
One of the most widely-used tools is Abbot, a framework for testing Java GUIs. Using simple XML-based scripts, we can launch a GUI, play back arbitrary user actions on it, and examine its state. It also includes a script editor (Costello) that records user actions. The framework may be invoked directly from Java code (JUnit) or accessed without programming through the use of scripts. It is suitable for use both by developers for unit tests and QA for functional testing, especially with JFC swing. This tool has an excellent development team, which constantly updates the software.
DejaGnu is a framework for testing applications such as Tcl, C, C++, Java, and network applications, and cross testing embedded systems. Its purpose is to provide a single frontend for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A "test harness" is the testing infrastructure that is created to support a specific program or tool. Each program can have multiple test suites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl, Tool Command Language. There has not been a new release in quite a while, although the snapshots are quite stable.
The JavaTest™ harness is a general purpose Open Source Java API for creating Java test software, a fully-featured, flexible, and configurable test harness very well suited for most types of unit testing, especially Technology Compatibility Kit (TCK) test suites. The JavaTest harness is an excellent tool for configuring, sequencing, and running test suites that consist of large numbers of discrete, independent tests. It is especially good at testing APIs and compilers. Originally developed as a test harness to run the JCK test suite, it has since evolved into a general-purpose test platform well suited for use with all TCK test suites.
JFunc is an extension to the JUnit testing framework to make it easier for use with functional tests. Functional testing (also called integration testing) significantly differs from unit testing in a number of respects. Part of this project is dedicated towards putting together code to address these differences; the other part of this project is putting together methodologies for functional testing.
Marathon is a general-purpose tool for both running and authoring acceptance tests geared at the Java Platform Version 1.3 or later. Included with marathon is a rich suite of components to help you interact with your application at the User Interface Level (GUI). To aid with the regression testing of existing applications, Marathon comes bundled with a recorder to capture events as you use and interact with your application. These events are then converted into a valid Marathon test, which can subsequently be played back.
Pounder is a utility for testing Java GUIs. It allows developers to dynamically load components, record scripts, and then use those scripts in JUnit. It supports custom components, drag-and-drop, and the examination of test runs in source.
QAT was developed to ease the issues encountered by having to perform Quality Assurance tests across a variety of hardware and software combinations. The way to achieve this is to abstract the process of running the test on a machine. So, from a tester's point of view, there is one test suite, which the tester executes, and then gets the results for further analysis. The specifics of the machine on which the test will be run is completely hidden, and is necessary for running the test.
The QAT tool can be divided into two main sections, the Agent, responsible for actually running each test or group of tests, and the Harness, which is responsible for test selection, management, results, and agent co-ordination. It can be accessed via a browser interface if remote access to the harness is required.
First, each test case is written in a platform-independent way in a simple scripting language, very similar to C shell, although it could be implemented as any language you want. All references to platform-specific items are delayed by referring to one or more configuration files. A configuration is then built for each test based on the architecture and operating system of the target machine. So, instead of directly executing a command within the test, we refer to variables, which will only be defined at the time of running the test.
Once the test case is ready, the test is transferred to whatever machine we want to run it on (agent). This is achieved by developing a relatively stupid agent that can respond to a limited set of requests from the harness via TCP sockets. Some examples are GETFILE, SENDFILE, DELFILE, EXECUTE, etc. The test script running on the harness may then use this command set to build a desired environment on the agent. We are able to transfer files, unzip files, delete files, set environment variables, and any of the other good things we need to run a test. Once the test is run, a result is decided on and sent back to the harness for processing.
Since the agent is written in Java, it runs on any architecture supporting a JVM, including restricted ones running pJava, kJava, and various other slimmed-down versions.
Web Application Testing Tools
Anteater is a testing framework designed around Ant, from the Apache Jakarta Project. It is basically a set of Ant tasks for the functional testing of Web sites and Web services (functional testing being: hit a URL and ensure the response meets certain criteria). One can test HTTP parameters, response codes, XPath, regexp, and Relax NG expressions. Anteater also includes HTML reporting (based on junitreport) and a hierarchical grouping system for quickly configuring large test scripts. When a Web request is received, Anteater can check the parameters of the request and send a response accordingly. This makes it useful for testing SOAP and XML applications.
The ability to wait for incoming HTTP messages is something unique to Anteater, which makes it especially useful when building tests for applications that use high level SOAP-based communication, like ebXML or BizTalk. Applications written using these protocols usually receive SOAP messages and send back a meaningless response. It is only later that they inform the client, using an HTTP request on the client, about the results of the processing. These are the so-called asynchronous SOAP messages, and are the heart of many high-level protocols based on SOAP or XML messages.
Written in Java, HttpUnit emulates the relevant portions of browser behavior, including form submission, Javascript, basic HTTP authentication, cookies, and automatic page redirection, and allows Java test code to examine returned pages either as text, an XML DOM, or containers of forms, tables, and links.
jWebUnit is a Java framework which facilitates creation of acceptance tests for Web applications. It provides a high-level API for navigating a Web application combined with a set of assertions to verify the application's correctness. This includes navigation via links, form entry and submission, validation of table contents, and other typical business Web application features. It utilizes HttpUnit behind the scenes. The simple navigation methods and ready-to-use assertions allow for more rapid test creation than using only JUnit and HttpUnit.
Bugkilla is a tool set to create, maintain, execute, and analyze functional system tests of Web applications. Specification and execution of tests is automated for both the Web frontend and business logic layers. One goal is to integrate with existing frameworks and tools (an Eclipse Plugin exists)
The Grinder, a Java load testing framework freely available under a BSD-style Open Source license, makes it easy to orchestrate the activities of a test script in many processes across many machines, using a graphical console application. Test scripts make use of client code embodied in Java plugins. Most users of The Grinder do not write plugins themselves; they use one of the supplied plugins. The Grinder comes with a mature plugin for testing HTTP services, as well as a tool which allows HTTP scripts to be automatically recorded.
Jameleon is an automated testing tool that separates applications into features and allows those features to be tied together independently, creating test cases. These test cases can then be data-driven and executed against different environments. Jameleon breaks applications into features and allows testing at any level, simply by passing in different data for the same test. Because Jameleon is based on Java and XML, there is no need to learn a proprietary technology.
It's an acceptance testing tool for testing the functionality provided by applications, and currently supports the testing of Web applications. It differs from regular HttpUnit and jWebUnit in that it separates testing of features from the actual test cases themselves. If I understand it correctly, you write the feature tests separately and then script them together into a reusable test case. Incidentally, you can also make these test cases data-driven, which gives an easy way of running specific tests on specific environments.
The framework has a plugin architecture, allowing different functional testing tools to be used, and there is a plugin for testing Web applications using HttpUnit/jWebUnit. The test case scripting is done with XML and Jelly.
Jameleon combines XDoclet, Ant and Jelly to provide a potentially powerful framework for solid functional testing of your Webapp. It strikes a good balance between scripting and coding, and allows you to set up multiple inputs per test by providing input via CSV files. Along with the flexibility come a complexity and maintenance overhead, but you are getting your Webapp tested for you.
LogiTest is the core application in the LogiTest suite. LogiTest is designed to aid in the testing of Web site functionality. It currently supports HTTP and HTTPS protocols, GET and POST methods, multiple document views, custom headers, and more. The LogiTest application provides a simple graphical user interface for creating and playing back tests for testing Internet-based applications.
Solex is a set of Eclipse plugins providing non-regression and stress tests of Web application servers. Test scripts are recorded from Internet browsers, thanks to a built-in Web proxy. For some Web applications, a request depends on a previous server's response. To address such a requirement, Solex introduces the concept of extraction and replacement rules. An extraction rule tied to an HTTP message's content will bind an extracted value with a variable. A replacement rule will replace any part of an HTTP message with variable content.
The tool therefore provides an easy way to extract URL parameters, Header values, or any part of a request or a response, bind their values with variables, and then replace URL parameters, Header values, or any part of a request with the variable content. The user has the ability to add assertions for each response. Once a response has been received, all assertions of this response will be called to ensure that it is valid. If not, the playback process is stopped. Several kinds of rules and assertions are provided. The most complicated ones support regular expressions and XPath.
Tclwebtest is a tool for writing automated tests of Web applications in Tcl. It implements some basic HTML parsing functionality to provide comfortable commands for operations on the HTML elements (most importantly forms) of the result pages.
TagUnit is a framework through which custom tags can be tested inside the container and in isolation from the pages on which they will ultimately be used. In essence, it's a tag library for testing tags within JSP pages. This means that it is easy to unit test tags, including the content that they generate and the side effects that they have on the environment, such as the introduction of scripting variables, page context attributes, cookies, etc.
Web Form Flooder is a Java console utility that analyzes a Web page, completes any forms present on the page with reasonable data, and submits the data. It crawls links within the site in order to identify and flood additional forms that may be present. It is great for load testing of Web forms, checking that all links work and that forms submit correctly.
XmlTestSuite provides a powerful way to test Web applications. Writing tests requires only knowledge of HTML and XML. The authors want XmlTestSuite to be adopted by testers, business analysts, and Web developers who don't have a Java background. XmlTestSuite supports "test-driven development". It lets you separate page structure from tests and test data. It can also verify databases. It's like JWebUnit, but has simple XML test definitions and reusable pages. The problems with raw HTTPUnit or JWebUnit are that:
- It's very hard to get non-programmers to write tests.
- Tests are so ugly you can't read them. (Trust me; HttpUnit test classes are a nightmare to maintain.)
- Web tests generally change far more often than unit tests, and so need to be altered, but your refactoring won't change them automatically (i.e., changing a JSP in IDEA will not cascade to the test like changing a class will).
Conclusions
When trying to choose Open Source test tools, several factors may be considered, such as how active the development is, the support for the tool, the quality of the documentation, etc. The table below will give an idea of the development activity, such as the current version as of September 2004, support via mailing list and public forums, documentation quality, etc. of the tools listed above.
Tool
|
Current Version
|
Date Released
|
Public Forums
|
Mailing Lists
|
CV Tracker
|
Documentation
|
Abbot
|
0.13.0
|
15/08/04
|
Very Active
|
2
|
Very Active
|
Excellent
|
DejaGNU
|
1.4.3
|
23/09/02
|
Fair
|
3
|
Fair
|
Good
|
JavaTest
|
0.00.08
|
23/01/03
|
Not Active
|
3
|
Not Active
|
None
|
Jfunc
|
1.1
|
13/05/02
|
Active
|
0
|
Active
|
Limited
|
Marathon
|
0.83
|
31/08/04
|
Very Active
|
3
|
Very Active
|
Excellent
|
Pounder
|
0.95
|
28/09/02
|
Not Active
|
0
|
Not Active
|
Good
|
QAT
|
2.7.2
|
03/05/04
|
Active
|
1
|
Active
|
Very Good
|
Tool
|
Current Version
|
Date Released
|
Public Forums
|
Mailing Lists
|
CV Tracker
|
Documentation
|
Anteater
|
0.9.16
|
12/04/03
|
Good
|
3
|
Very Active
|
Excellent
|
HttpUnit
|
1.5.4
|
20/08/03
|
Good
|
2
|
Active
|
Good
|
JwebUnit
|
1.2
|
12/06/04
|
Not Active
|
2
|
Fair
|
Good
|
Bugkilla
|
0-1-0
|
26/08/04
|
Active
|
2
|
Active
|
Very Good
|
Grinder
|
3.0-beta 22
|
05/07/04
|
Not Active
|
3
|
Not Active
|
Excellent
|
Jameleon
|
2.0.2
|
26/07/04
|
Very Active
|
3
|
Active
|
Excellent
|
LogiTest
|
0.8.1
|
13/06/01
|
Not Active
|
2
|
Not Active
|
Good
|
Solex
|
0.5.0
|
01/08/04
|
Active
|
1
|
Active
|
Very Good
|
Tclwebtest
|
1.0
|
13/05/04
|
Active
|
1
|
Active
|
Good
|
TagUnit
|
1.0.1
|
10/03/04
|
Not Active
|
2
|
Active
|
Very Good
|
Web Form Flooder
|
0.2.8
|
20/12/03
|
Active
|
0
|
Active
|
Good
|
XmlTestSuite
|
1.2
|
06/05/03
|
Not Active
|
0
|
Not Active
|
Good
|
The tools mentioned are not the definitive list of tools available, but just a few of the most popular ones hosted on SourceForge. In order to keep this article short and readable, I have focused on testing tools for Java and Web applications. I am soon planning to write a continuation to this article which will have details on testing tools for other applications like databases, .NET, etc., as well as stress, performance, regression, etc. testing tools hosted on SourceForge.
No comments:
Post a Comment