Introduction

With the launch of Visual Studio .NET many companies are embracing the new technology and already developing XML web services. It’s high time the software testing community engaged the .NET technology by acquiring the skills and resources necessary to test these fast emerging web applications. Testing is essential to achieving the goal of scalable, robust and successful web services. Faced with testing a web service, software testers need to know where to look for functional and performance-related problems in order to design test cases specific to the task. This White Paper will investigate how XML web services are implemented, consider the customizable features of web services and look at load testing a web service, specifically so that test strategies can be formulated. It is intended for testers who don’t want to be left behind and are already learning about .NET development and creating best practices for web services testing. It assumes knowledge of software testing practices.

What is a web service?

A web service is a web application which programmatically returns information to clients who request it. Data is exchanged with the web service using XML, regardless of the operating system or the programming language. The idea is that web services are integrated into other applications or websites, even though they exist on other servers. So for example, a website providing quotes for car insurance could make requests behind the scenes to a web service to get the estimated value of a particular car model and to another web service to get the current interest rate.

Companies exposing their information as web services will probably be charging for their use. What’s more, it is likely that web services are used in mission critical roles, therefore performance matters. Consumers of web services will want assurances that a web service won’t fail to return a response in a certain time period.

Basic functional testing

Despite web services not having a user interface, it is still possible to invoke web methods (or individual operations within a web service) for basic functional testing. The .NET framework is equipped with a test page for ASP.NET web services, allowing web methods to be called via a browser. This enables straightforward testing by permitting requests to be made to the web service and returning the XML response. For that reason it is the initial point to check for bugs in a web service and to establish if it is essentially functionally correct.

Figure 1 – Test page to test a web service

This test harness relies on HTTP GET methods to call the web service, whereas in production most web services will be invoked directly using the Simple Object Access Protocol (SOAP). Consequently, to test web services you must be able to write scripts in a programming language, calling the web service using SOAP in exactly the same way as a client application would. This knowledge will form the basis for writing automated testing scripts, described in the load testing section of this White Paper.

Adding a web reference and creating a proxy class

For a Visual Studio .NET application (or indeed a test script) to call a web service, a web reference must first be added, giving the address of the web service. This then allows you to call the web methods as if they were local to your machine. Behind the scenes Visual Studio .NET uses the wsdl.exe tool to build a proxy class (or proxy wrapper), which is responsible for serialising (or converting) the request to the web service into SOAP and de-serialising the SOAP response. For more information on writing a web service client, see the Visual Studio .NET online help: http://msdn.microsoft.com/vstudio/default.aspx.

Figure 2 – The proxy class created when a web reference is added

Your web service expects a SOAP request and once it’s been processed it returns a SOAP response, which is then de-serialised by the proxy class so that the application can go on to use the XML data returned. SOAP requests and responses are called SOAP messages.

Catching the SOAP

A SOAP message is comprised of three major components: a SOAP envelope, a SOAP header and a SOAP body. Trapping the actual SOAP messages sent in the request and response is a particularly useful debugging and testing device. Additional information other than in the method call can be sent to the web service in the SOAP header, so it is worth checking exactly what data is going to and from the web service. You may also get more ideas for test cases when you analyse the contents of a typical SOAP request and response to your web service.

One way of catching the SOAP messages is to use SOAP extensions. A SOAP extension is piece of code conventionally used to modify the contents of the SOAP message, such as to encrypt or decrypt the method calls, or compress and decompress the SOAP messages. You could write a bit of programming, to apply a new SOAP extension to a web service to catch the SOAP message and log it to a text file for testing analysis.

It is important to know how a web service is called, what information is being sent in the request and what is returned. Equally testers should take into account the web service design to make sure the test process concentrates on any design weaknesses and aids the development decisions.

The web service design

A web service is a remote application which will be accessed by multiple users. This means that response times and scalability are likely to be the issues foremost in the testing effort. By considering the design features which impact performance and carrying out proof-of-concept tests (cut-down load tests, for example) on early versions of the web service, you can certify the chosen architecture is the correct one.

If you are lucky enough to be involved in the design stage of the web service, it would be worth taking a careful look at whether there is a convincing reason for a web service to use user-defined data types, other than the simple SOAP data types. The data sent or returned could be in the form of a dataset, an array, a string, or integer for example. Alternatively, the web service might contain custom data types for example, a Customer class or OrderInfo class. One side effect of user-defined types is that the client needs to understand the custom types to be able to make use of the web service, which makes it less accessible in the first place. What’s more, analysis of user-defined data types could reveal inaccuracies and performance implications. For every new data type the proxy class has more work to do when it serialises and de-serialises the SOAP message and therefore could contribute to slower response times. We strongly recommend avoiding the use of custom data types in web services.

You should test for efficient use of web methods when examining the web service architecture. A web service can have any number of web methods but more may not be merrier. Often, many web methods can be combined into a single web method in the interest of performance. If a frequently used scenario for your web service requires the same subset of web methods to be called, it could be worth amalgamating the individual web methods. For example, the website providing car insurance quotes may make 2 individual requests to the same web service, passing in the car registration number each time and getting responses for the age of the car and also whose name the car is currently registered to. Since both these pieces of information are requested every time a quote is calculated, a better design would be to combine the web methods so that a single request is made and response includes both pieces of information. This improves the server’s performance by reducing the overhead associated with processing requests along with reducing the overhead for the client processing the responses. The client also benefits since it is more efficient to send and receive a block of data in one go (due to the latency of the network) as opposed to numerous smaller blocks of data. Remember that speed is key to the success of a web service.

Like all applications, the functional testing priorities for a web service should focus on the functionality most complicated to implement and the functionality most critical to its purpose. A good place to look for code that may fall into both these categories is where developers have chosen to override classes provided by the .NET framework with their own classes. If developers are restricted by the standard classes they may write their own, extending the implementation for their specific development scenario, in which case the code could be complex and therefore likely to be error prone.

Properties of a web service

Buffer the response

Keeping the communication to and from the client to a minimum is a factor when reviewing the performance of a web service. Consider then the BufferReponse property. By default the BufferResponse property is set to “true”, which ensures the web method sends the entire response back in one go, as opposed to separate chunks of data. If large amounts of data are being returned, a web method can be customised so that the BufferResponse is false.

[WebMethod(BufferResponse= false)]

The response is then returned as it is serialised, reducing the amount of memory consumed in the process. However, the BufferResponse could be a performance bottleneck. The size of the data returned may vary depending on the request, so you need to be assured the right decision has been made for the majority of responses returned by the web method. An effective test would be to use a Load testing tool to compare the results when the BufferReponse is set to true then to false.

Is the web service caching its output?

Another of the customisable properties of a .NET web service enables output caching. This means that the web service remembers what response it gave to method calls with certain parameters, so that performance can be improved when the same methods calls are requested again. A time value is also specified for how long ASP.NET should cache the result.

[WebMethod(CacheDuration=1000)]

Test cases should be formulated to check whether your web service is a good candidate for output caching. If the web method should be writing or updating data to a database for example, output caching would be dangerous, as it would mean the database interaction would not happen for repeat requests. However the performance would benefit from output caching when web methods don’t require the server to take any consequential action, such as if they are operating on static data.

When it comes to looking how the web service scales, it is important to check if the right trade-off between memory and time has been made. Generally, speed is more critical than space on web service servers. Thorough testing would, however, involve checking that neither incorrect data, nor data which becomes out of date or invalid after a certain period has been cached to the server.

Transactional SOAP calls

A potential, but probably unavoidable, problem area for scalability is the caching required for SOAP transactions. The TransactionOption property of the web method attribute allows a series of get and response calls to succeed or fail as a single unit, but requires the caching of information between each of the transactional SOAP calls.

[WebMethod(TransactionOption=TransactionOption.RequiresNew)]

Example of transactions used in a web service

Imagine a web service whose purpose is to process retailers’ requests for an item of stock from a factory depot. The web method would connect to the factory’s stock database to get the number in stock for the item requested. If there are enough items available, the web method will decrement the quantity in stock in the factory database and authorise the distribution to the retailer, then debit the retailer’s account.

If one of the steps in the process fails, (if the authorisation fails to distribute the stock, for example) then preliminary steps should be rolled back, and subsequent steps should be abandoned. In this case, the original stock value should be restored in the database and the retailer’s account should not be debited.

Although the web service using transactional SOAP calls may perform well when the system isn’t under stress, increasing the number of users could soon degrade the performance radically. Load testing will identify how well the system scales.

Waiting for the web service response

To avoid delays when a client application calls a web service, check what is happening when it’s waiting for the response. The client application utilising the web service could be coming to a stand still while it is waiting for a response back. Although there are several factors determining the speed of a web service, also consider whether the application could continue to function while the web service is churning away in the background. An interesting feature of web services is that they support asynchronous communication without any special coding. The proxy class generated when a web reference is added creates both synchronous and asynchronous versions of the web methods.

Load testing

A web service will undoubtedly behave unpredictably when concurrent calls to the web service are made. Not only should you test for functionality failures, you should also be confident the web service meets acceptable performance criteria when under stress. Test cases should look at the performance of the server (e.g. throughput, CPU usage etc.) and statistics to look at the user experience (e.g. the time taken to get a response from a web method) when there are, say, 5, 50, 500, 5000 concurrent users or more. While there are no definitive standards to be met, consumers paying for your web service won’t tolerate system failure or even slow response times.

For QA departments this will mean investing in software capable of running scripts simultaneously, making different SOAP requests to the web service.

An important difference to understand is that XML web services will almost certainly be called using SOAP. .NET lets you invoke web services using HTTP GET and HTTP POST; in fact the automatically generated test page uses HTTP GET to invoke a web service.

Nonetheless, it is not the same as using a load testing tool to invoke the web service using HTTP. Not only could the HTTP protocol be removed before going into production, a load test should try to mimic the real life environment it will be faced with. Make sure that the load testing tool uses scripts which invoke the web service using SOAP.

ANTS (Advanced .NET Testing System) is a load testing tool by Red Gate Software which allows web services to be tested using SOAP. An evaluation copy of ANTS Load can be downloaded from Red Gate’s website.

Scripts are written in ANTS to interact with the web service exactly how a client application would. Web services will be responding to requests from other applications, not directly from end-users, so this is the behaviour we should accurately reproduce when load and stress testing. In ANTS the web reference is added, which automatically creates the proxy wrapper, exposing the web service’s methods and allowing them to be called in a script. Figure 3 shows an example Visual Basic .NET script calling methods in a web service from within ANTS.

Figure 3 – Example script calling a web service

The web methods called in a script will mimic the communication client applications will have with the web service. In ANTS each script is called an actor, as it acts out a particular scenario, e.g. requesting the list of employees and then requesting a particular employee’s details. When a load test is performed, multiple threads are created running an actor ‘s script simultaneously. You may want to make parallels to the real world by setting up multiple actors in the same test, each requesting a different set of web methods.

ANTS allows each actor to take a different weighting value, to correspond with the expected likelihood of real users using the web service, so one actor’s script may be run twice as often as another. It is also likely that you want the data requested in a web method to be different every time the script is run. In this case a table of data is created and the script is modified to take values from this table. You could, for instance, have a list of login names and passwords in a table, and every time the actor’s script is run it uses different login details. Running load tests helps you predict the performance of the web service before it goes into production. The results show the break down for each web method and let you compare the performance under different loads, so you can identify bottlenecks and arbitrary errors.

Summary

Testing web services presents a variety of new and interesting challenges. How a web service is implemented can have direct implications on how well it will scale and perform. We have considered how a web service is called along with a number of development issues affecting the performance of web services. Test strategies should be incorporated in the development cycle of a web service to guarantee success, beginning with proof-of-concept testing early on to make sure the architecture chosen is the correct one. Load testing is vital throughout development, to make comparisons between builds to identify the affect code changes have on performance and to know your web service will cope with the expected number of users or more.