Calling Cross Domain Web Services in AJAX

The latest craze for mashups involves making cross-domain calls to Web Services from APIs made publicly available by companies such as Google, Flickr and so on. Unfortunately, the XMLHttpRequest object doesn't allow calls made in one domain to a web service in another.

The article surveys the current, somewhat unsatisfactory, solutions and then assesses future directions.

One of the current vogues in web applications is the creation of mashups. This involves the marrying of content and/or functionality from two different sources. Over the last few years, the opening up of formerly proprietary APIs from the likes of Google, Yahoo, Last.fm, Flickr, YouTube and Amazon has allowed developers to implement in their own applications, with simple one line calls to the requisite APIs, features such as adding photos, maps, booklists, videos and playlists.

The APIs allow the developer to select content through a series of filters such as location, date, user id, or membership of a particular group. The APIs are almost always thinly disguised wrappers for a web service.

With a lot of modern applications now employing Ajax techniques (with indeed whole web sites such as http://www.programmableweb.com/, dedicated to mashup applications which utilize a lot of Ajax techniques), it makes sense to call the web services from within your own JavaScript. However, there is one big hitch to this theory. The XMLHttpRequest object is prevented from calling web services from outside its own domain. This is sensible given that if you called a script in one place and it, in turn, called a script on another server, it could leave an applicationopen to all sorts of malicious scripts, hacks and exploits.

However in the case of web services, it isn’t so sensible. Web services are called with either SOAP or HTTP-GET/POST requests and return information in the same way. They are designed to be called from other domains, and in some ways it’s an inherent contradiction to prevent them being called this way. So, given that there are plenty of mashups out there, and plenty of applications developed calling cross-domain web services, what do developers do to get around this limitation?

There isn’t a single answer to this question, but in this article I’ll discuss the current most popular techniques and what several developers are proposing to do get around this problem more neatly.

The XMLHttpRequest object

At the heart of the problem lies the XMLHttpRequest object. This object is central to many Ajax applications, although not an essential feature. The XMLHttpRequest object was introduced to IE5 as an ActiveX control, although the original version was conceived in Outlook Web Access 2000. This was an Outlook web-mail service that allowed people to access functionality such as download email, check calendars, and update contacts on the go, by allowing the application to issue its own client-side HTTP requests. It was introduced as a native object to Mozilla for release 1.0 and ended up in Safari 1.2 and Opera 7.60.

The XMLHttpRequest object doesn’t allow calls to be made from code in one domain to a web service in another. The latest craze for mashups involves calling Web Services from APIs made publicly available by companies such as Google, Flickr, Yahoo, Last.fm and YouTube. This means that a call will always have to be made cross-domain, otherwise you can’t use them. The best way to illustrate the problem is to build an example.

The currency converter example

In the example the application calls the freely available currency conversion services from webservicex.net. There are two drop down lists containing the list of currencies. So for example, you can select US Dollar in the first and Great British Pound Sterling in the second and then click on a button, to get the current rate. When the button is click, JavaScript is used to call the web service. The result is returned to the XMLHttpRequest object and ultimately rendered in the page.

Our application must have at least two sections, the first is the HTML page and the second is the script that calls the service. Our default.htm page is as follows:

Our second section is the script IEAjax.js. If you create both of these items in Visual Studio, you’ll be able to try this for yourself. Our application passes the web service URL straight to the XMLHttpRequest object using JavaScript code, via three functions:

The code starts with initiateConversion which creates an XMLHttpRequest object in the function createRequestObject, then it sets the URL to point at the web service, and add two variables from the drop down list boxes, one for the “From” currency and one for the “To” currency. You call the open method of the XMLHttpRequest object, passing it the URL. You then prime the onreadystatechange event handler to run the getData function when a response is returned from the XMLHttpRequest object. The getData function receives the XML, extracts it (via the DOM), and injects it into a table.

328-AjaxArticle1.gif

Of course this code won’t work by default due to the cross domain restrictions, so let’s look at the simplest, but least satisfactory solution first.

Internet Explorer workaround

If you can be sure that your target audience will only be using Internet Explorer, and won’t be too fussy about a security work-around, then there is an immediate solution available. You can add the site to your trusted sites, and reduce the security by enabling the “Access data sources across domains” option.

You can add a trusted site in IE via the Tools | Internet Options | Security tab. Then you can set the security for trusted site via the Custom level | Miscellaneous section. You’ll see that, by default, the “Access data sources across domains” option is set to Prompt. However, Prompt is unsatisfactory because it will interrupt any current application that is running with a message box, asking the user to confirm that they want access to the service. Instead, set the option to Enable.

However, quite apart from the problem that each user of the application will need to configure IE before they can use it, Firefox doesn’t even allow you to perform this workaround, and the default behavior of Firefox is not even to return an error. You’d have to check the error console to discover that an error had been raised, and to locate the source. So, as you can see, this is a far from universal solution.

Application proxies

A common theme of many of the solutions, instead, is getting JavaScript to call a proxy program (either on the client or the server) which, in turn, calls the web service for you. The output can be written to the response stream and then is available, via the normal channels, such as the responseText and responseXML properties of XMLHttpRequest.

Flash – CrossDomain.xml

If you have a Flash application as part of the setup, then you can make use of an XML file that allow Flash applications to call web services, and which can be legally called. The Adobe site even specifies that one typical usage of crossdomain.xml is when

“a movie wishes to make a request to content hosted on a public server, such as web services, but is prohibited by the Flash Player cross-domain data access restrictions.”

The crossdomain.xml file specifies other domains that can be called, and is placed at the root level of the application on any server to which the Flash movie player has access. You can then make the call to the web service via ActionScript.

An example Crossdomain.xml file is as follows:

Of course this technique depends on you using somewhere Flash in your application. Another drawback is that it only works with the most recent versions of Flash (version 7 onwards), so if one of these two criteria isn’t met you will need another solution.

Server-side proxy

For the ASP.NET developer, probably the best solution is to load the XML content using ASP.NET and then return the content to the client side as pure XML. The downside of this is that you are adding an extra layer of indirection to your JavaScript code, and it requires you to have a server hosting ASP.NET to call on. Of course this is part and parcel of ASP.NET development, but if you have been developing your application only with JavaScript, then having IIS might not have been on your list of requirements.

In order to create an ASP.NET server side proxy, you have to amend your JavaScript code to add another layer of indirection. Instead of calling the web service directly, you now use the XMLHttpRequest object to call an ASP.NET page instead, and pass on any relevant variables.

Here is some sample JavaScript code:

In turn the ASP.NET code can load the URL as an XML document and then you can simply write the contents of the InnerXML property back to the response stream

The JavaScript callback function getData remains unchanged from the previous section.

cURL proxy

If you have access to PHP – or alternatively can’t use ASP.NET – then there is another alternative: you can use the cURL (client URL) library extensions in PHP. The JavaScript code can be amended as follows:

Your PHP code makes the call on behalf of the JavaScript, by adding the hostname to your URL, and using the cURL to perform the invocation of the web service. Although we’re passing the query string to the PHP code, it can accept the data as either a GET or a POST request:

The principle is the same as the ASP.NET code; the proxy opens a request to the web service for us and returns the response as XML. The XML is written to the response stream where it can be picked up by the XMLHttpRequest object.

Dynamic Script Tag hack

If you have a web service that returns JSON (JavaScript Object Notation format), then another option is open to you. You can dynamically create a script tag and assign the src attribute to the location of the web service. This will only work if the web service can return JSON, which is a format for data exchange based on a subset of JavaScript. JSON cannot represent functions or expressions; it can only represent data. It can be easily converted into a JavaScript value, and so is easy to reference via a script.

Unfortunately the Currency Conversion web service of webservicex.net doesn’t offer this option. However all of the Yahoo Web services currently return JSON and one such example is the ImageSearchService. This is a simple “Image Search Engine”, whereby you supply the name of the person (or item) of whom you photographs and it will return an object for each “hit”.

For example, the following URLS (web service calls) can be used to retrieve a list of all the images of Einstein on the web:

and:

It’s possible, in a few lines of code, to add this call to an Ajax application to create an image search engine so that whoever you type the name of, it will return the first 10 images it finds on the Web in order. Our HTML/ASPX page would look as follows:

When the button is clicked, we make the call to the JavaScript function, dynamicTag, which dynamically creates a <script> element. This script element has no src attribute, so we dynamically create one and we assign the src attribute’s URL as a call to the web service, so that when the page is opened, the web service is automatically called.

We pass as a parameter to our web service the value the user supplied to the text box. We also specify that the output should be JSON in the parameters, and the name of the callback function that the thread of execution should pick up at when control is returned to the client. The callback function is as follows:

The JSONData object has a ResultSet format, which returns a separate object for each result that is returned. The object returns Title, Summary, Url, ClickUrl, RefererUrl, FileSize, FileFormat, Height, Width and Thumbnail properties. So, for the first item in our ResultSet in the example, the properties would be returned as follows:

The application then dynamically creates an image and assigns the URL property as the src attribute. We loop through for the first 10 images displaying each to the page as follows:

328-AjaxArticle2.gif

There are several disadvantages to this technique, though. This solution is reliant on JSON and so excludes quite a few web services that don’t have a JSON option. Also, when you use dynamic scripts, you can’t tell if the script had loaded correctly or not, as there is no return from the script; either it works or it doesn’t. A further problem is that, in IE, the dynamic loading of scripts stops all other processing. Most seriously, using this technique with Internet IE5 and IE6 causes a potential memory leak.

Future directions

If you find none of these solutions completely satisfying, and feel that they all involve hacks or incomplete solutions, then you’re not alone here: so do I. There are a lot of ifs and buts involved, and your choice of solution will often be dictated not by what is best, but what is available to you.

As a consequence there are several other solutions being proposed by different developers and it’s worth taking a brief look at each.

FlashXMLHttpRequest

This is a project by Julian Couvreur, which leverages the existing crossdomain.xml files but instead uses a FlashXMLHttpRequest proxy. This is already in beta, although several links to it are now dead. It uses a small SWF file to make GET and POST requests on your behalf.

The reason that this is more interesting than just using crossdomain.xml is that this object, in addition, has SWF versions that support earlier versions of Flash and enables almost any browser (that has Flash installed) to make use of the Crossdomain.xml files.

The FlashXMLHttpRequest object has some limitations, (placed upon it by the version of Flash it is using), such as not being able to send or receive HTTP status codes or headers, or handle HTTP Puts and Deletes. It’s a pity that links to the beta were dead at the time of publication of this article, because the proposal is an interesting one.

ContextAgnosticXMLHttpRequest

Chris Holland’s proposal at:

http://chrisholland.blogspot.com/2005/03/contextagnosticxmlhttprequest-informal.html

involves creating a second object, ContextAgnosticXMLHttpRequest, which would be used instead of the XMLHttpRequest in situations that involve cross domain calling.

The ContextAgnosticXMLHttpRequest object would be able to pass security details to the application allowing the application permission to call up specific services in other domains. This proposal runs in tandem with debate from other developers, and Microsoft, who propose passing an extra HTTP Header, which is notionally called “X-Allow-Foreign-Hosts.” The ContextAgnosticXmlHttpRequest object would not expose any data from a service without this header.

This proposal is no further forward than the informal RFC stage, but indicates that many people are thinking along similar lines.

JSONRequest

Last is JSONRequest which is Doug Crockford’s proposal at http://json.org/JSONRequest.html.

He proposes a new browser service that will initiate a two-way data exchange between the browser and a JSON data server. This relies on future versions of browsers incorporating this facility into the browser.

Doug proposes the creation of a new object, JSONRequest (not to be confused with the one we created earlier), which can initiate GET, POST or CANCEL requests. The data that is sent and received via the object is serialized into the JSON format by the web service, The object would be restricted to using JSON encoded values.

As the proposal excludes the use of cookies or passwords, it neatly avoids the problems of the browser being given false authorization. It looks perhaps the most promising of the three alternatives, given that JSON is already being viewed as a possible alternative to XML in some situations. It also confers certain benefits such as being simpler, more readable and better suited to data exchange, as opposed to document exchange which is what XML deals in.

Conclusion

There is no one perfect way to call web services cross domain using Ajax. Currently I rely on an ASP.NET server-side proxy as a solution; although ultimately a solution involving JSON looks to be preferable. As things stand you are not supposed to be able call a web service cross domain using Ajax and any attempt to do so involves some sort of hackery or underhandedness. However if web services popularity is to continue to prosper, then a de-facto solution will have to emerge, and more likely than not, it will have come from the dusty backrooms of developers, rather than as the latest add-on from the browser makers.