Although FTP is commonly used to transfer files over the Internet, it’s somewhat unfriendly to firewalls, because of the various ports that need to be opened. A Microsoft solution you can use for Internet file transfer is the Background Intelligent Transfer Service, BITS for short. BITS runs in the background (hence its name) as a service and it transfers files over the browser ports (80, 443) using HTTP or HTTPS. In addition, BITS transfers are restartable, so if the transfer of a large file gets interrupted, it will resume where it left off.

When uploading, it requires the receiving server to be running IIS, and to be configured with a virtual directory that is set up to be a BITS destination. When downloading, you just need the BITS client – and that’s been part of the operating system since Windows 2000

In this article, I’ll be describing how to do an upload files using BITS, starting with a description of the required server configuration. BITS version 1.5 is required for uploads so if you ant to try out the code, you can download this (and later) versions from:

http://windowssdk.msdn.microsoft.com/en-us/library/ms678636.aspx [link deprecated]

Please note also that BITS requires IIS 5.0 on Windows 2000 Server and IIS 6.0 on the Windows Server 2003 family; BITS does not support IIS 5.1 on Windows XP.

Configuring a server for upload

Having first established a virtual directory in IIS, the BITS server extension properties need to be configured, as shown in Figure 1.

Fig.1: BITS server extension properties

The key setting here is the checkbox that allows clients to upload data to this virtual directory. It also shows the checkbox that is configured here to delete incomplete jobs after 14 days – if the upload was incomplete and has not been resumed after that time, it will be deleted.

For more information on configuring the BITS server, refer to this MSDN article:

http://msdn.microsoft.com/library/en-us/bits/bits/setting_up_the_server_for_uploads.asp?frame=true [link deprecated]

BITS runs over browser ports, so you might be concerned about the impact of data transfer on existing apps using those ports. To address this, Active Directory has settings to throttle the rate of transfer, so you don’t need to worry unnecessarily about BITS transfers taking over your network bandwidth.

BITS utilities

There is a tool called Bitsadmin.exe that you can use to upload or download files. It is a command-line program that you get as part of the support tools download for your operating system. For XP SP2, the link is here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=49AE8576-9BB9-4126-9761-BA8011FABF38&displaylang=en

In the context of developing your own BITS program, Bitsadmin is a useful tool for enumerating the jobs on the system, and deleting ones you know you don’t want. For example, to list all the current BITS jobs, use this command:

And use this one to delete them:

A BITS job is identified with UUID, a Guid. The jobs are listed by Guid, and you can obtain the error status of a particular job, using this command:

Programmatic interfaces

BITS is COM-based, which means that you’d typically write code in C++, including the header file bits.h from the Platform SDK:

There are samples in the Platform SDK (soon to called the Windows SDK) that show how to use BITS in this way. See the Samples\Web\bits folder for an IE extension program and a C++ upload program.

Where’s the .NET?

There is a Microsoft newsgroup, microsoft.public.windows.backgroundtransfer, devoted to BITS. In early 2006, the BITS folks at Microsoft posted interop files for using BITS with managed code, and the C# one is included in the project you can download. It is effectively the C# source code, describing the interfaces and methods you use to call BITS – you can compile it as a class library (which makes it an interop Dll) or just include the source directly into your program, which is what I did here.

Uploading a file

The sequence for an upload job is straightforward. The IBackgroundCopyManager interface has a CreateJob method where you specify that you’re creating an upload job. This CreateJob call returns an IBackgroundCopyJob interface to control the job, as well the Guid for the job. With this interface, you add the file to be uploaded, call the Resume method to upload the file, and then wait for the transfer to complete. Although this is the way the example program works, you don’t have to wait for the job to complete. If you save this Guid somewhere, you can use it later to enquire about the status of the job, so you could initiate the job, and then check its status later. If you really want to wait for the job to complete in your program, you have a couple of choices. First, you can set up a callback interface – BITS will call your methods when the job completes, gets an error, or changes status. Alternatively, you can wait, and poll the status of the job until it goes into a state indicating that it has completed. I chose this latter method, because I came across a situation related to failed proxy server authentication, where the job was transitioning to an error state without my callback being notified, so I never knew that the job had completed with an error. I’ll go over all the code later in this article but, before that, I’ll outline some issues that need to be considered.

Server credentials

You probably don’t want just anyone to be able to upload files to your server, so you should configure the server’s virtual directory with directory security. These credentials will need to be specified in your upload job with the SetCredentials method, described later.

BITS and user accounts

BITS has some restrictions associated with user accounts. Transfer takes place only when the user who owns the job is logged on to the system interactively. When not logged on, the job is suspended. The way to detour this restriction is to run the transfer job under a system account which is seen as being always logged on, such as the local system account. The implication here is that your file transfer application needs to run as a system service, and this causes some difficulties with proxy servers that I’ll describe later. There is no way that you can simulate a user being logged on interactively (such as by impersonation) so, if you need to transfer files when no users are interactively logged on, (very common on server systems) you must be resigned to running your BITS jobs from something like a service running with the local system account.

Proxy servers

Many connections between client systems and the Internet are moderated via a proxy server for security reasons. If you go to Internet Options on your system and select the Connections tab and click the LAN settings button you’ll see something like Figure 2.

Fig. 2: Proxy server configuration

The example above shows a specific proxy server (“proxy”) configured to use port 8080, a typical default. This means that client programs will connect to the Internet using port 8080 to that named proxy server, which will forward the request to the actual destination URL. In practice, the most common setting is “Automatically detect settings.” This invokes the Web Proxy Auto Discovery protocol (WPAD) to return the actual proxy settings to client programs. This is useful because the setting can be changed (such as changing the proxy server name) and you don’t need to go and change the client Internet Options settings on what might be a large number of computers.

There are two things you need to be concerned with which are common to all Internet access via a proxy server: How do you find out what the proxy server actually is, and what do you do if it requires authentication?

To answer the first part of the question, the 2.0 version of the .NET framework has enhanced support for proxy servers compared with the 1.1 version. There is an excellent article at:

http://msdn.microsoft.com/msdnmag/issues/05/08/AutomaticProxyDetection/default.aspx

In short, the result is that you can use an IWebProxy interface to get the proxy server location for a particular URI. Keep in mind that proxy server locations and ports are not the same for every URL, because they can be specific to the URL that you ultimately want to connect to. Once you have the proxy server location, the BITS IBackgroundCopyJob interface has a SetProxySettings method that you call to specify the proxy server location.

Regarding the second part of the question, proxy servers can be configured to require authentication. For example, Microsoft ISA Server can be configured to require a selection of authentication methods, including Windows authentication (NTLM), Basic and Digest. In general, these will be handled for you automatically, if you are running with a user account.

Proxy servers and system accounts

If you run your code as a service running with a system account (which would be required to use BITS when nobody is logged on to the system) you immediately run into some issues with proxy servers, because proxy server settings are associated with user accounts. When you go to the Connections tab in Internet Options (see Figure 2), these settings are not system-wide proxy server settings. They are proxy server settings for your account. You can alter them by logging on and changing them. But you can’t log on with the system account to set up proxy server settings. So the first issue to deal with is how you decide what proxy server to use.

The second issue is to do with credentials. Your service is running with the system account, so you have no implicit credentials that can be automatically provided if authentication is required by the proxy server. Can you run the service under an account to fix this issue? No, because the rule that says the transferring user must be logged on interactively applies at all times – that’s why we ended up running with the system account in the first place.

These proxy issues aren’t anything to do with a lack of APIs. The BITS interfaces have the SetProxySettings and SetCredentials methods to set the proxy server location and associated credentials. The issues are about configuration and how you know what the proxy is and what credentials to use, because there are no user settings for the system account.

To determine the proxy server, you could configure specific settings in a configuration file used by your service where you explicitly name the proxy server and the access port number that you can supply to the SetProxySettings method. You could also store the relevant credentials in a configuration file, although keeping a password in plain text is probably not a good idea. There is also the general issue that administrators now need to keep the proxy server settings and credentials up to date if servers or passwords change.

Because of these issues, I recommend avoiding BITS transfers from non-interactive user sessions when proxy servers are being used.

The upload program

Having covered the issues related to uploading a file, I’ll take you through how the upload works in C# code. The BITS definitions here, such as IBackgroundCopyJob, are from the C# interop file.

To initialize the upload job, you need an IBackgroundCopyManager interface, with which you can create an upload job as seen here:

The IBackgroundCopyJob interface has been updated to IBackgroundCopyJob2 with some new methods so, to use these new methods, we perform a cast (which does a COM QueryInterface underneath to get an IBackgroundCopyJob2 interface pointer from IBackgroundCopyJob):

There’s a timeout value you can set on a transfer. The value in seconds is the time that BITS will try to transfer after it encounters a transient error. An example of one these errors would be a failure to connect to the server. After this timeout, BITS will report the job as having failed the transfer. Here, the timeout is being set to one minute:

Setting credentials can take many forms. The SetCredentials method has two targets (proxy server or IIS server) and authentication schemes (such as NTLM, Basic). At the IIS server end of the upload, the virtual directory may have been configured with an account that you need to specify for this job, using code like this, where the target is the destination server and the authentication scheme is Basic:

The SetCredentials method also has a target parameter which indicates that the credentials are intended for use with a proxy server. As a general rule, you don’t need to set these credentials if you are running with implicit credentials, but you do need to set them in the following cases:

• You are running with a service account, such as localsystem or localservice, which cannot be authenticated by the proxy server.
• You are running with a valid user account, but the credentials that you are running with are not the ones you wish to be used at the proxy server.

Something to be aware of, when using this API with proxy servers, is that you should specify the user account in the domain name\account name format, even if it’s actually just the name of the machine on which you are running (because you’re not in a fully networked domain). The reason is that, if you don’t specify a domain name, at some point, one will be added for you by Windows, before the account name gets sent to the proxy server. The proxy server will see an account in the form domain\account. This domain that’s added will typically be the default domain for the user account you are running with. This is probably fine if you are actually running with a user account, but if you are supplying alternate credentials, you should specify the appropriate domain. For example, when running with the system account there is no default user domain. The snippet below sets the NTLM credentials for a proxy server:

However you don’t actually know which authentication mode the proxy server requires so, once you’re in the business of setting credentials for a proxy server, you’ll have to specify all the possibilities (it’s OK to call SetCredentials multiple times with different target values). In this snippet, we set the credentials for the Digest authentication mode:

You also need to know the actual proxy server to use. In the .NET 2.0 framework, you get the default web proxy by calling the WebRequest static method, DefaultWebProxy, and then you retrieve the specific proxy for your ultimate destination URL. Note that the settings for the URL may mean that you don’t use the proxy server, because it’s bypassed. The proxy server address returned by GetProxy cannot be used directly with the BITS SetProxySettings method, because it returns the address with a trailing slash which will cause BITS to fail to find the specified proxy server.

To perform the actual transfer, just call the AddFile method, passing the destination URL and the path to the file to be uploaded. Note that a download job can have multiple files per job but an upload job must contain only one file. Then call Resume to start the transfer.

Once the job is running, the code just loops until the job has transitioned to one of the states that means it has finished, successfully or not.

In the case of an error, the IBackgroundCopyJob2 interface has a GetError method that returns an interface pointer (IBackgroundCopyError) that can be used to obtain error information with its own GetError method.

Finally, it’s best to ensure that you call Complete or Cancel on the job (as shown above) to ensure that the job doesn’t linger around and get automatically retried by BITS, because if BITS detects a transient error it will attempt to retry the job later. There’s nothing wrong with that, of course, except that this program expects the upload to be monitored by the code, not to complete at some later time when the program isn’t aware of it. If the error is a permanent, non-retryable error, BITS will eventually time out the job and delete it.

One of the errors to watch out for is an access denied error when attempting a transfer. Although this is a security issue, the actual reason is probably that the file you are attempting to transfer already exists at the server. BITS can be configured at the server end to allow overwrites – see the BITSAllowOverwrites value in the documentation for the BITS IIS extension properties.

Conclusion

BITS is a useful technology for transferring files over the Internet, and this article should get you started with using it in your applications. It should also stand you in good stead for writing download programs, since the proxy issues with downloads are the same as with uploads.