Executing all SSIS packages in a folder: three methods

A common requirement is to execute several SSIS packages as part of one logical process. This is especially true when one is transferring existing data from an external system.

If the indivisual packages are placed in a folder and named in such a way that a batch can be executed via a wildcard filemask, then they can be altered without changing any code, and can be used for different processes by changing the filemask. This article presents three different methods of executing all packages in a folder:

1. SSIS control package
2. Stored procedure
3. DOS batch file

The methods are of differing complexity and flexibility, the SSIS package being by far the most difficult to code, and the DOS command being by far the simplest but also the least flexible.

The folder that contains the packages is d:\TestPackages\.

All the packages to be run have names of the format t1…….dtsx.

SSIS control package

For this we create a package that uses a for each loop task to call an execute package task for all packages in the folder.

The first task is to create an SSIS package with a Foreach Loop container that will loop round the packages in the folder, setting a variable called “PackageToRun” to the file name for each package.

1. Load Business Intelligence Development Studio and start a SSIS project.
2. Create a new package.
3. Add a Foreach Loop container to the package.
4. Right-click on the Foreach Loop container and select Edit.
5. Click on Collection.
6. Set the Enumerator to Foreach File Enumerator.
7. In the Enumerator configuration:
   1. Set Folder to “d:\TestPackages\”
   2. Set Files to “t1*.dtsx”
   3. Under Retrieve file name select Fully qualified.
8. Click OK.

1. Click on Variable Mappings.
2. Click on the Variable drop-down list and select New Variable.
3. Set Name to PackageToRun.
4. Click OK.

Next, we need to add the Execute Package task to the Foreach Loop container so that this task will be executed for each package that we wish to run. We then set the variable value to be the name of the package to be executed by the ExecutePackage task.

1. Drag an Execute Package task into the Foreach Loop container.
2. Right-click on the Execute Package task and select Edit.
3. Select Package.
4. Set Location to File system

1. Click on the Connection drop-down list and select <New connection…>.
2. Set the File to an existing package.

1. Click OK to save the connection.
2. Click OK to complete the Execute Package task configuration.

Finally, we configure the connection to use the variable package name:

1. Right-click on the connection and select Properties.
2. In the Properties window change the name to PackageToRunConnection.

Note: this will automatically update the connection name in the Execute Package Task.

1. Select Expressions and add a new expression.
2. In the property drop-down list select ConnectionString.
3. Click on the Expression Editor button.
4. From the Variables tree drag @[User::PackageToRun] into the Expression window.

1. Click OK twice, to save the expression.

The package should now look like this:

If you now run the package, it will execute each package in the folder with the correct filename mask.

Stored Procedure

This involves the following steps:

1. Retrieve a list of the package names to run from the folder
2. Loop through each package name
3. Execute each package

First of all, we need to get a list of all packages in the folder. The following code executes the DOS command dir /B d:\TestPackages\ t1*.dtsx and places the output in the temporary table #dir. The DOS command gets a list of the filenames in the folder which satisfy the filemask.

The following is a useful technique for looping through values in a table and here we use it to loop through the filenames in #dir:

Now we just need to create the command to execute the package:

This takes the package name and uses the dtexec command to execute it. Putting this all together, we get:

The above completes the code needed for the requirement but to simplify the implementation I like to keep a utility stored procedure which calls another stored procedure for each file (SSIS package) in a folder. The following utility stored procedure, ProcessAllFilesInDir, uses the above techniques to obtain a list of files from a folder and calls another stored procedure for each one. The file path and mask are input as parameters together with the @ProcSP parameter that defines the stored procedure to be called for each file (in this case this will be the ExecutePackage procedure that executes each SSIS package).

With this, the only coding needed is to create the procedure to execute the package and to call it. The ExecutePackage procedure is as follows:

And the call:

DOS batch file

The DOS command required is dtexec /F “d:\TestPackages\mypackagename“ as above. We can make use of a DOS For loop to execute this command for all files in a folder:

This uses the for loop construct:

• (d:\TestPackages\t1*.dtsx) returns a list of the files which meet the path (similar to a dir command).
• For %1 in (d:\TestPackages\t1*.dtsx) loops through the file list setting the variable %1 to the filename.
• Do dtexec /F % executes dtexec /F with each filename in turn.

Note that you can place this command in a bat file to execute.

Summary

We have introduced three different methods of executing all packages in a folder. The methods are of differing complexity, the SSIS control package being by far the most difficult to code, and the DOS command by far the simplest.

There are a few things missing from the solutions presented which would (or should) prevent them from being used in a production system. At a minimum this should include logging and error handling. The start and end time of each package execution should be logged by the control process and also errors should be detected and logged.

At first sight, it may seem that the DOS command is the one that we should be using, but this method loses a lot of flexibility. There is much you can do with batch files, but not many people have skills in writing such code beyond very simple commands, so you can quickly end up with an un-maintainable system.

The stored procedure solution gives a lot of flexibility – it is easy to add logging and pre/post processing to the process. However, it does require an installation of SQL Server. Although this process would usually be run via SQL Server, it may sometimes need to be run on a client machine (or not involve SQL Server at all); this is quite difficult with a stored procedure as it is server based whereas the other two solutions are client applications.

The SSIS solution keeps the process within SSIS, so you can take advantage of the integration of package processing, e.g. inheritance of properties from the calling package, development and debugging of packages, within the development studio.

Overall, none of these solutions is very difficult to implement, so the decision which to use will probably depend on the experience of the developer, how much flexibility is required, and the environment in which it is to be run.