Developing Metadata Design Patterns in BIML

3
68
You may have already experienced ETL Hell, where you have a large number of similar of SSIS tasks, and a small change, such as an alteration to the network topology, means that they all need to be altered with the correct connection details. Perhaps you should consider creating design patterns for all the standard components of integration tasks in BIML, and generating the SSIS packages from these? Amarendra walks you through the process.… Read more

Scala and Apache Spark in Tandem as a Next-Generation ETL Framework

8
26
Scala and Apache Spark might seem an unlikely medium for implementing an ETL process, but there are reasons for considering it as an alternative. After all, many Big Data solutions are ideally suited to the preparation of data for input into a relational database, and Scala is a well thought-out and expressive language. Krzysztof Stanaszek describes some of the advantages and disadvantages of a scala-based approach to implementing and testing an ETL solution. … Read more

Deployment Automation for SQL Server Integration Services (SSIS)

4
11
SSIS was not originally designed with automated deployment in mind. However, any database that uses SSIS projects and their packages needs a way of scripting the deployment of them if the database is to be deployed rapidly, or via a build server. Nat Sundar shows a technique that can be extended to manage the scripted deployment of SSIS projects. … Read more

Reconciling Data Across Systems Using a Reconciliation Hub

0
16
In many enterprises, where there are a number of separate systems engaged in processing data, there arises the daunting task of checking and reconciling data as it flows between systems. Discrepancies in data must be detected, tracked and corrected as quickly as possible: there is no room for error in doing this. A Reconciliation Hub can provide the answer, as Rahul Gupta explains.… Read more

How to get ETL Horribly Wrong

4
111
ETL ( Extract, transform, load) doesn't have to be like a spell on hell. To make a success of ETL systems, you need the freedom and ability to make graceful U-turns when you detect a mistake in architecture or configuration: to fix the root problem rather than to merely tackle the symptoms. Feodor lists the eight most common root causes of failure in ETL systems, and how to fix them.… Read more

SSIS 2012 Projects: Deployment Configurations and Monitoring

6
114
The Project Deployment Model introduced in SSIS 2012, which was explained in the first part of this series, speeds up the deployment of database projects in which there may be hundreds of SSIS packages per project. Not only that, but deployments can be configured differently for each environments such as test and staging, and there are now ways of monitoring the status and performance of packages and of versioning the SSIS Catalog.… Read more

SSIS 2012 Projects: Setup, Project Creation and Deployment

9
136
It used to be that SQL Server Integration Services (SSIS) packages had to be deployed individually. Now, they can be all deployed together from a single file by means of the Project Deployment Model introduced in SSIS 2012. Where there are tens or even hundreds of SSIS packages to deploy, this system is essential. Feodor Georgiev talks us through the basics in the first of a three-part series.… Read more

Importing Excel Data into SQL Server Via SSIS: Questions You Were Too Shy to Ask

17
125
The job of retrieving data from Excel, and importing it into SQL Server hasn't the same appeal or glamour as, for example, performing heroics with ill-performing queries. This could be why one hesitates before asking questions about how to do it. Rob Sheldon calms your private doubts and fears by answering those embarrassing questions.… Read more

Developing a Custom SSIS Source Component

5
56
SSIS was designed to be extensible. Although you can create tasks that will take data from a wide variety of sources, transform the data is a number of ways and write the results a wide choice of destinations, using the components provided, there will always be occasions when you need to customise your own SSIS component. Yes, it is time to hone up your C# skills and cut some code, as Saurabh explains.… Read more

SSIS Basics: Using the Merge Join Transformation

15
170
SSIS is able to take sorted data from more than one OLE DB data source and merge them into one table which can then be sent to an OLE DB destination. This 'Merge Join' transformation works in a similar way to a SQL join by specifying a 'join key' relationship. This transformation can save a great deal of processing on the destination. Annette Allen, as usual, gives clear guidance on how to do it.… Read more

Working with SSIS Data Types

6
134
In order to be able to take data from a variety of sources, manipulate it, and then export it to one of several destinations, SSIS has to use its own data types. Generally, the conversions of data types is handled implicitly, but if you hit problems, then you may need to intervene in one of several ways to ensure an appropriate conversion.… Read more

SSIS Basics: Setting Up Your Initial Package

21
314
When working with databases, the use of SQL Server Integration Services (SSIS) is a skill that often needs to be acquired quickly, from scratch. Up until now, it has been a curiously frustrating search to find out the basics, fast, in order to get up and running quickly. No longer, as Annette comes up with a simple introduction for the rest of us.… Read more