You may have already experienced ETL Hell, where you have a large number of similar of SSIS tasks, and a small change, such as an alteration to the network topology, means that they all need to be altered with the correct connection details. Perhaps you should consider creating design patterns for all the standard components of integration tasks in BIML, and generating the SSIS packages from these? Amarendra walks you through the process.… Read more
Scala and Apache Spark might seem an unlikely medium for implementing an ETL process, but there are reasons for considering it as an alternative. After all, many Big Data solutions are ideally suited to the preparation of data for input into a relational database, and Scala is a well thought-out and expressive language. Krzysztof Stanaszek describes some of the advantages and disadvantages of a scala-based approach to implementing and testing an ETL solution. … Read more
When there are several SSIS projects with packages in a SQL Server Database or Data Warehouse development, automated deployments as part of Continuous Integration can get tricky. Nat Sundar describes how he created a Deployment script that is intended to provision a Data Warehouse for System Integrated testing (SIT).… Read more
SSIS was not originally designed with automated deployment in mind. However, any database that uses SSIS projects and their packages needs a way of scripting the deployment of them if the database is to be deployed rapidly, or via a build server. Nat Sundar shows a technique that can be extended to manage the scripted deployment of SSIS projects. … Read more
In many enterprises, where there are a number of separate systems engaged in processing data, there arises the daunting task of checking and reconciling data as it flows between systems. Discrepancies in data must be detected, tracked and corrected as quickly as possible: there is no room for error in doing this. A Reconciliation Hub can provide the answer, as Rahul Gupta explains.… Read more
ETL ( Extract, transform, load) doesn't have to be like a spell on hell. To make a success of ETL systems, you need the freedom and ability to make graceful U-turns when you detect a mistake in architecture or configuration: to fix the root problem rather than to merely tackle the symptoms. Feodor lists the eight most common root causes of failure in ETL systems, and how to fix them.… Read more
The Project Deployment Model introduced in SSIS 2012, which was explained in the first part of this series, speeds up the deployment of database projects in which there may be hundreds of SSIS packages per project. Not only that, but deployments can be configured differently for each environments such as test and staging, and there are now ways of monitoring the status and performance of packages and of versioning the SSIS Catalog.… Read more
It used to be that SQL Server Integration Services (SSIS) packages had to be deployed individually. Now, they can be all deployed together from a single file by means of the Project Deployment Model introduced in SSIS 2012. Where there are tens or even hundreds of SSIS packages to deploy, this system is essential. Feodor Georgiev talks us through the basics in the first of a three-part series.… Read more
The job of retrieving data from Excel, and importing it into SQL Server hasn't the same appeal or glamour as, for example, performing heroics with ill-performing queries. This could be why one hesitates before asking questions about how to do it. Rob Sheldon calms your private doubts and fears by answering those embarrassing questions.… Read more
SSIS was designed to be extensible. Although you can create tasks that will take data from a wide variety of sources, transform the data is a number of ways and write the results a wide choice of destinations, using the components provided, there will always be occasions when you need to customise your own SSIS component. Yes, it is time to hone up your C# skills and cut some code, as Saurabh explains.… Read more
SSIS is able to take sorted data from more than one OLE DB data source and merge them into one table which can then be sent to an OLE DB destination. This 'Merge Join' transformation works in a similar way to a SQL join by specifying a 'join key' relationship. This transformation can save a great deal of processing on the destination. Annette Allen, as usual, gives clear guidance on how to do it.… Read more
SQL Server Integration Services provide a versatile way of reading Excel files into SQL Server. A task like this illustrates the advantages of the graphical approach of SSIS. Andy Brown explains.… Read more
The Conditional Split can route data rows to different outputs depending on whatever criteria of the data that you wish. It works rather like the SWITCH block in a procedural language. Annette shows how to add a conditional split to your data flow.… Read more
The Control flow components of SSIS can cause a great deal of puzzlement to anyone learning SSIS, especially the Foreach Loop. Annette continues her popular 'Basics' series by showing how to use the Foreach Loop container for copying files.… Read more
The Execute SQL Task of SSIS is extraordinarily useful, but it can cause a lot of difficulty for developers learning SSIS, or only using it occasionally. What it needed, we felt, was a clear step-by-step guide that showed the basics of how to use it effectively. Annette has once again cleared the fog of confusion… Read more
In the third of her SSIS Basics articles, Annette Allen shows you how to use Variables in your SSIS Packages, and explains the functions of the system-defined variables.… Read more
In order to be able to take data from a variety of sources, manipulate it, and then export it to one of several destinations, SSIS has to use its own data types. Generally, the conversions of data types is handled implicitly, but if you hit problems, then you may need to intervene in one of several ways to ensure an appropriate conversion.… Read more
Annette continues her popular series for SSIS beginners by showing how a data flow task can be used in a package to move data from a SQL Server database to an Excel file and insert an additional column into the Excel file that's based on derived data.… Read more
When working with databases, the use of SQL Server Integration Services (SSIS) is a skill that often needs to be acquired quickly, from scratch. Up until now, it has been a curiously frustrating search to find out the basics, fast, in order to get up and running quickly. No longer, as Annette comes up with a simple introduction for the rest of us.… Read more
With SSIS, you can perform a lookup on data in the course of a task, using referenced data from any OLE DB source. It is a useful feature that enables you to check on the validity of data, or interpret it before proceeding. Robert Sheldon explains.… Read more
Subscribe for more articles
Fortnightly newsletters help sharpen your skills and keep you ahead, with articles, ebooks and opinion to keep you informed.
Compliant Database DevOps
