Labels

Apache Hadoop (3) ASP.NET (2) AWS S3 (2) Batch Script (3) BigQuery (21) BlobStorage (1) C# (3) Cloudera (1) Command (2) Data Model (3) Data Science (1) Django (1) Docker (1) ETL (7) Google Cloud (5) GPG (2) Hadoop (2) Hive (3) Luigi (1) MDX (21) Mongo (3) MYSQL (3) Pandas (1) Pentaho Data Integration (5) PentahoAdmin (13) Polybase (1) Postgres (1) PPS 2007 (2) Python (13) R Program (1) Redshift (3) SQL 2016 (2) SQL Error Fix (18) SQL Performance (1) SQL2012 (7) SQOOP (1) SSAS (20) SSH (1) SSIS (42) SSRS (17) T-SQL (75) Talend (3) Vagrant (1) Virtual Machine (2) WinSCP (1)

Monday, December 19, 2011

SSIS Merge Join for Incremental Load

We can use Merge Join transaction to achieve incremental loading of data:

Add a Data Flow task containing the following process:

Step 1: Add two OLEDBSource configured to Customer table in Source DB and DimCustomer table in Target DB.

Step 2: Add two Sort transaction (see Image 1 below) and sort data by Customer ID.




















Step 3: Add a Merge Join, and config as shown below:



















Step 4: Add a Conditional Split and create a condition to get CustomerID with Null values.




















Step 5: Now redirect rows containing CustomerID with Null values to OLEDB Insert DimCustomer and other records to OLEDB Update DimCustomer (OLEDB Command).

Step 6: In OLEDB Command add the below query and in SQL Command and perform the mapping as shown in images below:

UPDATE [Demo].[dbo].[DimCustomer]
   SET [CustomerName] = ?
      ,[MaritalStatus] = ?
      ,[Gender] = ?
      ,[AddressLine1] = ?
      ,[AddressLine2] = ?
      ,[Phone] = ?
 WHERE [CustomerID] = ?







Now execute the package to perform incremental load.




No comments:

Post a Comment