Consider you want to perform an incremental load between Source and Target table. Let assume you have both source and target table in MS SQL database. Perform the below steps for incremental loading:
1. Create a reference table for incremental load in target database (e.g., dbo.Incrementalload with columns TableName and Incrementaldate1). Your table will look like this:
---------------------------------------------------------------
TableName | Incrementaldate1 | Incrementaldate2
----------------------------------------------------------------
Event | 2000-01-01 | NULL
----------------------------------------------------------------
User | 2010-01-01 | NULL
----------------------------------------------------------------
2. Design your Talend job as like below:
3. Create a context variable 'Incrementaldate1'
4. In tMSSQLInput_2: Get incremental date to assign to context variable.
5. tJavaRow_1: Assign incremental date to context variable.
6. tMSSQLInput_3: Retrive data from source.
In WHERE condition call the context variable.
7. tMSSQLOutput_2: Load data to target table.
8. tMSSQLRow_2 : Update the Incrementaldate1 in IncrementalLoad table with Talend Getdate()
So in next load, only the data with modified date greater than incremental date will be retrieved and loaded.
1. Create a reference table for incremental load in target database (e.g., dbo.Incrementalload with columns TableName and Incrementaldate1). Your table will look like this:
---------------------------------------------------------------
TableName | Incrementaldate1 | Incrementaldate2
----------------------------------------------------------------
Event | 2000-01-01 | NULL
----------------------------------------------------------------
User | 2010-01-01 | NULL
----------------------------------------------------------------
2. Design your Talend job as like below:
3. Create a context variable 'Incrementaldate1'
4. In tMSSQLInput_2: Get incremental date to assign to context variable.
5. tJavaRow_1: Assign incremental date to context variable.
6. tMSSQLInput_3: Retrive data from source.
In WHERE condition call the context variable.
7. tMSSQLOutput_2: Load data to target table.
8. tMSSQLRow_2 : Update the Incrementaldate1 in IncrementalLoad table with Talend Getdate()
So in next load, only the data with modified date greater than incremental date will be retrieved and loaded.
same date is taken from incremental load and again put back in same table . So the date remains same in incremental load table
ReplyDeleteHi Sir,
ReplyDeleteCould you please explain why you added "minus -15 in the below query.
"select incrementaldate1-15 from dbo.incrementalload where ******.
Thanks
shridhar
Sir where is your source table
ReplyDeletethe variable is not passing in the where condition ....any other thing want to predefined
ReplyDelete