Consider you have a text file with User details, the text file doesn't have any data about created or modified date, in that case when you need to move this data to OLEDB database or to Data warehouse you may need stage modified date and Checksum value to improve the performance of the data load.
In this case, make sure you pass StageModifiedDate and CheckSum value to your staging tables, so those values helps you to perform incremental load to target main table.
To do this with Talend, design your job as like below:
In tMap, create a new variable 'stagemodifieddate' as shown below (make sure you use date format and output date pattern provide the required date format), this will add new output StageModifiedDate to schema.
To get CheckSum output, add tAddCRCRow component, and check all those columns that may undergo update, in below example I checked all. By doing so a new CRC column will be generated.
To verify tAddCRCRow, click on Edit Schema, you can see new column CRC
Now, in text output you can add these two columns along with other data.
In this case, make sure you pass StageModifiedDate and CheckSum value to your staging tables, so those values helps you to perform incremental load to target main table.
To do this with Talend, design your job as like below:
In tMap, create a new variable 'stagemodifieddate' as shown below (make sure you use date format and output date pattern provide the required date format), this will add new output StageModifiedDate to schema.
To get CheckSum output, add tAddCRCRow component, and check all those columns that may undergo update, in below example I checked all. By doing so a new CRC column will be generated.
To verify tAddCRCRow, click on Edit Schema, you can see new column CRC
Now, in text output you can add these two columns along with other data.
No comments:
Post a Comment