Labels

Apache Hadoop (3) ASP.NET (2) AWS S3 (2) Batch Script (3) BigQuery (21) BlobStorage (1) C# (3) Cloudera (1) Command (2) Data Model (3) Data Science (1) Django (1) Docker (1) ETL (7) Google Cloud (5) GPG (2) Hadoop (2) Hive (3) Luigi (1) MDX (21) Mongo (3) MYSQL (3) Pandas (1) Pentaho Data Integration (5) PentahoAdmin (13) Polybase (1) Postgres (1) PPS 2007 (2) Python (13) R Program (1) Redshift (3) SQL 2016 (2) SQL Error Fix (18) SQL Performance (1) SQL2012 (7) SQOOP (1) SSAS (20) SSH (1) SSIS (42) SSRS (17) T-SQL (75) Talend (3) Vagrant (1) Virtual Machine (2) WinSCP (1)

Monday, April 25, 2016

SSAS MDX: Converting a Dimension Value to Calculated Measure



I have a dimensional attribute "Target TOS Monthly", in order to convert this dimensional attribute to measure added a calculated measure with below code.


SUM([Account].[Target TOS Monthly].[Target TOS Monthly].MEMBERS, [Account].[Target TOS Monthly].MEMBER_VALUE)

Monday, April 11, 2016

AWS S3 to Redshift Incremental load


Performing incremental load between AWS S3 and Redshift table.

Step 1: Create a stageing table in AWS Redshift database similar to your  target table. Execute below command to load data from S3 to Staging table.


QUERY 1

copy stage.logtracking from 's3://Bucket/FolderName'
credentials aws_access_key_id="S3Accesskey";aws_secret_access_key="S3Secretkey"
delimiter   '|'  ACCEPTINVCHARS iGNOREHEADER 1;


 Step 2: Run the below query to perform incremental load similar to UPSERT/MERGE statement

QUERY 2

-- Start a new transaction
BEGIN TRANSACTION;

-- Update the target table using an inner join with the staging table
-- The join includes a redundant predicate to collocate on the distribution key
-- A filter on saletime enables a range-restricted scan on SALES

UPDATE ga.logtracking
SET  usertype= stage.logtracking.usertype, sessioncount= stage.logtracking.sessioncount
   , dayssincelastsession = stage.logtracking.dayssincelastsession, userdefinedvalue = stage.logtracking.userdefinedvalue
   , date = CAST(stage.logtracking.date AS DATETIME)
   , userid = stage.logtracking.userid
   , users = stage.logtracking.users
   , newusers = stage.logtracking.newusers
   , percentnewsessions = stage.logtracking.percentnewsessions
   , sessionsperuser = stage.logtracking.sessionsperuser
   , timeonsite = stage.logtracking.timeonsite
   , checksum = stage.logtracking.checksum
FROM stage.logtracking
WHERE ga.logtracking.checksum = stage.logtracking.checksum;

-- Delete matching rows from the staging table
-- using an inner join with the target table

DELETE FROM stage.logtracking
USING ga.logtracking
WHERE ga.logtracking.checksum = stage.logtracking.checksum;

-- Insert the remaining rows from the staging table into the target table
INSERT INTO ga.logtracking
SELECT usertype, sessioncount, dayssincelastsession, userdefinedvalue, CAST(date AS DATETIME), userid
, users, newusers, percentnewsessions, sessionsperuser, timeonsite, checksum
FROM stage.logtracking;

-- End transaction and commit
END TRANSACTION;