Labels

Apache Hadoop (3) ASP.NET (2) AWS S3 (2) Batch Script (3) BigQuery (21) BlobStorage (1) C# (3) Cloudera (1) Command (2) Data Model (3) Data Science (1) Django (1) Docker (1) ETL (7) Google Cloud (5) GPG (2) Hadoop (2) Hive (3) Luigi (1) MDX (21) Mongo (3) MYSQL (3) Pandas (1) Pentaho Data Integration (5) PentahoAdmin (13) Polybase (1) Postgres (1) PPS 2007 (2) Python (13) R Program (1) Redshift (3) SQL 2016 (2) SQL Error Fix (18) SQL Performance (1) SQL2012 (7) SQOOP (1) SSAS (20) SSH (1) SSIS (42) SSRS (17) T-SQL (75) Talend (3) Vagrant (1) Virtual Machine (2) WinSCP (1)

Friday, January 27, 2017

Setting up Python Virtual Environment


Follow below steps to create a virtual environment with python 2.7 version, 
you can replace it with any other version.

    $ pip install virtualenv
    $ virtualenv -p python2.7 mypython27
   Run below command to activate python 2.7 virtualenv

   > Go to mypython27 directory
   $ . bin/activate

Wednesday, January 4, 2017

Determining _PARTITION details in BigQuery Partitioned Table



Run below query to check the partition summary of Bigquery table:

SELECT DATE(_PARTITIONDATE) AS PT, DATE(CURRENT_TIMESTAMP()) , DATE(DATE_ADD(CURRENT_TIMESTAMP(), -1, 'DAY'))
FROM [ProjectId:Dataset.Table]
GROUP BY PT

SELECT project_id, dataset_id, table_id, partition_id
, MSEC_TO_TIMESTAMP(creation_time) Created_date, MSEC_TO_TIMESTAMP(last_modified_time) modified_time
from [ProjectId:Dataset.Table$__PARTITIONS_SUMMARY__]