Google Cloud BQ Command Line Data Load

'bq' is command line tool provided by Google Cloud Platform to access bigquery table and perform operations like DDL, DML, etc. 

Refer http://mahadevanrv.blogspot.in/2016/06/install-google-bigquery-command-line.html for GCloud installation.

Data load using bq involves three types:

1. Empty (Default): It writes data into an empty table, if data already exists it throws error.

bq query  ---n=1000 --destination_table=<table_name> 'SELECT * FROM [project:dataset.source_table];'

2. Replace: It replace a current table with newly obtained data output. It involves loss of existing data in a destination table. Use it wisely to perform incremental load which involves update and inserts.

bq query  ---replace --destination_table=<table_name> 'SELECT * FROM [project:dataset.source_table];'

3. Append: It appends new records to the existing table. If same command is executed more than one time it will create duplicate records. Can be used for incremental load which involves only data insert.

bq query  ---append_table --destination_table=<table_name> 'SELECT * FROM [project:dataset.source_table];'