Google BigQuery is a serverless, highly scalable data warehouse that is entirely cloud hosted. Because of this you can use its columnar storage to run SQL queries on your chosen quantity of data. This post will go over how you can import data from Google BigQuery and go through the few easy steps needed to execute a data migration.
Data Governor Online is a data orchestration tool with over 80 native connection types available. It takes only a few uncomplicated steps to migrate and transform your data without using code. Using our Google BigQuery Connector, we support data migration as a source and target, and the execution of arbitrary Google BigQuery SQL statements.
Data Governor Online’s connectivity to many other data sources means you have a greater set of data to perform analytics with, as anything that is a source in Data Governor Online can be migrated into Google BigQuery. You can also move data from BigQuery using any of our connectors that is a target. Another feature is that you can automatically schedule migrations both to and from Google BigQuery, set task dependencies and perform mutations and SQL Statements on your databases. Data Governor Online provides end-to-end ELT and so a standard ELT workload can be completely integrated into one job.
How to import data using Data Governor Online’s BigQuery Connector
The first step is to configure your connector for your own BigQuery database. You will need to add a new connection that you can find in the Connections Settings menu at the top right of your page.
You will need an Authentication file and for this you will need to create or use a Google service account and set an environment variable. Further details can be found here.
Once you have configured this connection for your BigQuery database, you can add a new Project (or a new Job in an existing Project) for your Google BigQuery tasks. Here you can add your data migration and SQL Statement tasks.
This example will import data from Google BigQuery using a data migration task. So as shown below, I have created a new Data Migration task and named it.
You then select Google BigQuery as your Source Connection from the drop-down menu, and in this window you also pick your Target Connection and Schemas.
The next step is to choose to migrate data from Tables, Views or a custom SQL Query. For this example, we will choose to migrate tables. You can select them individually or just click the Select all option above it. It will display the tables you have selected below under the Target table heading.
You can then submit the task, and choose to run it manually or create a one-time or recurring schedule at the time that you would like it to run. Additionally, you can still edit this task once submitted.
In order to demonstrate the transferred data, in the GCP console we can see the data from our source table compared with next image of the target table.
In the following image, you can see the resulting imported data in SQL Server Management Studio.
With that complete we have just imported data from Google BigQuery by using our clear and simple data orchestration tool. In our next blog post we will detail how to migrate data into Google BigQuery.