Import data from Google BigQuery

Zoho DataPrep supports importing data from Google BigQuery. BigQuery is a fully managed enterprise data warehouse that helps you manage organization data. With this connector, you can streamline your ETL workflows, enable smooth data movement, and simplify data integration between Google BigQuery and Zoho DataPrep.

Prerequisite

The Project Id and the Dataset location details are required to fetch the data from Google BigQuery.

Login to your Google account.
Choose the Project which has the datasets that you wish to import. The Project ID is displayed in the Project Selector at the top left of the BigQuery console, where you can select or switch between different projects.
Click any table name to view the details. The Dataset info pane will list the details about the Dataset Location. Refer to BigQuery locations article to learn more.

Please ensure the following conditions are met before proceeding with connecting to your Google BigQuery

1. Provide database read permissions to the Zoho DataPrep server.

2. Ensure you have the correct login credentials for your database. As a best practice, try connecting to your target database from the Zoho DataPrep server, using the native database management software available for the respective database.

To import data from Google BigQuery

1. Create a pipeline or open an existing pipeline from the Home Page, Pipelines tab or Workspaces tab and click the Add data option. You can also click the Import data option under the Workspaces tab to import data.

Info: You can also click the Import data

icon at the top of the pipeline builder and bring data from multiple sources into the pipeline.

2. Choose the Cloud databases category from the left pane and click Google BigQuery. You can also directly search the option in the search box.

3. Select an account from the saved data connections, or add a new account using the Add new option.

4. To add a new connection, enter the Project ID and click the Sign in with Google button.

Note: Project ID can be found on your Google cloud console. Google BigQuery authorizes access to resources based on verified identity.

5. After signing in with your google account, you can click Continue to allow Zoho DataPrep to access data from your Google BigQuery project for the first time.

Note: The connection configuration is securely saved for importing data from the Google BigQuery project in the future.

6. Select the tables that need to be imported. Click Import.

7. You can also use SQL query to select and import data.

The incremental fetch option is not available when the data is imported using a query from databases. Click here to know more about incremental fetch from cloud database.

8. Click Import to begin importing data from your Google BigQuery project.

9. Once you complete importing data, the Pipeline builder page opens and you can start applying transforms to the ETL pipeline. You can also right-click the stage and choose the Prepare data option to prepare your data using various transforms in the DataPrep Studio page. Click here to know more about the transforms.

10. Once you are done creating your data flow and applying necessary transforms in your stages, you can right-click a stage and add a destination to complete your data flow.

Note: After adding a destination to the ETL pipeline, you can try executing your pipeline using a manual run at first. Once you make sure manual run works, you can then set up schedule to automate the pipeline and data movement. Learn about the different types of runs here.

While configuring the Schedule, Backfill, Manual reload, Webhooks, or Zoho Flow, the import configuration needs to be mandatorily setup for all the sources. Without setting up the import configuration, the run cannot be saved. Click here to know more about how to set up import configuration.

11. After configuring a run, a pipeline job will be created at the run time. You can view the status of a job with the granular details in the Job summary. Click here to know more about the job summary.

To edit the Google BigQuery connection

DataPrep saves your data connections to avoid the hassle of keying in the credentials every time you need to connect to a data source or destination. You can always edit the saved data connection and update them with new parameters or credentials using the Edit connection option.

1. Click Saved connections from the left pane under the Choose your data source box while creating a new dataset.

2. You can manage your saved connections right from the data import screen. Click the (ellipses) icon to share, edit, view the connection overview, or remove the connection.

3. Click the Edit connection option. You can update the Project ID in the saved connection and click Update.

FAQs

1. I’m getting the error, “Loading tables has failed. Error connecting to Google BigQuery. Error details: reauth related error (invalid_rapt)” while connecting Google BigQuery in Zoho DataPrep. How can I resolve this?

Zoho DataPrep connects to Google BigQuery using an access token that expires regularly. We use a refresh token to get a new access token automatically, so the connection stays active without requiring you to sign in again. The problem is that the refresh token can expire, be revoked, or become invalid based on Google Cloud settings or admin policies around token expiration. In that case, we can't refresh the token automatically, causing connection errors and forcing you to re-authenticate the existing connection.

To avoid these errors, please make sure to add Zoho DataPrep as a trusted app, which can prevent frequent token revocations. Click here to learn more.