Kaizen #12 - Bulk Read API

Kaizen #12 - Bulk Read API

Hello everyone!
We hope you all are having a wonderful start in the New Year!
Welcome back to another post in the Kaizen series!

In this post, we will discuss the Bulk Read API in detail.

What is the Bulk Read API?

The Bulk Read API allows you to export data from a module in Zoho CRM in bulk. The primary difference between the GET records API and the Bulk Read API is the number of records you can retrieve.
While you can get only 200 records per Get records API call, you can fetch 200,000 records per bulk read API call.

When should you use this API?

  • When you want to export more than 200 records through an API call.
  • When you want to perform background processes like migration, data backup, and initial data sync between Zoho CRM and external services.

What is the difference between GET Records API and Bulk Read API?

GET Records
Bulk Read API


You can fetch a maximum of 200 records per API call.


You can fetch a maximum of 200,000 records per API call.

The response is available instantly.

The response is not available instantly; the bulk read job is scheduled, and the status is available after job completion in the callback URL.

The records are available as JSON objects in the response.

The records are available in a downloadable CSV file or ICS file (for events).

What kind of export does the Bulk Read API support?

You can export 
  • Records available in a custom view. You can also apply the criteria on the records available in that custom view to filter the records further. Note that you cannot export records in the Co-owner, Shared by Me, and Shared to Me views.
  • Lookup fields of records in a module using the dot (.) operator.
  • Records in the events module as an ICS file.

How does the Bulk Read API work?

Scheduling a bulk read job encompasses the following steps.
  1. Creating a bulk read job.
  2. Getting the status of the scheduled job.
  3. Downloading the result.
We will now see each of these steps in detail.

Step-1: Creating a bulk read job

This involves making a POST API call to schedule a job to fetch records that match certain criteria, along with the required fields.
When this call is successful, the response returns an id. You can use this id to know the status of the job by polling, or check the job completion in the callback URL.

Request URL: {{api-domain}}/crm/bulk/v2/read
Request Method: POST

Consider an example where you want to export records from the Contacts module.
The below query fetches the records that matches one of the three criteria— 
  • The lead source is Advertisement (or)
  • The owner's last name is Boyle (or)
  • The phone number of the account associated with the contact contains the numbers 99807. 
The fields to be fetched are mentioned in the fields array. To fetch all fields in the module, do not input the fields key.

In this query, Account_Name is the default lookup field in the Contacts module. Owner.last_name returns the last name of the owner of the contact, Account_Name returns the ID and Account_Name.Account_Name returns the name of the account associated with the contact, and Account_Name.Phone returns the phone number of the account associated with the contact.

{
    "callback": {
        "url": "https://www.example.com/callback",
        "method": "post"
    },
    "query": {
        "module": "Contacts",
        "fields": [
            "Last_Name",
            "Owner",
            "Owner.last_name",
            "Account_Name.Account_Name",
            "Account_Name.Phone",
            "Lead_Source",
            "Created_Time"
        ],
        "criteria": {
            "group_operator": "or",
            "group": [
                {
                    "api_name": "Lead_Source",
                    "comparator": "equal",
                    "value": "Advertisement"
                },
                {
                    "api_name": "Owner.last_name",
                    "comparator": "equal",
                    "value": "Boyle"
                },
                {
                    "api_name": "Account_Name.Phone",
                    "comparator": "contains",
                    "value": "99807"
                }
            ]
        },
        "page": 1
    }
}

The below table gives the description of the keys in the sample input.

Key and Data Type
Mandatory
Description
callback
JSON Object
Yes
Contains a valid URL, which should allow the HTTP POST method. The Bulk Read Job's details are posted to this URL on successful completion or failure of the job.
query
JSON Object
Yes
Contains the module, fields that you want to export, criteria based on which you want to export the records, and the page. Refer to the query properties table below for more details.
file_type
String
Yes, when you want to export the events as an ICS file.
Specify the value of this key as "ics" to export all records in the Events module as an ICS file.

query properties

Key and Data Type
Mandatory
Description
module
String
Yes
The API name of the module from which you want to export the records. Specify the module name as "Events" if you want to export the records in the Events module as an ics file.
cvid
String
Yes, when you want to export records in a custom view
The unique ID of the custom view whose records you want to export. You can obtain the cvid from the Custom View Metadata API.
fields
JSON Array
No
The API Name of the fields you want to export from the module. Example: First_Name, Last_Name, Email, Owner.last_name. Do not input this key when you want to export the records in the Events module as an ICS file. The system throws FIELDS_NOT_SUPPORTED error, otherwise.
page
Integer
No
Default value for page is 1. The page value '1' means that the first 200,000 records matching your query will be exported. If you want to fetch the records from the range of 200,001 to 400,000, then you should mention the page as '2'. The maximum value for this key is 500.
criteria
JSON Object
No
To filter the records you want to export. This JSON object contains the API name of the field, comparator, and a group. Refer to the criteria properties table below for more details.

criteria properties

Key and Data Type
Mandatory
Description
api_name
String
Yes, if group and group_operator are not specified.
API name of the field you want to compare. Example: First_Name, Last_Name, Owner.last_name etc,. Example:
"criteria": {
        "api_name": "Lead_Source",
        "comparator": "equal",
        "value": "Advertisement"
    }
value
String or JSON array
Yes, if group and group_operator are not specified.
Positive integer values only.
group_operator
String
Yes, if api_name, comparator & value are not specified.
Logical operators. Supported values are and, or.
group
JSON array
Yes, if api_name, comparator & value are not specified.
Array of criteria objects.
comparator
String
Yes, if group and group_operator are not specified.
Specifies the comparator. Example: equal, greater_than. Refer to the comparators table for more details.

comparators

Data Type
Comparator
Value and Limits
Number (Integer),
Decimal/BigInteger/
Currency/Percent)
equal, not_equal, in, not_in, less_than, less_equal, greater_than, greater_equal
Any number values or ${EMPTY} for empty value. 
Not more than 19 digits for big integer, allows decimal values for decimal and currency fields. In multi-currency enabled accounts, only home currency value is supported.
Text (Email, Phone, URL, Picklist, Multi-select, etc)
equal, not_equal, in, not_in, contains, not_contains, starts_with, ends_with
Any text or ${EMPTY} for empty value.
Not more than 255 characters.
Date
equal, not_equal, in, not_in, between, not_between
Any date value in ISO 8601 format or ${EMPTY} for empty value.
DateTime
equal, not_equal, in, not_in, between, not_between
Any date time value in ISO 8601 format or ${EMPTY} for empty value. Example: 2019-04-01T14:24:04+05:30.
Milliseconds are not supported.
Boolean
equal
true or false.
Lookup
equal, not_equal, in, not_in
Biginteger value of the lookup, ${EMPTY} for empty value, or use the .(dot) operator to establish a relation between two modules.
Example: In the Contacts module, Owner fetches the ID of the Owner, whereas Owner.last_name fetches the last name of the owner. Account_Name fetches the ID of the Account associated with the base module, whereas Account_Name.Phone fetches the phone number of the account associated with the base module.
Text Area (Multi-line)
Not supported
Not supported

The response to the sample query is as follows. You can use the id in the details key to check the job status periodically as explained in step-2.

{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully.",
            "details": {
                "id": "3652397000000646004",
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "3652397000000186017",
                    "name": "Patricia Boyle"
                },
                "created_time": "2019-04-01T14:24:04+05:30"
            }
        }
    ],
    "info": {}
}

For more details and examples, refer to the Create Bulk Read Job page of our API guide.


Step-2: Getting the status of the scheduled bulk read job 


The bulk read API supports polling and callback.

Polling 
You can poll (check the status of the scheduled bulk read job) with the job ID you received in the previous step.
If you do not want to poll for the status of the job, you can wait for the system to notify you of job completion on the callback URL provided in the POST request.

Request URL: {{api-domain}}/crm/bulk/v2/read/{job_id}
Request Method: GET

The response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url.
You can also find the download_url in the callback response from which you can download the zip file containing the CSV or ICS file.
If the more_records key is true, there are more records you need to export. Simply change the value of the key page in the POST request, and schedule another bulk read job to fetch the next set of records.
The download_url contains the URL to download the CSV file. Follow the instructions in step-3 to download the CSV file.
Here's a sample response to the above query when the job is completed.

{
    "data": [
        {
            "id": "3652397000000646004",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "count": 3,
                "download_url": "/crm/bulk/v2/read/3652397000000646004/result",
                "per_page": 200000,
                "more_records": false
            },
            "query": {
                "fields": [
                    "Last_Name",
                    "Owner",
                    "Owner.last_name",
                    "Account_Name.Account_Name",
                    "Account_Name.Phone",
                    "Lead_Source",
                    "Created_Time"
                ],
                "module": "Contacts",
                "criteria": {
                    "group_operator": "or",
                    "group": [
                        {
                            "api_name": "Lead_Source",
                            "comparator": "equal",
                            "value": "Advertisement"
                        },
                        {
                            "api_name": "Owner.last_name",
                            "comparator": "equal",
                            "value": "Boyle"
                        },
                        {
                            "api_name": "Account_Name.Phone",
                            "comparator": "contains",
                            "value": "99807"
                        }
                    ]
                },
                "page": 1,
                "cvid": "554023000000093005"
            },
            "created_by": {
                "id": "554023000000235011",
                "name": "Patricia Boyle"
            },
            "created_time": "2019-05-09T14:01:24+05:30"
        }
    ]
}
The other possible values for the key state are 
  • ADDED - Indicates that the job is scheduled.
  • IN PROGRESS - Indicates that the job is under processing.
Note that only a completed job will have the download URL in its response.

Callback
If you do not want to poll for the status of the job, you can wait for the system to notify you of job completion on the callback URL provided in the POST request.
  • The state indicates the successful completion ("state":"COMPLETED") or failure ("state":"FAILED") of the job.
  • The callback response will also contain the download URL if the job was completed successfully.
Below is a sample callback response for a completed job.

{
    "job_id": "554023000000568002",
    "operation": "read",
    "state": "COMPLETED",
    "query": {
        "module": "Contacts",
        "criteria": {
            "group": [
                {
                    "api_name": "Lead_Source",
                    "comparator": "equal",
                    "value": "Advertisement"
                },
                {
                    "api_name": "Owner.last_name",
                    "comparator": "equal",
                    "value": "Boyle"
                },
                {
                    "api_name": "Account_Name.Phone",
                    "comparator": "contains",
                    "value": "99807"
                }
            ],
            "group_operator": "or"
        },
        "page": 1,
        "fields": [
            "Last_Name",
            "Owner",
            "Owner.last_name",
            "Account_Name.Account_Name",
            "Account_Name.Phone",
            "Lead_Source",
            "Created_Time"
        ],
        "cvid": "554023000000093005"
    },
    "result": {
        "page": 1,
        "count": 1588,
        "download_url": "/crm/bulk/v2/read/554023000000568002/result",
        "per_page": 200000,
        "more_records": false
    }
}

For more details, refer to the Get Bulk Read Job Details page of our API guide.

This is the final step to complete exporting records from a module in Zoho CRM.
Make a GET request to the download_url you received in the response of step-2.

Step-3: Downloading the CSV file
Request URL: {{api-domain}}/crm/bulk/v2/read/{job_id}/result
Request Method: GET
This request downloads the zip file. Extract it to get the CSV or ICS file.

Limitations

  • Only 10 requests for download are allowed for a one-minute interval. Crossing the limit throws the error with the HTTP code 429. All subsequent downloads will be unsuccessful.
  • After completing the bulk read job, you can access the downloadable file only for a period of one day. After that, you cannot access the file via the endpoints.
  • You can specify a maximum of 200 select fields via an endpoint. If you specify more than that, the system exports all fields available in that module.
  • The maximum value of the page key in the export request body is 500.
  • You can specify a maximum of 25 criteria in a query. This also includes the number of criteria in a custom view or a standard view. For example, if you specify the ID of the standard view "My converted Leads", which uses two criteria, the remaining criteria that you can use in the query will be 23.
  • in and not_in comparators can accept up to 20 values. For example: 'Lead Status' - 'in' - 'Cold,Warm,Hot,Junk,Contacted,Not Contacted,....(20 values)'.
  • You cannot retrieve multi-line, multi-select lookup fields.
  • Custom views in the Activities module, Co-Owner view in the Contacts module, the standard views Shared By Me and Shared To Me are not supported in this API.

For more details, refer to the Limitations page of our API guide.

We hope you found this post useful. Let us know your thoughts in the comment section, or reach out to us at support@zohocrm.com if you have any questions.

Cheers!


    Access your files securely from anywhere

        Zoho Developer Community




                                  Zoho Desk Resources

                                  • Desk Community Learning Series


                                  • Digest


                                  • Functions


                                  • Meetups


                                  • Kbase


                                  • Resources


                                  • Glossary


                                  • Desk Marketplace


                                  • MVP Corner


                                  • Word of the Day



                                      Zoho Marketing Automation


                                              Manage your brands on social media



                                                    Zoho TeamInbox Resources

                                                      Zoho DataPrep Resources



                                                        Zoho CRM Plus Resources

                                                          Zoho Books Resources


                                                            Zoho Subscriptions Resources

                                                              Zoho Projects Resources


                                                                Zoho Sprints Resources


                                                                  Qntrl Resources


                                                                    Zoho Creator Resources



                                                                        Zoho Campaigns Resources


                                                                          Zoho CRM Resources

                                                                          • CRM Community Learning Series

                                                                            CRM Community Learning Series


                                                                          • Kaizen

                                                                            Kaizen

                                                                          • Functions

                                                                            Functions

                                                                          • Meetups

                                                                            Meetups

                                                                          • Kbase

                                                                            Kbase

                                                                          • Resources

                                                                            Resources

                                                                          • Digest

                                                                            Digest

                                                                          • CRM Marketplace

                                                                            CRM Marketplace

                                                                          • MVP Corner

                                                                            MVP Corner





                                                                              Design. Discuss. Deliver.

                                                                              Create visually engaging stories with Zoho Show.

                                                                              Get Started Now