Kaizen #12 - Bulk Read API

Kaizen #12 - Bulk Read API

Hello everyone!
We hope you all are having a wonderful start in the New Year!
Welcome back to another post in the Kaizen series!

In this post, we will discuss the Bulk Read API in detail.
​
What is the Bulk Read API?
​
The Bulk Read API allows you to export data from a module in Zoho CRM in bulk. The primary difference between the GET records API and the Bulk Read API is the number of records you can retrieve.
While you can get only 200 records per Get records API call, you can fetch 200,000 records per bulk read API call.

When should you use this API?

  • When you want to export more than 200 records through an API call.
  • When you want to perform background processes like migration, data backup, and initial data sync between Zoho CRM and external services.

What is the difference between GET Records API and Bulk Read API?

GET Records
Bulk Read API


You can fetch a maximum of 200 records per API call.


You can fetch a maximum of 200,000 records per API call.

The response is available instantly.

The response is not available instantly; the bulk read job is scheduled, and the status is available after job completion in the callback URL.

The records are available as JSON objects in the response.

The records are available in a downloadable CSV file or ICS file (for events).

What kind of export does the Bulk Read API support?

You can export 
  • Records available in a custom view. You can also apply the criteria on the records available in that custom view to filter the records further. Note that you cannot export records in the Co-owner, Shared by Me, and Shared to Me views.
  • Lookup fields of records in a module using the dot (.) operator.
  • Records in the events module as an ICS file.

How does the Bulk Read API work?

Scheduling a bulk read job encompasses the following steps.
  1. Creating a bulk read job.
  2. Getting the status of the scheduled job.
  3. Downloading the result.
We will now see each of these steps in detail.

Step-1: Creating a bulk read job

This involves making a POST API call to schedule a job to fetch records that match certain criteria, along with the required fields.
When this call is successful, the response returns an id. You can use this id to know the status of the job by polling, or check the job completion in the callback URL.

Request URL: {{api-domain}}/crm/bulk/v2/read
Request Method: POST

Consider an example where you want to export records from the Contacts module.
The below query fetches the records that matches one of the three criteria— 
  • The lead source is Advertisement (or)
  • The owner's last name is Boyle (or)
  • The phone number of the account associated with the contact contains the numbers 99807. 
The fields to be fetched are mentioned in the fields array. To fetch all fields in the module, do not input the fields key.

In this query, Account_Name is the default lookup field in the Contacts module. Owner.last_name returns the last name of the owner of the contact, Account_Name returns the ID and Account_Name.Account_Name returns the name of the account associated with the contact, and Account_Name.Phone returns the phone number of the account associated with the contact.

{
    "callback": {
        "url": "https://www.example.com/callback",
        "method": "post"
    },
    "query": {
        "module": "Contacts",
        "fields": [
            "Last_Name",
            "Owner",
            "Owner.last_name",
            "Account_Name.Account_Name",
            "Account_Name.Phone",
            "Lead_Source",
            "Created_Time"
        ],
        "criteria": {
            "group_operator": "or",
            "group": [
                {
                    "api_name": "Lead_Source",
                    "comparator": "equal",
                    "value": "Advertisement"
                },
                {
                    "api_name": "Owner.last_name",
                    "comparator": "equal",
                    "value": "Boyle"
                },
                {
                    "api_name": "Account_Name.Phone",
                    "comparator": "contains",
                    "value": "99807"
                }
            ]
        },
        "page": 1
    }
}

The below table gives the description of the keys in the sample input.

Key and Data Type
Mandatory
Description
callback
JSON Object
Yes
Contains a valid URL, which should allow the HTTP POST method. The Bulk Read Job's details are posted to this URL on successful completion or failure of the job.
query
JSON Object
Yes
Contains the module, fields that you want to export, criteria based on which you want to export the records, and the page. Refer to the query properties table below for more details.
file_type
String
Yes, when you want to export the events as an ICS file.
Specify the value of this key as "ics" to export all records in the Events module as an ICS file.

query properties

Key and Data Type
Mandatory
Description
module
String
Yes
The API name of the module from which you want to export the records. Specify the module name as "Events" if you want to export the records in the Events module as an ics file.
cvid
String
Yes, when you want to export records in a custom view
The unique ID of the custom view whose records you want to export. You can obtain the cvid from the Custom View Metadata API.
fields
JSON Array
No
The API Name of the fields you want to export from the module. Example: First_Name, Last_Name, Email, Owner.last_name. Do not input this key when you want to export the records in the Events module as an ICS file. The system throws FIELDS_NOT_SUPPORTED error, otherwise.
page
Integer
No
Default value for page is 1. The page value '1' means that the first 200,000 records matching your query will be exported. If you want to fetch the records from the range of 200,001 to 400,000, then you should mention the page as '2'. The maximum value for this key is 500.
criteria
JSON Object
No
To filter the records you want to export. This JSON object contains the API name of the field, comparator, and a group. Refer to the criteria properties table below for more details.

criteria properties

Key and Data Type
Mandatory
Description
api_name
String
Yes, if group and group_operator are not specified.
API name of the field you want to compare. Example: First_Name, Last_Name, Owner.last_name etc,. Example:
"criteria": {
        "api_name": "Lead_Source",
        "comparator": "equal",
        "value": "Advertisement"
    }
value
String or JSON array
Yes, if group and group_operator are not specified.
Positive integer values only.
group_operator
String
Yes, if api_name, comparator & value are not specified.
Logical operators. Supported values are and, or.
group
JSON array
Yes, if api_name, comparator & value are not specified.
Array of criteria objects.
comparator
String
Yes, if group and group_operator are not specified.
Specifies the comparator. Example: equal, greater_than. Refer to the comparators table for more details.

comparators

Data Type
Comparator
Value and Limits
Number (Integer),
Decimal/BigInteger/
Currency/Percent)
equal, not_equal, in, not_in, less_than, less_equal, greater_than, greater_equal
Any number values or ${EMPTY} for empty value. 
Not more than 19 digits for big integer, allows decimal values for decimal and currency fields. In multi-currency enabled accounts, only home currency value is supported.
Text (Email, Phone, URL, Picklist, Multi-select, etc)
equal, not_equal, in, not_in, contains, not_contains, starts_with, ends_with
Any text or ${EMPTY} for empty value.
Not more than 255 characters.
Date
equal, not_equal, in, not_in, between, not_between
Any date value in ISO 8601 format or ${EMPTY} for empty value.
DateTime
equal, not_equal, in, not_in, between, not_between
Any date time value in ISO 8601 format or ${EMPTY} for empty value. Example: 2019-04-01T14:24:04+05:30.
Milliseconds are not supported.
Boolean
equal
true or false.
Lookup
equal, not_equal, in, not_in
Biginteger value of the lookup, ${EMPTY} for empty value, or use the .(dot) operator to establish a relation between two modules.
Example: In the Contacts module, Owner fetches the ID of the Owner, whereas Owner.last_name fetches the last name of the owner. Account_Name fetches the ID of the Account associated with the base module, whereas Account_Name.Phone fetches the phone number of the account associated with the base module.
Text Area (Multi-line)
Not supported
Not supported

The response to the sample query is as follows. You can use the id in the details key to check the job status periodically as explained in step-2.

{
    "data": [
        {
            "status": "success",
            "code": "ADDED_SUCCESSFULLY",
            "message": "Added successfully.",
            "details": {
                "id": "3652397000000646004",
                "operation": "read",
                "state": "ADDED",
                "created_by": {
                    "id": "3652397000000186017",
                    "name": "Patricia Boyle"
                },
                "created_time": "2019-04-01T14:24:04+05:30"
            }
        }
    ],
    "info": {}
}

For more details and examples, refer to the Create Bulk Read Job page of our API guide.


Step-2: Getting the status of the scheduled bulk read job 


The bulk read API supports polling and callback.

Polling 
You can poll (check the status of the scheduled bulk read job) with the job ID you received in the previous step.
If you do not want to poll for the status of the job, you can wait for the system to notify you of job completion on the callback URL provided in the POST request.

Request URL: {{api-domain}}/crm/bulk/v2/read/{job_id}
Request Method: GET

The response contains the status of the scheduled job as either ADDED, IN PROGRESS, or COMPLETED.
When the job is complete, the response contains the result JSON object with the keys page, count, more_records, and download_url.
You can also find the download_url in the callback response from which you can download the zip file containing the CSV or ICS file.
If the more_records key is true, there are more records you need to export. Simply change the value of the key page in the POST request, and schedule another bulk read job to fetch the next set of records.
The download_url contains the URL to download the CSV file. Follow the instructions in step-3 to download the CSV file.
Here's a sample response to the above query when the job is completed.

{
    "data": [
        {
            "id": "3652397000000646004",
            "operation": "read",
            "state": "COMPLETED",
            "result": {
                "page": 1,
                "count": 3,
                "download_url": "/crm/bulk/v2/read/3652397000000646004/result",
                "per_page": 200000,
                "more_records": false
            },
            "query": {
                "fields": [
                    "Last_Name",
                    "Owner",
                    "Owner.last_name",
                    "Account_Name.Account_Name",
                    "Account_Name.Phone",
                    "Lead_Source",
                    "Created_Time"
                ],
                "module": "Contacts",
                "criteria": {
                    "group_operator": "or",
                    "group": [
                        {
                            "api_name": "Lead_Source",
                            "comparator": "equal",
                            "value": "Advertisement"
                        },
                        {
                            "api_name": "Owner.last_name",
                            "comparator": "equal",
                            "value": "Boyle"
                        },
                        {
                            "api_name": "Account_Name.Phone",
                            "comparator": "contains",
                            "value": "99807"
                        }
                    ]
                },
                "page": 1,
                "cvid": "554023000000093005"
            },
            "created_by": {
                "id": "554023000000235011",
                "name": "Patricia Boyle"
            },
            "created_time": "2019-05-09T14:01:24+05:30"
        }
    ]
}
The other possible values for the key state are 
  • ADDED - Indicates that the job is scheduled.
  • IN PROGRESS - Indicates that the job is under processing.
Note that only a completed job will have the download URL in its response.

Callback
If you do not want to poll for the status of the job, you can wait for the system to notify you of job completion on the callback URL provided in the POST request.
  • The state indicates the successful completion ("state":"COMPLETED") or failure ("state":"FAILED") of the job.
  • The callback response will also contain the download URL if the job was completed successfully.
Below is a sample callback response for a completed job.

{
    "job_id": "554023000000568002",
    "operation": "read",
    "state": "COMPLETED",
    "query": {
        "module": "Contacts",
        "criteria": {
            "group": [
                {
                    "api_name": "Lead_Source",
                    "comparator": "equal",
                    "value": "Advertisement"
                },
                {
                    "api_name": "Owner.last_name",
                    "comparator": "equal",
                    "value": "Boyle"
                },
                {
                    "api_name": "Account_Name.Phone",
                    "comparator": "contains",
                    "value": "99807"
                }
            ],
            "group_operator": "or"
        },
        "page": 1,
        "fields": [
            "Last_Name",
            "Owner",
            "Owner.last_name",
            "Account_Name.Account_Name",
            "Account_Name.Phone",
            "Lead_Source",
            "Created_Time"
        ],
        "cvid": "554023000000093005"
    },
    "result": {
        "page": 1,
        "count": 1588,
        "download_url": "/crm/bulk/v2/read/554023000000568002/result",
        "per_page": 200000,
        "more_records": false
    }
}

For more details, refer to the Get Bulk Read Job Details page of our API guide.

This is the final step to complete exporting records from a module in Zoho CRM.
Make a GET request to the download_url you received in the response of step-2.

Step-3: Downloading the CSV file
Request URL: {{api-domain}}/crm/bulk/v2/read/{job_id}/result
Request Method: GET
This request downloads the zip file. Extract it to get the CSV or ICS file.

Limitations

  • Only 10 requests for download are allowed for a one-minute interval. Crossing the limit throws the error with the HTTP code 429. All subsequent downloads will be unsuccessful.
  • After completing the bulk read job, you can access the downloadable file only for a period of one day. After that, you cannot access the file via the endpoints.
  • You can specify a maximum of 200 select fields via an endpoint. If you specify more than that, the system exports all fields available in that module.
  • The maximum value of the page key in the export request body is 500.
  • You can specify a maximum of 25 criteria in a query. This also includes the number of criteria in a custom view or a standard view. For example, if you specify the ID of the standard view "My converted Leads", which uses two criteria, the remaining criteria that you can use in the query will be 23.
  • in and not_in comparators can accept up to 20 values. For example: 'Lead Status' - 'in' - 'Cold,Warm,Hot,Junk,Contacted,Not Contacted,....(20 values)'.
  • You cannot retrieve multi-line, multi-select lookup fields.
  • Custom views in the Activities module, Co-Owner view in the Contacts module, the standard views Shared By Me and Shared To Me are not supported in this API.

For more details, refer to the Limitations page of our API guide.

We hope you found this post useful. Let us know your thoughts in the comment section, or reach out to us at support@zohocrm.com if you have any questions.

Cheers!


    • Sticky Posts

    • Kaizen #197: Frequently Asked Questions on GraphQL APIs

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Kaizen #198: Using Client Script for Custom Validation in Blueprint

      Nearing 200th Kaizen Post – 1 More to the Big Two-Oh-Oh! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Celebrating 200 posts of Kaizen! Share your ideas for the milestone post

      Hello Developers, We launched the Kaizen series in 2019 to share helpful content to support your Zoho CRM development journey. Staying true to its spirit—Kaizen Series: Continuous Improvement for Developer Experience—we've shared everything from FAQs

    • Kaizen #193: Creating different fields in Zoho CRM through API

      🎊 Nearing 200th Kaizen Post – We want to hear from you! Do you have any questions, suggestions, or topics you would like us to cover in future posts? Your insights and suggestions help us shape future content and make this series better for everyone.

    • Client Script | Update - Introducing Commands in Client Script!

      Have you ever wished you could trigger Client Script from contexts other than just the supported pages and events? Have you ever wanted to leverage the advantage of Client Script at your finger tip? Discover the power of Client Script - Commands! Commands

    • Recent Topics

    • SalesIQとPageSenseの利用について

      初めての投稿で場違いだったらすいません。 弊社ではSalesIQを運用しているのですが、追加でPageSenseの導入もしたいと現場からの声があります。 両サービスともクッキー同意バナーが必要なサービスなのですが 弊社では同意無しに情報はとりませんという方針なので 2つ入れると2つバナーを出す必要がでてきます・・・ 両サービスを運用されてる方があれば運用状況とか教えてほしいです。 PageSenseについては詳細まで機能を理解してないなかでの質問です。

    • How to integrate Zoho Forms with Zoho CRM on Standard Plan

      Hello Zoho Support Team, I am using the Standard Zoho Forms plan (USD 30/user) and I would like to integrate Zoho Forms with Zoho CRM so that certain fields in my forms can be automatically prefilled using data from Deals in CRM. Specifically, I want

    • CRM : Function to add user name to text field

      I have a lookup field in a module that is linked to the CRM users so we can assign a Project Lead to the customer. Sadly Zoho Marketing Automation doesn't sync Lookup fields so I need to extract information from the lookup to text fields: Lookup field

    • Export PDF File Name

      Is it possible to change the default Zoho .pdf naming scheme for inventory items like quotations? Would like to use the the Subject as the default quote name. Is this possible?

    • How to change the from address from 'no reply' for an email template in CRM

      Hi, We have our CRM set up with the from field as sales@XXX. I have just created a series of email templates and sent a test and they are sending from noreply@zoho I have tried searching for how to change the email template but don't have the options

    • Zoho CRM Client Script - SetCriteria in lookup Field

      Hello All One More Zoho CRM Client Script Tips & Trick. Now you can Set the Criteria on Your lookup in zoho CRM, It Comes With a Create Page, Edit Page, and Details Page (Standard). Example:- We have a Room Module that includes Room Name, Status, Campus,

    • Kaizen #71 - Client Script ZDKs for Detail (Canvas) Page

      Hello everyone! Welcome back to another interesting Kaizen post. In this post, we can discuss Client Script ZDKs support for Detail (Canvas) Page. What is Detail (Canvas) Page? A Detail(Canvas) Page allows you to customize the record detail page to your

    • how to use validation rules in subform

      Is it possible to use validation rules for subforms? I tried the following code: entityMap = crmAPIRequest.toMap().get("record"); sum = 0; direct_billing = entityMap.get("direct_billing_details"); response = Map(); for each i in direct_billing { if(i.get("type")

    • Add Custom Reports To Dashboard or Home Tab

      Hi there, I think it would be great to be able to add our custom reports to the Home Tab or Dashboards. Thanks! Chad

    • Rich-text fields in Zoho CRM

      Hello everyone, We're thrilled to announce an important enhancement that will significantly enhance the readability and formatting capabilities of your information: rich text options for multi-line fields. With this update, you can now enjoy a more versatile

    • Zoho Reports Duplicating Entries

      I have a custom costing tab with a table where we entre invoices. These are under a Heading (PO Subject) and notes added in the form with different line items. In the reports, I have organised the report to group per PO Subject, with the total of the

    • Validation Rule Not Working for Mandatory Field in Zoho Blueprint

      As a Zoho user, we created a validation rule for a specific field. However, we noticed that when we made the same field mandatory within a Blueprint, the validation rule we defined did not work. When we reported this issue to Zoho Support, they stated

    • Notes Issues

      Been having issues with Notes in the CRM. Yesterday it wasn't showing the notes, but it got resolved after a few minutes., Now I have been having a hard time saving notes the whole day. Notes can't be saved by the save button. it's grayed out or not grayed

    • Export from Contacts module to Products module in Zoho CRM

      Good afternoon, I would like to send a number of contact info from the Contacts module into the customized module (tickets to an event) in one operation. I have selected several contacts in the Contact module (people who I have labelled as people I want

    • Zoho Commerce

      Hi, I have zoho one and use Zoho Books. I am very interested in Zoho Commerce , especially with how all is integrated but have a question. I do not want my store to show prices for customers that are not log in. Is there a way to hide the prices if not

    • Can’t receive emailI c

      I have generated a basic for but when I submit it I don’t get a email, I’ve been in the settings and tested me email, all appears correct, can you please help me

    • Data Capture for Historical Activity (Especially One Lead Downloading Variois reports without Overwriting the info)

      Is there a better way in Zoho CRM to capture and archive a lead’s historical activity—specifically whenever they download reports—so that the data is stored without being overwritten?”

    • Client Script - Updating Field Value in Detail Page of a Lead

      Hello, I'm trying to use Client Script To enrich some data of the Lead when one of my User fill the "City" field in the detail page of the Lead. This is my Script: log (value); var response = ZDK.Apps.CRM.Functions.execute("getInfoCitta", { "nomeCitta":

    • Auto shapes in Zoho sheet.

      Does Zoho sheet supports inserting auto shapes (rectangle, circle...). I did not see any option to do so.  If its not supported currently, is there any plans on bring in this features. Any timelines ?

    • I Can't Clone Webinar that I Co-Organize

      How do i get our account admin to give me permission to clone our webinars? I am a co-organizer

    • I want to Make the due date of the project appears automatically

      hello everyone I want to know if the zoho projects app can add automatically the END DATE of the project I added the duration, start after dates of each tasks in the project template & the starting date of the project itself and the end date of it doesn't

    • Introducing Profile Summary: Faster Candidate Insights with Zia

      We’re excited to launch Profile Summary, a powerful new feature in Zoho Recruit that transforms how you review candidate profiles. What used to take minutes of resume scanning can now be assessed in seconds—thanks to Zia. A Quick Example Say you’re hiring

    • How to get the call recording external ID via desk API

      I have enabled phonbridge integration with Zoom Call. I am trying to access the call recording in Zoom by calling Zoom API. I have built a Desk workflow to trigger on a new call, to call a custom function. when calling the API, the response doesn't contain

    • Can't View Project Names in Mobile App

      I can't view project names on PO's in the app, nor can I add that as a viewable PDF field in inventory on the computer. I've attached screenshots showing that in the mobile version whether you are on the PO, editing the PO, or viewing the PO line items,

    • How do you print a refund check to customer?

      Maybe this is a dumb question, but how does anyone print a refund check to a customer? We cant find anywhere to either just print a check and pick a customer, or where to do so from a credit note.

    • Notebook

      I have purchased the monthly pro subscription of Notebook. But it does not support my XP-Pen to write something in it. So it is not useful to me. Hence I am requesting you to help me to discontinue this subscription.

    • Domain Mapping & Image Publishing Issues on Zoho Sites

      Hello, I am facing two issues with my Zoho Sites account: 1. Images not visible after publishing. 2. Domain mapping error: "Domain already exists". I am a paid customer. Please connect me with Live Chat Support or Zoho Assist so I can show my issue

    • Prevent duplicate with custom fields?

      I was wondering something about custom field/custom modules in Zoho Desk. For some reason you can make a custom field mandatory but not unique? For example, if I create a custom module to manage equipment and renewal and make a field serial number no

    • Round robin

      Hi, I'm trying to set up a round robin to automatically distribute tickets between agents in my team but only those tickets that are not otherwise distributed by other workflows or direct assignments. Is that possible and if so which criteria should I

    • "Is Zoho CRM customer" vs "Is linked with Zoho CRM"

      Recently while building a Flow, I was setting up a Decision action following a Zoho Invoice Fetch record action. There were 2 choices that I had not seen as something I could manually action in Zoho Invoice: "Is Zoho CRM customer" and "Is linked with

    • Two currencies

      More and more I am finding that internattional payments' fees are unpredictable. I would like, on my invoices that are in a foreign currency (eg. USD$ or EUR€) for there to be a GBP£ TOTAL display alongside the invoice's currency total. This would make

    • Automatic Matching from Bank Statements / Feeds

      Is it possible to have transactions from a feed or bank statement automatically match when certain criteria are met? My use case, which is pretty broadly applicable, is e-commerce transactions for merchant services accounts (clearing accounts). In these

    • Generate leads from instagram

      hello i have question. If connect instagram using zoho social, it is possible to get lead from instagram? example if someone send me direct message or comment on my post and then they generate to lead

    • Zoho PDF editor has a lot of issues.

      Zoho PDF editor needs a lot of work. It hangs and glitches a lot. Deletes annotations and clearings randomly.

    • Where is the desktop app for Zoho Projects???

      As a project manager, I need a desktop app for the projects I manage. Yes, there's the web app, which is AWESOME for cross browser and platform compatibility... but I need a real desktop app for Projects that allow me to enter offline information where

    • How to Automate Monthly PDF Reports with Filters in Zoho Creator

      Hi everyone, I’m trying to build an automated monthly reporting process in Zoho Creator and would appreciate suggestions or best practices from anyone who has done something similar. What I’m trying to do: I have a form called New_Customer with fields

    • Feedback: Streamlining Note Management in Zoho Notebook

      Dear Team/Support, I would like to share some feedback regarding the note management system that could help improve usability and accessibility for users like myself. Notebook 1 (screenshot attached): Currently, the system does not allow selecting and

    • showing Limit exceeded

      Good afternoon...trust you're good. I've been having issues working with but it's not responding. it's showing Limit exceeded, sorry it seems like too many people are working on the sheet right now please try again later. meanwhile no one is working on

    • Converting Sales Order to Invoice via API; Problem with decimal places tax

      We are having problems converting a Sales Order to an Invoice via API Call. The cause of the issue is, that the Tax value in a Sales Order is sometimes calculated with up to 16 decimal places (e.g. 0.8730000000000001). The max decimal places allowed in

    • Sorting columns in Zoho Projects

      Hi, In project management best practice, sorting columns (ascending, descending) is an important tool. Sorting dates to see the order of tasks starting, sorting on priority or even on planned hours is a must for an efficient project control. Currently,

    • Next Page