Bulk Write API using PHP SDK v5 Part II

Bulk Write API using PHP SDK v5 Part II

Welcome to another exciting week of Kaizen! This week, we will continue our discussion on Bulk Write API.  In last week's post, we covered the features of Bulk Write API, and explained in detail the first two steps involved in using Bulk Write API. Today, we'll discuss how to create a bulk write job, fetch the job status, and download the results.

1. Prepare the CSV file

2. Upload your zip file

3. Create a Bulk Write job

After uploading the file and obtaining the file ID, the next step is to create a bulk write job. For this, you will require the file ID, callback URL, and the field API names to perform one of the three operations: insert, update, or upsert. Upon a successful API call, the response will provide a job ID, with which you can check the job status in the next step.

The following are the keys you need to set.
  • callback (JSON object, optional) :  The CallBack object defines the callback URL and method for receiving notifications or updates about the job's status.
  • character_encoding (string, optional) : Specify the encoding to be used for the data being written.
  • operation (string, mandatory) : Specify the type of operation to be performed in the bulk write job. Possible values are insert, update and upsert.
  • ignore_empty (Boolean, optional) : If you have a few empty fields while updating the record and you want the system to ignore it, input the value as TRUE.
  • resource (JSON array, optional): Initiate an object of Resource class that contains the details of the module, file, and field mappings associated with the bulk write job. The resource array contains the file_id, module, find_by, field_mappings and type keys. For more details, refer to our help document here.

Mapping fields in bulk write jobs

To specify which value in the CSV file should be mapped to which field in the CRM module, you can use the field_mappings key in the resource array. This is an optional key, and if you omit it, the system will use the headers of the CSV file to map the fields. When the field_mappings is skipped, the CSV headers must be the field API names and, all the columns of your CSV file will be used for import. In this case, you should remove columns that you do not want to import into your CRM account.

Please note that the CSV headers will be ignored if field_mappings is used while creating the bulk write job.

When to use the field_mappings key?

You should use the field_mappings key if:
  • The column headers in your CSV file are different from the field API names in the CRM module.
  • You want to ignore certain columns in the CSV file.
  • You want to specify a default value for a field in the CRM module.

Rules for using the field_mappings key

  • When you map a field by specifying the field_mappings key, it is mandatory to specify either the index or the default_value key, or the system will throw an error.
  • The index is the position of the field in the CSV file, starting from 0.
  • The default_value key is used when you want to provide a default value for a field. By specifying a default value, you ensure that if a cell in the CSV file is empty, the system will populate it with the default value.
  • Use the ignore_empty key when you want the system to skip updating fields that are empty in the source file. Please note that for insert operation, ignore_empty will be set as false by default.
Index
Default Value
Ignore Empty
Empty Cell
Action
YES
YES/NO
TRUE/FALSE
NO
CSV Value
YES
YES
TRUE/FALSE
YES
Default Value
YES
NO
TRUE
YES
No Update
YES
NO
FALSE
YES
Empty
NO
YES
TRUE/FALSE
YES/NO
Default Value

  1. If you specify the index and there are no empty cells in the source file, the system will use the values from the CSV file for the corresponding fields. The default_value will be ignored in this case.
  2. For update and insert operations, when the index is specified and there are a few cells for which the value is empty in the source file, then for the corresponding records with empty value:
    • If default_value is specified, the field will be updated with the default_value.
    • If default_value is not specified, the system will handle the corresponding fields differently depending on the value of the ignore_empty key.
      • The field will be left untouched if ignore_empty is set to TRUE.
      • The field will be updated to an empty value if ignore_empty is FALSE.
  3. When the index is not specified, but a default_value is given, the system will update the corresponding fields in the CRM module with the specified default value

Bulk Insert with lookup and user lookup data

Here is an example of a simple bulk write job request:

<?php
namespace bulkwrite;

use com\zoho\crm\api\bulkwrite\BulkWriteOperations;
use com\zoho\crm\api\bulkwrite\RequestWrapper;
use com\zoho\crm\api\bulkwrite\CallBack;
use com\zoho\crm\api\util\Choice;
use com\zoho\crm\api\bulkwrite\Resource;
use com\zoho\crm\api\bulkwrite\FieldMapping;
use com\zoho\crm\api\modules\MinifiedModule;
use com\zoho\crm\api\bulkwrite\DefaultValue;

require_once "vendor/autoload.php";

class CreateBulkWriteJob
{
    public static function initialize()
    {
        // Add init code here.
        // Refer to this post for more details
    }

   public static function createBulkWriteJob(string $moduleAPIName, string $fileId)
    {
        $bulkWriteOperations = new BulkWriteOperations();
        $requestWrapper = new RequestWrapper();

       // Set the callback URL, and method
        $callback = new CallBack();
        $callback->setUrl("add_your_callback_url_here");
        $callback->setMethod(new Choice("post"));
        $requestWrapper->setCallback($callback);

        $requestWrapper->setCharacterEncoding("UTF-8"); //character encoding
   $requestWrapper->setOperation(new Choice("insert")); //specify the type of operation
       // module, file and field mapping details
        
        $resourceIns = new Resource();
        // To set the type of module that you want to import. The value is data.
        $resourceIns->setType(new Choice("data"));
        $module = new MinifiedModule();
        $module->setAPIName($moduleAPIName);
        $resourceIns->setModule($module);
        $resourceIns->setFileId($fileId);
        $resourceIns->setIgnoreEmpty(true);

        $fieldMapping1 = new FieldMapping();
        $fieldMapping1->setAPIName("Account_Name");
        $fieldMapping1->setIndex(0);
      
        $fieldMapping2 = new FieldMapping();
        $fieldMapping2->setAPIName("Phone");
        $fieldMapping2->setIndex(1);
    
        $fieldMapping3 = new FieldMapping();
        $fieldMapping3->setAPIName("Parent_Account");
        $fieldMapping3->setIndex(2);
        $fieldMapping3->setFindBy("id");
      
        $fieldMapping4 = new FieldMapping();
        $fieldMapping4->setAPIName("Referred_User");
        $fieldMapping4->setIndex(3);
        $fieldMapping4->setFindBy("id");
        $defaultValue = new DefaultValue();
        $defaultValue->setValue("4876876327565");
        $fieldMapping4->setDefaultValue($defaultValue);

        $resourceIns->setFieldMappings([$fieldMapping1, $fieldMapping2, $fieldMapping3, $fieldMapping4]);
        $requestWrapper->setResource([$resourceIns]);

    //Call createBulkWriteJob method that takes RequestWrapper instance as parameter
        $response = $bulkWriteOperations->createBulkWriteJob($requestWrapper);
        // Add your code to handle the response received in the $response.
    }
}

CreateBulkWriteJob::initialize();
$moduleAPIName = "Accounts"; //The module to which you want to insert the records
$fileId = "48768764054001"; // add your file_id received in the response of Upload File
CreateBulkWriteJob::createBulkWriteJob($moduleAPIName, $fileId);


In the above code, Referred_User is a user lookup field, and Parent_Account is a lookup field in the Accounts module. The CSV file contains the unique ID of the corresponding records in the lookup modules.



Here is a sample CSV file used for the above Bulk Insert job. Note that the unique record IDs of the lookup (Parent_Account) and user lookup(Referred_User ) fields. If you choose to schedule the job without using field mappings, ensure that you provide headers for these fields in the format field_api_name.id with the record IDs in the corresponding columns.

Updating or Upserting records by unique key:

During an update operation, the find_by field acts as a unique key that identifies the record to be updated. You can use the unique record id, or any other unique field for this. For example, if the Phone field is a unique field in the Accounts module, specifying setFindBy("Phone") allows updates based on the values present in the Phone field. Similarly, for an upsert operation, if there is no existing record matching the provided Phone number, a new record will be inserted. Conversely, if a matching record is found, it will be updated. 

Here is the code snippet for the field_mapping for an update operation using the Phone field as the unique key.


//You can use setFindBy method to update/upsert records based on a unique field
         $resourceIns->setFindBy("Phone");

//fieldMapping for the unique field
        $fieldMapping1 = new FieldMapping();
        $fieldMapping1->setAPIName("Phone");
        $fieldMapping1->setIndex(1);
        
        $fieldMapping2 = new FieldMapping();
        $fieldMapping2->setAPIName("Description");
        $fieldMapping2->setIndex(8);

Importing Subform Data

The Bulk Write API allows you to import and update records with subforms. To insert or update records with subforms, you must import the parent records first with Bulk Write API. For this bulk write job, $moduleAPIName has to be set with parent module with proper $fieldMapping. Refer to the first sample for more details.  

Once you have imported the parent records, get the parent record ids, prepare the CSV file with the subform data, and create a Bulk Write job for the subform module. You can get the parent record ids from the result zip file in Step 5 (Download the result). Refer to our previous post to know how to prepare the CSV file with the Subform data.

Consider a subform named Contact_details in the Accounts module, with the fields Contact_Name, Contact_Email, and Contact_Phone. Here is the code snippet for the field_mapping to import the subform data. 

//field_mapping for the Parent_Id
        $fieldMapping1 = new FieldMapping();
        $fieldMapping1->setAPIName("Parent_Id");
        $fieldMapping1->setIndex(0);
        $fieldMapping1->setFindBy("id");

//add proper field_mapping for the subform fields with the subform field API names

Note that in this case, the $moduleAPIName is the api name of the subform (Contact_details) and the operation type should be insert. The subform data is added to the corresponding record specified by Parent_Id field in the subform CSV. 

Multiselect Lookup Fields in Bulk Write API

To upload records with multiselect lookup fields, you can follow the same steps as for subforms. First, import the parent records using a Bulk Write job. Then, import the multiselect lookup field data using another Bulk Write job. For the multiselect lookup field data, the CSV file should contain both the parent record ID and the multiselect lookup field data. Please note that this method does not work for line items.
Consider a multiselect lookup field named Clients in the Accounts module, which is linked to the Contacts module.  In your CSV file, ensure that you include both the Parent Record ID and the linked record IDs. For multiselect lookup data, the $moduleAPIName should be the API name of the linked module, and in the field_mappings, the API names used should correspond to the field API names from the lookup module. Check the following sample code for more details.

//field_mapping for module 1 (Contacts)
         $fieldMapping1 = new FieldMapping();
    $fieldMapping1->setAPIName("Client_Account"); //field API name in the linked module
        $fieldMapping1->setIndex(0);
        $fieldMapping1->setFindBy("id");
  
// field_mapping for module 2 (Accounts)    
        $fieldMapping2 = new FieldMapping();
        $fieldMapping2->setAPIName("Account"); // field API name in the linked module
        $fieldMapping2->setIndex(1);
        $fieldMapping2->setFindBy("id");

4. Check job status

The Bulk Write API offers two options for monitoring the status of a job: polling and callback. If you prefer not to poll for job status, you can wait for the system to notify you of job completion by using the provided callback URL in the Create Bulk Write Job request. Alternatively, you can choose to poll the API using the job ID obtained from the previous step to check the status of the scheduled bulk write job. The job status can be one of the following: ADDED, INPROGRESS, or COMPLETED. Only completed jobs will include the download-URL key in the response.

Here is a code snippet to check the job status and get the download URL from the response.

public static function getBulkWriteJobDetails(string $jobId)
  {
    $bulkWriteOperations = new BulkWriteOperations();
    //Call getBulkWriteJobDetails method that takes jobId as parameter
    $response = $bulkWriteOperations->getBulkWriteJobDetails($jobId);
    if ($response != null) {
        $responseWrapper = $response->getObject();
        if ($responseWrapper instanceof BulkWriteResponse) {
            $result = $responseWrapper->getResult();
            if ($result != null) 
                echo ("Bulkwrite DownloadUrl: " . $result->getDownloadUrl() . "\n");
            }
        }
    }


5. Download the result

In this step, you can obtain the result of the bulk write job in a ZIP file that includes a CSV file containing the details of the job. Please note that you can get the details of only one bulk write job in a single API call. 

To download the result, you need to use the download_url provided in the response. The CSV file will consist of the first three mapped columns from the uploaded file, as well as three additional columns: STATUS, RECORD_ID, and ERRORS.

{
    public static function initialize()
    {
        // Add init code here.
        // Refer to this post for more details
    }

    public static function downloadBulkWriteResult(string $downloadUrl, string $destinationFolder) 
    {
        $bulkWriteOperations = new BulkWriteOperations();
        $response = $bulkWriteOperations->downloadBulkWriteResult($downloadUrl);
        // Add your code to handle the response received in the $response.
    }
}

DownloadBulkWriteResult::initialize();
$downloadUrl = "paste_your_download_url_here";
$destinationFolder = "/Documents";
DownloadBulkWriteResult::downloadBulkWriteResult( $downloadUrl,$destinationFolder);



The image shows the result of an Upsert operation. As you can see, few of the records were skipped because the unique field had a duplicate value. 

We hope you found this post useful. If you have any further questions, reach out to us at support@zohocrm.com or let us know in the comments section. We are looking forward to hearing from you!

Stay tuned for more useful posts in the series. See you all next week!

    Access your files securely from anywhere


            Zoho Developer Community





                                      • Desk Community Learning Series


                                      • Digest


                                      • Functions


                                      • Meetups


                                      • Kbase


                                      • Resources


                                      • Glossary


                                      • Desk Marketplace


                                      • MVP Corner


                                      • Word of the Day


                                      • Ask the Experts



                                          Zoho Marketing Automation


                                                  Manage your brands on social media



                                                        Zoho TeamInbox Resources

                                                          Zoho DataPrep Resources



                                                            Zoho CRM Plus Resources

                                                              Zoho Books Resources


                                                                Zoho Subscriptions Resources

                                                                  Zoho Projects Resources


                                                                    Zoho Sprints Resources


                                                                      Qntrl Resources


                                                                        Zoho Creator Resources



                                                                            Zoho CRM Resources

                                                                            • CRM Community Learning Series

                                                                              CRM Community Learning Series


                                                                            • Kaizen

                                                                              Kaizen

                                                                            • Functions

                                                                              Functions

                                                                            • Meetups

                                                                              Meetups

                                                                            • Kbase

                                                                              Kbase

                                                                            • Resources

                                                                              Resources

                                                                            • Digest

                                                                              Digest

                                                                            • CRM Marketplace

                                                                              CRM Marketplace

                                                                            • MVP Corner

                                                                              MVP Corner





                                                                                Design. Discuss. Deliver.

                                                                                Create visually engaging stories with Zoho Show.

                                                                                Get Started Now


                                                                                  Zoho Show Resources


                                                                                    Zoho Writer Writer

                                                                                    Get Started. Write Away!

                                                                                    Writer is a powerful online word processor, designed for collaborative work.

                                                                                      Zoho CRM コンテンツ








                                                                                        Nederlandse Hulpbronnen


                                                                                            ご検討中の方





                                                                                                  • Recent Topics

                                                                                                  • Custom Status for Purchase Orders

                                                                                                    Currently Zoho books has functionality to create custom statuses for Sales Orders. Can this be extended to include custom status for purchase orders as well? It was a great decision to add this functionality to sales orders. Our use case is for tracking
                                                                                                  • My workflows not working

                                                                                                    Hi I have created my first workflow I want it so when a customer responds to a ticket that the status is changed to "reopened" We want this so its clear there is a ticket to deal with again. We only have 2 agents responding to tickets and do not have
                                                                                                  • Waiting Room

                                                                                                    Is there any plans to implement a Virtual Waiting Room for Attendees on Meetings and or Webinars?
                                                                                                  • Error AS101 when adding new email alias

                                                                                                    Hi, I am trying to add apple@(mydomain).com The error AS101 is shown while I try to add the alias.
                                                                                                  • Global Sets for Multi-Select pick lists

                                                                                                    When is this feature coming to Zoho CRM? It would be very useful now we have got used to having it for the normal pick lists.
                                                                                                  • What's New - January 2025 | Zoho Backstage

                                                                                                    Hi there! As we step into a brand-new year, we’re thrilled to bring you some exciting updates to make event management more enjoyable and efficient. Here’s what we rolled out in January to kick off 2025: Backstage 2.0 now enforced for all users We've
                                                                                                  • How do you do ticket add ons in Backstage?

                                                                                                    Hi Everyone, If you wanted to have general admin tickets and allow for add ons, like camping, or car or Carbon offset. What would you do? Peace Robin
                                                                                                  • Re-create auth token

                                                                                                    I need to re-create a self-client auth token, with the same scopes (and additional ones) as the current token. Is there a way to view the scopes assigned to the current token to ensure that no scopes are missing when creating the new one?
                                                                                                  • Writing on sketch cards is bugged when zoomed in

                                                                                                    When zoomed in, it writes a noticeable distance above or to the side of where you're actually trying to write. The further you're zoomed in, the more noticeable it is. Zooming is also entirely absent on the desktop version.
                                                                                                  • Doesn't let me test my newsletter.

                                                                                                    Hello, I click "Preview and Test". I select the email address to send the test (usually my own email). I send the test email, appears the message "Your test email has been sent", but nothing happens. I mean, there is nothing in my inbox. I also tried
                                                                                                  • Custom Related List anchor tag with tel protocol

                                                                                                    The following code worked great for about a year to create clickable tel and mailto links in a related list in the Deals module. Just this morning, it started displaying the HTML as text. For example: <a href='tel:8882490100'>8882490100</a> Zoho support
                                                                                                  • reset of user password

                                                                                                    User forgot her password, so she requests a password rest, but never gets the email. How do I reset her password, since that function is not an option, even for an administrator?
                                                                                                  • In what case it's better to use Developer Console? Only for marketplace?

                                                                                                    As I can see from the documentation, the limitations are challenging. No Custom API No Page Variables No Custom Connections.
                                                                                                  • commision tracker

                                                                                                    I am after a zoho developer to finish my commission tracker as I hired someone on oDesk that started but now will respond to my emails.I really need to get It finished as it is holding up my plans Thanks Budget : $500 | Duration : a.s.a.p To leave a private message, please click the link for private response in post Action section.
                                                                                                  • Generate a Zoho Sign link

                                                                                                    From time to time I get a response "I never received your you e-document for electronic signature" is there a way to generate a Zoho Sign link to share.
                                                                                                  • Error 403: Forbidden When Updating Email Signature via API

                                                                                                    Hi Zoho Desk team, First, congratulations again on the excellent Zoho API. But, I’m encountering an issue while attempting to update an email signature via the API. Whenever I make a request to update the signature, the response returns an HTTP 403 Forbidden
                                                                                                  • Queues Management ?

                                                                                                    I need to assign the new requirements equally between my support agents. How I can configure this feature in Zoho? Thanks
                                                                                                  • Client Can't View Ticket

                                                                                                    Hi, I created a ticket for my Client since he had an issue. I added him as a Contact as well. My client got a notification in his email that the ticket has been created. When he went to view the ticket on his desktop, the screen was blank. He then viewed
                                                                                                  • Automating Custom Web Link Messages to Customers via Instant Messaging in Zoho Desk

                                                                                                    Hello, I am looking for assistance with Zoho Desk's Instant Messaging feature. Is there a way to automate the sending of a custom web link to customers every day at 6:30 PM? Any guidance or suggestions on how to achieve this would be greatly appreciated.
                                                                                                  • How do you apply field validation to a date field?

                                                                                                    We are using Sign for remote e-signature of liability waivers. The signer must enter a date of birth, and if they are a minor their legal guardian must sign for them. However we have had numerous problems with signers returning an invalid date of birth
                                                                                                  • Implement Regex in Layout and Validation rules

                                                                                                    Hello all, We are excited to announce that users can now implement Regular Expressions (Regex) in our layout and validation rules. This new functionality allows for more flexible rules to be created when designing and validating forms. What is Regex?
                                                                                                  • Zoho Desk Validation Rule Using Custom Function

                                                                                                    Hi all, I tried to find the way to validate fields using custom function just like in Zoho CRM but to no avail. Is there a way to do this?
                                                                                                  • Editing landing page after signup for a webinar

                                                                                                    Hi, how can I edit the landing page after signing up for a webinar. I personalized the email "registration confirmation", which is working fine. Nevertheless after submitting the form, the participant gets redirected to a page, which looks like the standard
                                                                                                  • Kiosk GetRecords can't compare formulas

                                                                                                    (submitted as a help request by accident - should have been a topic) Scenario: I have an "email domain" formula whose value is "@xyz.com" if the e-mail address is brent@xyz.com . I have the same field in Leads and Contacts. So, theoretically, I should
                                                                                                  • Kiosk can't merge picklist or multiselect

                                                                                                    There is no ability to load a multiselect or picklikst field into a kiosk with the values that have been previously selected. So, I essentially have 3 unacceptable options: 1.)Load the value into a text string and include instructions like this: "Picklist
                                                                                                  • Fillable template with dynamic tables?

                                                                                                    Is there a way to build a fillable template so that users can add rows to a table? To describe what I'm trying to accomplish the table has 3 sections; a header row, some number of rows with custom information, and a summary row with totals. I can't figure
                                                                                                  • Tip of the Week - Timeline User Filter for Dashboard

                                                                                                    Does the Date User Filter in your Dashboard leave out a few reports from filtering? This happens when the reports are not created over the same table and the (time) column over which the user filter is created.   To solve this, Zoho Reports has a Timeline User Filter. This filter applies the criteria on all reports, matching it with the best possible date column associated with a report. i.e., if your reports are based on a date column, then it will filter it for that data column.  To know more on
                                                                                                  • Zoho Forms Webhooks - Only sends on submission not on edit.

                                                                                                    Hi There, We have a slightly unique use case for Zoho Forms. We used Zoho Forms for the collection of Guest Lists. When a customer submits their guest list they get an email confirmation. Within this is a link for them to update their guest list before
                                                                                                  • Add Knowledge Base KB Articles to multiple categories

                                                                                                    Greetings,  Love you help center system.  One item that would be incredibly helpful to many of us would be able to add a single Knowledge Base KB article to multiple categories in our system.  It seems it could be quite easy to use a checkbox form, instead
                                                                                                  • What does "Tickets for Review" do?

                                                                                                    What is the purpose of the va nilla view " Tickets for Review?"
                                                                                                  • Report of Leads/Accounts/Deals with no open activities

                                                                                                    How can we replicate the option in filter to only show Leads/Accounts/Deals with no open activities ? The option is present in the filters (please find screenshot) but I am unable to find them in the report. Thanks
                                                                                                  • Cannot set Subform Multiline field to read-only using Client Script

                                                                                                    I am using client script to set different subform fields as read-only It works for Single Line Fields, but when I try to set a Multiline field as Read-only it doesn't work var subform = ZDK.Page.getSubform("Subform_1"); var item_Code_old = subform.getField('Item_Code_Old').setReadOnly(true);
                                                                                                  • Changing Color Theme of Guided Conversations

                                                                                                    Hello, We have recently added Guided Conversations to one of our websites, but I am wondering if there is a way to customize the color scheme so it matches the appearance of the website? Thank you in advance!
                                                                                                  • Lost the ability to sort by ticket owner

                                                                                                    Hi all, in the last week or so, we have lost the ability to sort tickets by Ticket Owner. Unlike the other columns which we can hover over and click on to sort, Ticket Owner is no longer clickable. Is it just us, or are other customers seeing this too?
                                                                                                  • Can't login to Zoho mail

                                                                                                    I'm logged into Zoho but when I try to go in zoho mail I get: Invalid request! The input passed is invalid or the URL is invoked without valid parameters. Please check your input and try again. I just set up my mx records and stuff with namecheap a few
                                                                                                  • Social Media Simplified with Zoho Social: Bridge sales and social media marketing

                                                                                                    Social media marketing has become the go-to strategy for most businesses who are trying to boost awareness of their product/service, create a brand image in the minds of their audience, increase website traffic and engagement, and generate leads. Posting
                                                                                                  • This domain is not allowed to add. Please contact support-as@zohocorp.com for further details

                                                                                                    I am trying to setup the free version of Zoho Mail. When I tried to add my domain, theselfreunion.com I got the error message that is the subject of this Topic. I've read your other community forum topics, and this is NOT a free domain. So what is the
                                                                                                  • Work Orders / Bundle Requests

                                                                                                    Zoho Inventory needs a work order / bundle request system. This record would be analogous to a purchase order in the purchasing workflow or a sales order in the sales cycle. It would be non-journaling, but it would reserve the appropriate inventory of
                                                                                                  • Create Quote does not show the "Product Description" entered as part of the Product setup.

                                                                                                    The product description created as part of the product setup page, does not show in the Create Quote module; The module allows for an additional description to be added but without access to the original stored description. By selecting the product from the "Product Name Lookup" pop-up, the "Product Description" part should be populated from the product record where the user is allowed to further modify it.
                                                                                                  • Allow Itemization for Recurring Expenses

                                                                                                    For whatever reason, one cannot itemize a Recurring Expense. This capability should be added. The use cases to support this is largely the same as what they were to allow for itemization in Expenses. Anything that would need to be itemized for a regular
                                                                                                  • Next Page