What are article crawlability and indexability?

What are article crawlability and indexability?

All you need to know about no-follow and no-index in KB.


What are article crawlability and indexability?  

Crawlability and indexability are vital for making an article visible and accessible to search engines. When a search engine crawls an article, it reviews the site's content to assess its topic and relevance to specific search queries. If a site lacks crawlability, meaning search engines struggle to scan it effectively, the content will not be indexed and will consequently not appear in search results.
 
Additionally, a site’s content must be appropriately optimized for search engine crawlers, allowing it to be indexed accurately and appear in relevant search outcomes. Ensuring every article is crawlable and indexable can significantly enhance its visibility and improve its search engine rankings.

How do you block search indexing?

There are two methods to apply the no-index directive: using a <meta> tag or via an HTTP response header. Both methods achieve the same result, so select the one that suits your website and content type best. Additionally, you can pair the no-index directive with other indexing rules. For instance, you can combine a no-follow directive with an index like this: <meta name="robots" content=" index, nofollow" />.
Quote<meta> tag 

To prevent all search engines that support the no-index rule from indexing a page on your site, place the following <meta> tag into the<head> section of your page:

Quote<meta name="robots" content="noindex"> 
Notes
Certain search engines may interpret the no-index directive in various ways. Therefore, your page could still appear in other search engines' results.

Usage of the no-index tag 

The no-index rule can be implemented using a <meta> tag or an HTTP response header to restrict search engines that recognize the no-index directive, like Google, from indexing certain content. When Googlebot comes across a page with this tag or header, it will remove that page entirely from Google Search results, regardless of whether there are links from other websites.

Notes
To ensure the index rule works, the page or resource must be accessible and not blocked by a robots.txt file. If blocked, the crawler won't see the index rule, though the page might still show in search results if linked from other pages. An index is beneficial if you lack root server access, as it lets you manage page access individually.

What are no-follow links? 

Typically, it's best to allow robots to follow all links on a webpage. Being overly strict in indicating which links should be followed or marked as no-follow may give the impression that the site is trying to influence how a robot views it.

 

Page sculpting is the practice of using no-follow commands to influence the transfer of signals between pages. In the best-case scenario, these strategies to manipulate bots are ineffective. In the worst-case scenario, trying to control bots with nofollow may result in penalties.

When should you use meta no-follow?

 Consequently, there are limited scenarios in which using meta robots no-follow on websites is appropriate. Typically, encountering meta robots no-follow during SEO audits may indicate that a website has been over-optimized.

What are dofollow links?

 Dofollow links, often called "follow" links, are typical online hyperlinks. Unlike links with specific attributes such as “nofollow” or “sponsored,” these links do not carry unique identifiers. As a result, the following links contribute to the ranking control of the pages they connect to.

Experimenting with crawlability and indexability

Before a website's pages can be displayed in search results, search engine bots must first discover all the pages on the site and then analyze them to determine their rankings. The initial phase of finding pages is called crawling, while the subsequent analysis is called indexing.

 

Crawling starts with bots identifying all the URLs of a website's pages. They primarily find these URLs through internal links within the site or through backlinks from external sites. Once a bot identifies a page’s URL, it retrieves content, including the title, text, images, and additional data, such as the date it was last updated. Restrictions can dictate which files and pages a bot can crawl.

 

Indexing occurs after the crawling process. During this stage, bots assess all the information collected from the crawl. They evaluate whether the content is valuable and authoritative and identify the topics associated with the page and how it compares to other relevant pages. Additionally, search engine bots will determine which search results a page might appear in, if any, and its position within those results.

No-follow vs no-index

Let me give you a typical example of an ad campaign landing page designed solely for traffic from typical ads. It's not ideal to prevent organic search traffic from reaching these pages. Implementing meta robots and a no-index tag will keep search engines from ranking them. 

Summing up

  • No-index tells a robot not to index the page. It should be used to keep pages out of search results or to help with low-quality content issues.
  • Meta nofollow tells a robot not to follow a specific link or all of the links on a page. This should typically not be used.
  • Robots should be allowed to determine what gets indexed and shown in search results. If a page is not intended to be visible in search results, it often shouldn't be part of the website.

 

Please watch this space for more detailed use cases of SEO tags that you can apply to your organisation's knowledge base.

 

Cheers,

 

Kavya Rao,

The Zoho Desk Team


    • Sticky Posts

    • Register for Zoho Desk Beta Community

      With the start of the year, we have decided to take a small step in making the life of our customers a little easier. We now have easy access to all our upcoming features and a faster way to request for beta access. We open betas for some of our features

    • Share your Zoho Desk story with us!

      Tell us how you use Zoho Desk for your business and inspire others with your story. Be it a simple workflow rule that helps you navigate complex processes or a macro that saves your team a lot of time; share it here and help the community learn and grow with shared knowledge. 

    • Tip #1: Learn to pick the right channels

      Mail, live chat, telephony, social media, web forms—there are so many support channels out there. Trying to pick the right channels to offer your customers can get pretty confusing. Emails are most useful when the customer wants to put things on record. However, escalated or complicated issues should not be resolved over email because it's slow and impersonal.  When you need immediate responses, live chat is more suitable. It's also quick and convenient, so it's the go-to channel for small issues. 

    • Welcome to Zoho Desk Community - Say hello here!

      Hello everyone! Though we have been here for a while, it’s time to formally establish the Zoho Desk Community; we’re really happy to have you all here! This can be the place where you take a moment to introduce yourself to the rest of the community. We’d love to hear all about you, what you do, what company or industry you work for, how you use Zoho Desk and anything else that you will like to share! Here’s a little about me. I am Chinmayee. I have been associated with Zoho since 2014. I joined here

    • Webinar 1: Blueprint for Customer Service

      With the launch of a host of new features in Zoho Desk, we thought it’ll be great to have a few webinars to help our customers make the most of them. We’re starting off with our most talked about feature, Blueprint in Zoho Desk. You can register for the Blueprint webinar here: The webinar will be delivered by our in-house product experts. This is a good opportunity to ask questions to our experts and understand how Blueprint can help you automate your service processes. We look forward to seeing

      • Recent Topics

      • Add Mileage Tab on Expenses

        Hello, the 'Add Mileage' tab is not visible. How can I enable it? I checked the settings, and my user has admin privileges. I also enabled mileage in the policy, but the 'Add Mileage' tab is still not visible. What could be the issue?

      • Currency transition

        We are using Zoho CRM in Curacao, Dutch Caribbean. Our currency is currently the ANG. Curacao will be transition ing from using the ANG (Antillean Guilder) to using the XCG currency (Caribbean Guilder) on March 31st 2025, see: https://www.mcb-bank.com/caribbean-guilder.

      • Error sending an email

        Hi, I'm trying to send an automated email (receiver is always the same) via Deluge function. It was working perfectly, but yesterday the receiver's mailbox got full and started returning this error: { "data": [ { "code": "NOT_ALLOWED", "details": { "reason":

      • [ Free Webinar] Transform your hiring process with Zoho Recruit's AI tools

        We are conducting a free webinar (on 26th November, 05:00 AM IST - 06:00 AM IST) designed to help you make the most of Zoho Recruit’s AI capabilities and give you a sneak peek into our future roadmap. Why Attend? In this session, you will: Discover how

      • Email signature duplicate

        Hi, For a few weeks, opening the email writer would show an error. After clicking ok, the signature would change slighty (font size, I believe). After that it worked fine, so we thought nothing of it. However, now it no longer shows the error puts the

      • It's time to say goodbye to Zoho Recruit for me.

        Hello, I have been a Zoho Recruit user since 2013. The tool has evolved, with a better UI, new features and so on. The pricing is still a great asset too. BUT, that being said, important features are not coming fast enough. I tried to share my point of

      • Remove "Additional Information" options from user-created tickets

        I'd like to configure customers to be able to create tickets themselves, with and without logging in. However, I don't want them to set things like the priority or due date. Is there a way to remove these options from the help center when customers create

      • Getting Started All Zoho Desk APIs require these two mandatory fields in the header.

        Hey, according to API Dokumentation, All Zoho Desk APIs require these two mandatory fields in the header. Authorization - Authentication request token orgId -ID of the organization to access. All API endpoints except /organizations mandatorily require

      • ZOHO Work Drive Back Up

        I am looking for a ZOHO Work Drive backup solution. Something that is cloud based. There's lots of these kinds of options for Google Drive and other providers, but I have not seen anything for WorkDrive. Any suggestions?

      • Export PDF File Name

        Is it possible to change the default Zoho .pdf naming scheme for inventory items like quotations? Would like to use the the Subject as the default quote name. Is this possible?

      • Facing issue in getting data through api calls

        I have send data of users and want to get the data oof users but facing issue with one field for that I want help

      • Kaizen #126 - Circuits in Zoho CRM - Part 1

        Hello everyone! Welcome back to another week of Kaizen! Today, we will discuss an exciting topic—Circuits in Zoho CRM. For starters, we will discuss what Circuits are, how beneficial they are for businesses, different views of a Circuit, and the different

      • Google Ads Data is Publicly available in Zoho CRM

        We recently discovered that ALL of the following Google Ads fields are visible to all users in our CRM that have access to either Leads or Contacts modules. Not only is this troubling and inconvenient, it should be unacceptable. It also creates a mess

      • Campaigns workflow emails going to Spam

        Hi Everyone, Does anyone have a solution for workflow emails going to recipients SPAM folders? Our domain is not on any Blacklists, We have our domain authenticated with SPF, DKIM and DMARC all set up and we still have leads who tell us they are not getting

      • Best Way to Integrate a Shared Sales Inbox with Zoho CRM

        I’m setting up a shared email inbox (e.g., sales@mycompany.com) for our sales team and want to integrate it with Zoho CRM Leads and Deals so that multiple team members can collaborate efficiently. Since Zoho CRM doesn’t support IMAP for shared inboxes,

      • Add and Delete Customer Portal user via Deluge

        Using Creator to build an app involving a customer portal. I'm trying to integrate the customer portal for the first time and it is raising questions, mainly because it's new territory. Is it possible to add a new user to the customer portal via deluge? For example, if an existing user invites/refers another potential user to try our app, is there code to automate via deluge, setting up an account ready to go for them? The referral would include their email address, and a workflow would set up the

      • Unexpected Error in Reports (Pivot Chart, Pivot Table)

        Dear Zoho Support Team, I am facing an issue while creating reports (Pivot Chart, Pivot Table, etc.) in Zoho Creator. When I try to access the report, I receive an unexpected error, and an alert message saying "Undefined" appears with the loading UI.

      • Access invoice custom modules data from API

        Hi, I am using $url = "https://www.zohoapis.eu/invoice/v3/invoices"; $opts = ["http" => [method" => "GET",header" => [ "Authorization: Zoho-oauthtoken $accessToken", "X-com-zoho-invoice-organizationid: $orgId" ] ] ]; From my app to retrieve the invoices,

      • Using IMAP configuration for shared email inboxes

        Our customer service team utilizes shared email boxes to allow multiple people to view and handle incoming customer requests. For example, the customer sends an email to info@xxxx.com and multiple people can view it and handle the request. How can I configure

      • Zoho CRM and Google Sync - delete Contacts?

        If I delete a contact in Zoho CRM, will it also be deleted in Google contacts?

      • Customize the ticket ID for enhanced ticket tracking

        Hello everyone, Ticket IDs in Zoho Desk are the unique reference numbers that are assigned to tickets when they're created. Agents generally use the ticket ID to locate a particular ticket and to reference a ticket in conversations and feeds. When customers

      • For Each (Desk Ticket) Loop

        What I need is some sort of "for each" loop functionality in Flow, this doesn't seem to be a thing, but I'm wondering if someone found some sort of workaround? Here's the whole situation: I have a Flow, which is supposed to be triggered by a custom field

      • Turn off/Disable Lead Approval

        Is there a way to disable the need to approve leads? I thought there was but I can't find any reference to it in help, etc. We don't need an approval process for incoming leads at the moment. Thanks.

      • Change colour on task lists?

        I have many task lists on my project boards, since they are all the same colour they blend in with each other. Is there a way to change the colour of these task lists in the list view?

      • Zoho Flow Doesn't Detect Desk Ticket Custom Field Change

        I have a Flow that is configured to be triggered when a custom field on a ticket changes. I also have a Schedule in Desk that runs a script that changes the custom field. When I change the custom field manually in the Desk interface, the Flow runs as

      • invokeURL to change custom field

        I have a deluge script that currently runs as a Schedule. It works exactly as intended, however I've recently been informed by Zoho that there's some mechanism in place to block changes made by a script when it runs on a schedule from being detected by

      • Email transfer to projects

        We are unable to add comment/map documents to this task. WHY DID THIS HAPPEN? The possible reasons are : 1. Email ID not confirmed. To confirm use the link http://zohosso.wiki.zoho.com/#resendconfirmation 2. You do not have permission to add comment/

      • Stop adding Default ID column to xls exports

        When anything is exported to xls, Zoho adds a column with an ID.  WE DO NOT WANT THIS COLUMN.  We use an automated report to a team.  We have our own tracking number.  1. This makes the report messy, it just pushes OUR data off to the right.  2. We have

      • Este domínio já está associado a esta conta

        Fui fazer meu cadastro na zoho e quando digitei meu domínio recebi essa mensagem que meu domínio estava associado a uma conta que eu nem faço idéia de quem seja. Como que faço pra resolver isso? Atenciosamente, Anderson Souza.

      • Transferring Attachments from Lead to Account

        Hi All, I'm trying to create a function that will transfer attachments in leads to the newly created account. I know there's an option to choose where the attachments go when you click the standard convert button but we have a high volume of conversions

      • New Line in Double Optin Email

        Hello, is it possible to add new lines to the double optin email? This is necessary to add legal informations to the mail, which would look very messy if it is all in one line. Thank you in advance Carsten

      • Cursor Keeps Disappearing in the Email Fields

        I did the recent update to the Zoho desktop app - I'm on PC Windows 11. My cursor keeps disappearing when I try to type in the "To" "CC" "Subject" boxes and typing in the body of the email. I've been trying to just tab through the sections instead of

      • Hourly Permission not getting Calculated

        That is our settings The total calculation should be from 9:37 AM to 3:37 PM, but the hourly permission isn't getting calculated The last entry is hourly permission, it's not

      • Client Script field.setCriteria need the list of possible operators for criteria or the opposite of "equal"

        The gist is to just filter the lookup the countries not listed in the multiselect field. I've been using a client script and this time I would like to set this criteria for the lookup: op.setCriteria("((Specialty_Requirement:equals:" + spec + ")and(Countries_Excluded:not_equal:"

      • Highlight a candidate who is "off limits"

        Hello: Is there a way to highlight a candidate who is "off limits"?  I would like to have the ability to make certain candidate and / or Client records highlighted in RED or something like that.   This would be used for example when we may have placed a candidate somewhere and we want everyone in our company to quickly and easily see that they are off limits.  The same would apply when we want to put a client or former client off limits so no one recruits out of there. How can this be done? Cheers,

      • old invoices (Exchange rate)

        We have been facing an issue and searching for a solution for over a month. Issue: Previously, the exchange rate in Zoho Books and Zoho CRM was incorrect, and invoices were being recorded in Zoho CRM with the wrong exchange rate. After adjusting the exchange

      • Zoho Mail Android app update - View emails shared via Permalink on the app.

        Hello everyone! In the latest version(v2.8.2) of the Zoho Mail Android app update, we have brought in support to access the emails shared via permalink within the app. Earlier, when you click the permalink of an email, you'll be redirected to a mobile

      • How to download Renamed File that already updated to Zoho Creator.

        Hi members, I construct a button with report workflow. link = "https://creatorexport.zoho.com" + zoho.appuri + "report_link_name/File_upload/image-download/" + input.File_upload; openUrl(link,"new window"); This script able to download the file. But I

      • Set a global default filter to All Candidates view

        Hi, Is it possible to set a global filter to the All Candidates view? As a use case would be for off-limit candidates. These are those who are off limits due to previous bad client feedback on performance/behaviour. Ideal use case would be to set these

      • Merge Join PDFs Zoho Creator

        Hi all, I have a field where users upload PDF, is it possible to join those pdfs into one with a function or something? Regards.

      • Next Page