What are article crawlability and indexability?

What are article crawlability and indexability?

All you need to know about no-follow and no-index in KB.


What are article crawlability and indexability?  

Crawlability and indexability are vital for making an article visible and accessible to search engines. When a search engine crawls an article, it reviews the site's content to assess its topic and relevance to specific search queries. If a site lacks crawlability, meaning search engines struggle to scan it effectively, the content will not be indexed and will consequently not appear in search results.
 
Additionally, a site’s content must be appropriately optimized for search engine crawlers, allowing it to be indexed accurately and appear in relevant search outcomes. Ensuring every article is crawlable and indexable can significantly enhance its visibility and improve its search engine rankings.

How do you block search indexing?

There are two methods to apply the no-index directive: using a <meta> tag or via an HTTP response header. Both methods achieve the same result, so select the one that suits your website and content type best. Additionally, you can pair the no-index directive with other indexing rules. For instance, you can combine a no-follow directive with an index like this: <meta name="robots" content=" index, nofollow" />.
Quote<meta> tag 

To prevent all search engines that support the no-index rule from indexing a page on your site, place the following <meta> tag into the<head> section of your page:

Quote<meta name="robots" content="noindex"> 
Notes
Certain search engines may interpret the no-index directive in various ways. Therefore, your page could still appear in other search engines' results.

Usage of the no-index tag 

The no-index rule can be implemented using a <meta> tag or an HTTP response header to restrict search engines that recognize the no-index directive, like Google, from indexing certain content. When Googlebot comes across a page with this tag or header, it will remove that page entirely from Google Search results, regardless of whether there are links from other websites.

Notes
To ensure the index rule works, the page or resource must be accessible and not blocked by a robots.txt file. If blocked, the crawler won't see the index rule, though the page might still show in search results if linked from other pages. An index is beneficial if you lack root server access, as it lets you manage page access individually.

What are no-follow links? 

Typically, it's best to allow robots to follow all links on a webpage. Being overly strict in indicating which links should be followed or marked as no-follow may give the impression that the site is trying to influence how a robot views it.

 

Page sculpting is the practice of using no-follow commands to influence the transfer of signals between pages. In the best-case scenario, these strategies to manipulate bots are ineffective. In the worst-case scenario, trying to control bots with nofollow may result in penalties.

When should you use meta no-follow?

 Consequently, there are limited scenarios in which using meta robots no-follow on websites is appropriate. Typically, encountering meta robots no-follow during SEO audits may indicate that a website has been over-optimized.

What are dofollow links?

 Dofollow links, often called "follow" links, are typical online hyperlinks. Unlike links with specific attributes such as “nofollow” or “sponsored,” these links do not carry unique identifiers. As a result, the following links contribute to the ranking control of the pages they connect to.

Experimenting with crawlability and indexability

Before a website's pages can be displayed in search results, search engine bots must first discover all the pages on the site and then analyze them to determine their rankings. The initial phase of finding pages is called crawling, while the subsequent analysis is called indexing.

 

Crawling starts with bots identifying all the URLs of a website's pages. They primarily find these URLs through internal links within the site or through backlinks from external sites. Once a bot identifies a page’s URL, it retrieves content, including the title, text, images, and additional data, such as the date it was last updated. Restrictions can dictate which files and pages a bot can crawl.

 

Indexing occurs after the crawling process. During this stage, bots assess all the information collected from the crawl. They evaluate whether the content is valuable and authoritative and identify the topics associated with the page and how it compares to other relevant pages. Additionally, search engine bots will determine which search results a page might appear in, if any, and its position within those results.

No-follow vs no-index

Let me give you a typical example of an ad campaign landing page designed solely for traffic from typical ads. It's not ideal to prevent organic search traffic from reaching these pages. Implementing meta robots and a no-index tag will keep search engines from ranking them. 

Summing up

  • No-index tells a robot not to index the page. It should be used to keep pages out of search results or to help with low-quality content issues.
  • Meta nofollow tells a robot not to follow a specific link or all of the links on a page. This should typically not be used.
  • Robots should be allowed to determine what gets indexed and shown in search results. If a page is not intended to be visible in search results, it often shouldn't be part of the website.

 

Please watch this space for more detailed use cases of SEO tags that you can apply to your organisation's knowledge base.

 

Cheers,

 

Kavya Rao,

The Zoho Desk Team


    • Sticky Posts

    • Register for Zoho Desk Beta Community

      With the start of the year, we have decided to take a small step in making the life of our customers a little easier. We now have easy access to all our upcoming features and a faster way to request for beta access. We open betas for some of our features
    • Share your Zoho Desk story with us!

      Tell us how you use Zoho Desk for your business and inspire others with your story. Be it a simple workflow rule that helps you navigate complex processes or a macro that saves your team a lot of time; share it here and help the community learn and grow with shared knowledge. 
    • Tip #1: Learn to pick the right channels

      Mail, live chat, telephony, social media, web forms—there are so many support channels out there. Trying to pick the right channels to offer your customers can get pretty confusing. Emails are most useful when the customer wants to put things on record. However, escalated or complicated issues should not be resolved over email because it's slow and impersonal.  When you need immediate responses, live chat is more suitable. It's also quick and convenient, so it's the go-to channel for small issues. 
    • Welcome to Zoho Desk Community - Say hello here!

      Hello everyone! Though we have been here for a while, it’s time to formally establish the Zoho Desk Community; we’re really happy to have you all here! This can be the place where you take a moment to introduce yourself to the rest of the community. We’d love to hear all about you, what you do, what company or industry you work for, how you use Zoho Desk and anything else that you will like to share! Here’s a little about me. I am Chinmayee. I have been associated with Zoho since 2014. I joined here
    • Webinar 1: Blueprint for Customer Service

      With the launch of a host of new features in Zoho Desk, we thought it’ll be great to have a few webinars to help our customers make the most of them. We’re starting off with our most talked about feature, Blueprint in Zoho Desk. You can register for the Blueprint webinar here: The webinar will be delivered by our in-house product experts. This is a good opportunity to ask questions to our experts and understand how Blueprint can help you automate your service processes. We look forward to seeing

    Nederlandse Hulpbronnen


      • Recent Topics

      • Question regarding import of previous deals...

        Good afternoon, I'm working on importing some older deal records from an external sheet into the CRM; however, when I manually click "Add New Deal" and enter the pertinent information, the deal isn't appearing when I look at the "Deals" bar on the account's
      • Adding Social Media Buttons to Basic Campaigns

        Hi, I'm quote new to using Zoho Campaigns and I can't work out how to add Social Media Buttons into my basic campaign? In MailChimp there's a button that brings the icons into your campaign for you. I've tried adding the social media icons as 'buttons' in Zoho but it's not looking great. Can anyone help? Thanks!
      • Dropshipping Address - Does Not Show on Invoice Correctly

        When a dropshipping address is used for a customer, the correct ship-to address does not seem to show on the Invoice. It shows correctly on the Sales Order, Shipment Order, and Package, just not the Invoice. This is a problem, because the company being
      • An Exclusive Session for Zoho Desk Users: AI in Zoho Desk

        A Zoho Community Learning Initiative Hello everyone! This is an announcement for Zoho Desk users and anyone exploring Zoho Desk. With every nook and corner buzzing, "AI's here, AI's there," it's the right time for us to take a closer look at how the AI
      • Best way to schedule bill payments to vendors

        I've integrated Forte so that I can convert POs to bills and make payments to my vendors all through Books. Is there a way to schedule the bill payments as some of my vendors are net 30, net 60 and even net 90 days. If I can't get this to work, I'll have
      • Link(s) between Notes

        Hello Everyone, It would be great if links could be created between notes. Let's say we have 5 Notes A, B, C , D, E. I would like to be able to link Note A to Note B but not in other way, so no link appears in Note B linking to Note A. An so on, linking
      • Regex in Zoho Mail custom filters is not supported - but it works!

        I recently asked Zoho for help using regex in Zoho Mail custom filters and was told it was NOT supported. This was surprising (and frustrating) as regex in Zoho Mail certainly works, although it does have some quirks* To encourage others, here are 3 regex
      • Need Easy Way to Update Item Prices in Bulk

        Hello Everyone, In Zoho Books, updating selling prices is taking too much time. Right now we have to either edit items one by one or do Excel export/import. It will be very useful if Zoho gives a simple option to: Select multiple items and update prices
      • Feature Request: Assign Documents to Already Entered Bills, Expenses, Invoices, etc.

        Hi Zoho Team, We are regular users of the Documents module in Zoho Books and appreciate its ability to keep financial records well-organized. However, we’ve noticed a limitation: There is no way to attach a document from the "Documents > Files" section
      • I don't see any WITHDRAWL transaction at all

        Hi I manually imported my bank statement to Zoho books today and I am a complete newbie. I have been reading the knowledgebase but unable to fix this. I only see "Uncategorized 91 DEPOSIT transactions". I don't see any WITHDRAWL transaction at all. Also,
      • Shared inbox unable to see replies

        Hi we are a small company me and someone else, we have a shared inbox for our sale@ and contact@ however we have this issue where by if i reply to an email or the other person reply to the email, it does not show it to them and therefore we end up replying
      • Workdrive on Android - Gallery Photo Backups

        Hello, Is there any way of backing up the photos on my android phone directly to a specific folder on Workdrive? Assuming i have the workdrive app installed on the phone in question. Emma
      • Zoho Books | Product updates | August 2025

        Hello users, We’ve rolled out new features and enhancements in Zoho Books. From the right sidebar where you can manage all your widgets, to integrating Zoho Payments feeds in Zoho Books, explore the updates designed to enhance your bookkeeping experience.
      • Kaizen #136 - Zoho CRM Widgets using ReactJS

        Hey there! Welcome back to yet another insightful post in our Kaizen series! In this post, let's explore how to use ReactJS for Zoho CRM widgets. We will utilize the sample widget from one of our previous posts - Geocoding Leads' Addresses in ZOHO CRM
      • 404 error at checkout

        Our customers are getting a 404 error at checkout. Anyone else with the same problem?
      • FONT Sizing in Notebook

        Hi Kishore - What is the status of adding font sizing to the application? I have several things that I have pasted directly into Notebook and the fonts are HUGE! I would like the ability to highlight them and reduce the font to a legible size. Nothing
      • Can managers Upload documents to their direct rapports?

        Admin employees have the ability to upload documents to employees' files; however, managers do not have add/manage button - is it possible for managers to upload their direct reports' documents, such as absence documents or 121 documents. Is there something
      • Leave balance display for next year

        Is there a way to not have a rollover or not limit the leave balance depending on the date. For example an employee has 10 days leave balance and wants to apply for January leave in December. They cant because the rollover doesnt show the leave balance
      • Create a list of customers who participated in specific Zoho Backstage events and send them an email via Zoho CRM

        How to create a list of customers who participated in specific Zoho Backstage events and send them an email via Zoho CRM? I was able to do a view in CRM based on customer that registered to an event, but I don't seems to be able to include the filter
      • Please add an “Auto-Apply Unused Credits” toggle

        Hello — please add a simple org-level option to automatically apply unused credits (credit notes, excess payments, retainers) to new invoices and/or bills. An ON/OFF toggle with choices “invoices”, “bills”, or “both” would save lots of manual work for
      • Zoho Books not working/loading

        Hi! I haven't been able to access/load Zoho Books for the past hours. I get a time out (and it is not due to my internet connection). Could you please check this asap? Thank you!
      • WhatsApp Channels in Zoho Campaigns

        Now that Meta has opened WhatsApp Channels globally, will you add it to Zoho Campaigns? It's another top channel for marketing communications as email and SMS. Thanks.
      • Custom Fields with Data Types for Expense and Payments Received in Zoho Books

        Hi all, We are glad to present to you, the option to create Custom Fields for the Expense and Payments received modules in Zoho Books. This also comes with an icing on top of it - Yes, the custom fields can now be created with different data types. Types like Text, Number, Decimal, Amount, Auto Number and Check Box are supported as of now. Rush to the gear icon at the top right corner, select 'More Settings', choose 'Preferences' in the left pane. Click the Expense/Payment preferences where you can
      • Custom Fonts in Zoho CRM Template Builder

        Hi, I am currently creating a new template for our quotes using the Zoho CRM template builder. However, I noticed that there is no option to add custom fonts to the template builder. It would greatly enhance the flexibility and branding capabilities if
      • [Webinar] Automate sales and presales workflows with Writer

        Sales involves sharing a wide range of documents with customers across the presales, sales, and post-sales stages: NDAs, quotes, invoices, sales orders, and delivery paperwork. Generating and managing these documents manually slows down the overall sales
      • Follow-up emails via Workflow Automation not staying in the same thread

        Dear Zoho Support Team, I am currently using Workflow Automation in Zoho Campaigns to send follow-up emails. In my test case, I noticed the following behavior: All emails in the automation have the same subject line. If the follow-up email is sent within
      • Zoho Cliq - Incident alert (Server outage - IN DC) | August 28

        We've received server down alerts and are currently investigating the issue (IN DC) to find the root cause. Our team is actively working to restore normal operations at the earliest. Status: Under investigation Start time: 09:44:21 AM IST Affected location:
      • Claude + MCP Server + Zoho CRM Integration – AI-Powered Sales Automation

        Hello Zoho Community 👋 I’m excited to share a recent integration we’ve worked on at OfficehubTech: ✅ Claude + MCP Server + Zoho CRM This integration connects Zoho CRM with Claude AI through our custom MCP Server, enabling intelligent AI-driven responses
      • Sync more than one Workdrive

        Hello Please I'm facing some difficulties since some days. In my company we have many zoho accounts in different organisations. And I have to find a way to sync all these Workdrives. I spend many hours to search it on zoho Workdrive but no solution. Could someone help me ? Any idea how I can achieve it ? Thanks in advance. Regards
      • How can I see content of system generated mails from zBooks?

        System generated mails for offers or invices appear in the mail tab of the designated customer. How can I view the content? It also doesn't appear in zMail sent folder.
      • Limitations on editing a message in Cliq

        Hi I've checked the documentations and there's no mention of how many times a message can be edited. When trying with code, I get various numbers such as ~1000 edits or so. Please mention if there's a limit on how many times one can change a message via
      • Problem with reports due to "Connected" items change - Yes this IS a problem

        Now that the change has been made to use "connected" items I can no longer run the reporting I need in CRM. I should be able to start with Deals as the parent, connect down to the Account (Account_Name) on the deal as the child, then to any child items
      • CRM x WorkDrive: File storage for new CRM signups is now powered by WorkDrive

        Availability Editions: All DCs: All Release plan: Released for new signups in all DCs. It will be enabled for existing users in a phased manner in the upcoming months. Help documentation: Documents in Zoho CRM Manage folders in Documents tab Manage files
      • Narrative 10: Assignment Rules - Streamlining Ticket Management

        Behind the scenes of a successful ticketing system: BTS Series Narrative 10: Assignment Rules - Streamlining Ticket Management In the complex world of customer support, a flood of incoming tickets can hit the help desk in seconds. Businesses must do more
      • Free webinar! Digitize recruitment and onboarding with Zoho Sign and Zoho Recruit

        Hello, Tired of being buried in onboarding paperwork? With the integration between Zoho Sign and Zoho Recruit, a powerful applicant tracking system, you can digitize and streamline the entire recruitment and onboarding process, all from one platform.
      • Custom form - Duplicate Value Present

        I am new to Zoho People. I create a new form on Zoho People quite simple. A date (by default the current date) and a radio button with 3 options (Yes, No, Not applicable) I defined the date as ‘No duplicate’ as only one entry by date is allowed. I added:
      • Zoho API connection issues

        Hi, Today at around 1930 GMT our application started experiencing intermittent timeouts from the Zoho API. These intermittent timeouts are not enabling our app to work properly. The API connection was working just fine before. HTTPSConnectionPool(host='www.zohoapis.com',
      • Query Regarding our Partnership between AIC-JKLU and Zoho

        Dear Zoho Team, I am writing to raise a concern on behalf of AIC-JKLU, one of Zoho’s incubator partners. Recently, our startups have been facing difficulties while trying to get themselves onboarded on Zoho through our dedicated partner link. Unfortunately,
      • Getting events in the future

        Hi I am trying to get events in the future by calling this API Endpoint https://www.zohoapis.eu/crm/v8/Events?fields=Event_Title,Created_By,Created_Time,Start_DateTime But that gives me all events in the database. How do I make a query that returns all
      • How can I see content of system generated mails from zBooks?

        System generated mails for offers or invices appear in the mail tab of the designated customer. How can I view the content? It also doesn't appear in zMail sent folder.
      • Next Page