What are article crawlability and indexability?

What are article crawlability and indexability?

All you need to know about no-follow and no-index in KB.


What are article crawlability and indexability?  

Crawlability and indexability are vital for making an article visible and accessible to search engines. When a search engine crawls an article, it reviews the site's content to assess its topic and relevance to specific search queries. If a site lacks crawlability, meaning search engines struggle to scan it effectively, the content will not be indexed and will consequently not appear in search results.
 
Additionally, a site’s content must be appropriately optimized for search engine crawlers, allowing it to be indexed accurately and appear in relevant search outcomes. Ensuring every article is crawlable and indexable can significantly enhance its visibility and improve its search engine rankings.

How do you block search indexing?

There are two methods to apply the no-index directive: using a <meta> tag or via an HTTP response header. Both methods achieve the same result, so select the one that suits your website and content type best. Additionally, you can pair the no-index directive with other indexing rules. For instance, you can combine a no-follow directive with an index like this: <meta name="robots" content=" index, nofollow" />.
Quote<meta> tag 

To prevent all search engines that support the no-index rule from indexing a page on your site, place the following <meta> tag into the<head> section of your page:

Quote<meta name="robots" content="noindex"> 
Notes
Certain search engines may interpret the no-index directive in various ways. Therefore, your page could still appear in other search engines' results.

Usage of the no-index tag 

The no-index rule can be implemented using a <meta> tag or an HTTP response header to restrict search engines that recognize the no-index directive, like Google, from indexing certain content. When Googlebot comes across a page with this tag or header, it will remove that page entirely from Google Search results, regardless of whether there are links from other websites.

Notes
To ensure the index rule works, the page or resource must be accessible and not blocked by a robots.txt file. If blocked, the crawler won't see the index rule, though the page might still show in search results if linked from other pages. An index is beneficial if you lack root server access, as it lets you manage page access individually.

What are no-follow links? 

Typically, it's best to allow robots to follow all links on a webpage. Being overly strict in indicating which links should be followed or marked as no-follow may give the impression that the site is trying to influence how a robot views it.

 

Page sculpting is the practice of using no-follow commands to influence the transfer of signals between pages. In the best-case scenario, these strategies to manipulate bots are ineffective. In the worst-case scenario, trying to control bots with nofollow may result in penalties.

When should you use meta no-follow?

 Consequently, there are limited scenarios in which using meta robots no-follow on websites is appropriate. Typically, encountering meta robots no-follow during SEO audits may indicate that a website has been over-optimized.

What are dofollow links?

 Dofollow links, often called "follow" links, are typical online hyperlinks. Unlike links with specific attributes such as “nofollow” or “sponsored,” these links do not carry unique identifiers. As a result, the following links contribute to the ranking control of the pages they connect to.

Experimenting with crawlability and indexability

Before a website's pages can be displayed in search results, search engine bots must first discover all the pages on the site and then analyze them to determine their rankings. The initial phase of finding pages is called crawling, while the subsequent analysis is called indexing.

 

Crawling starts with bots identifying all the URLs of a website's pages. They primarily find these URLs through internal links within the site or through backlinks from external sites. Once a bot identifies a page’s URL, it retrieves content, including the title, text, images, and additional data, such as the date it was last updated. Restrictions can dictate which files and pages a bot can crawl.

 

Indexing occurs after the crawling process. During this stage, bots assess all the information collected from the crawl. They evaluate whether the content is valuable and authoritative and identify the topics associated with the page and how it compares to other relevant pages. Additionally, search engine bots will determine which search results a page might appear in, if any, and its position within those results.

No-follow vs no-index

Let me give you a typical example of an ad campaign landing page designed solely for traffic from typical ads. It's not ideal to prevent organic search traffic from reaching these pages. Implementing meta robots and a no-index tag will keep search engines from ranking them. 

Summing up

  • No-index tells a robot not to index the page. It should be used to keep pages out of search results or to help with low-quality content issues.
  • Meta nofollow tells a robot not to follow a specific link or all of the links on a page. This should typically not be used.
  • Robots should be allowed to determine what gets indexed and shown in search results. If a page is not intended to be visible in search results, it often shouldn't be part of the website.

 

Please watch this space for more detailed use cases of SEO tags that you can apply to your organisation's knowledge base.

 

Cheers,

 

Kavya Rao,

The Zoho Desk Team


    • Sticky Posts

    • Register for Zoho Desk Beta Community

      With the start of the year, we have decided to take a small step in making the life of our customers a little easier. We now have easy access to all our upcoming features and a faster way to request for beta access. We open betas for some of our features

    • Share your Zoho Desk story with us!

      Tell us how you use Zoho Desk for your business and inspire others with your story. Be it a simple workflow rule that helps you navigate complex processes or a macro that saves your team a lot of time; share it here and help the community learn and grow with shared knowledge. 

    • Tip #1: Learn to pick the right channels

      Mail, live chat, telephony, social media, web forms—there are so many support channels out there. Trying to pick the right channels to offer your customers can get pretty confusing. Emails are most useful when the customer wants to put things on record. However, escalated or complicated issues should not be resolved over email because it's slow and impersonal.  When you need immediate responses, live chat is more suitable. It's also quick and convenient, so it's the go-to channel for small issues. 

    • Welcome to Zoho Desk Community - Say hello here!

      Hello everyone! Though we have been here for a while, it’s time to formally establish the Zoho Desk Community; we’re really happy to have you all here! This can be the place where you take a moment to introduce yourself to the rest of the community. We’d love to hear all about you, what you do, what company or industry you work for, how you use Zoho Desk and anything else that you will like to share! Here’s a little about me. I am Chinmayee. I have been associated with Zoho since 2014. I joined here

    • Webinar 1: Blueprint for Customer Service

      With the launch of a host of new features in Zoho Desk, we thought it’ll be great to have a few webinars to help our customers make the most of them. We’re starting off with our most talked about feature, Blueprint in Zoho Desk. You can register for the Blueprint webinar here: The webinar will be delivered by our in-house product experts. This is a good opportunity to ask questions to our experts and understand how Blueprint can help you automate your service processes. We look forward to seeing

    • Recent Topics

    • Recording overpayment?

      So a customer just overpaid me and how do I record this? I can't enter an amount that is higher than the invoice amount. Eg. Invoice is $195 and he sent $200. He's a reccuring customer so is there a way to record so that he has a $5 advance for future invoice?

    • How can I add Material cost to a project?

      Hello, We use Zoho project to manage scheduled work. This consists in labour tasks and matrials cost. How can I add the material budget, then teh cost to the project? Thanks Barbara

    • Bigin-Zoho Projects Integration

      We're pleased to announce the Bigin-Zoho Projects integration, which enables you to manage client relationships and project tasks simultaneously in one system. With this integration, once you close a deal in Bigin, you can create a project in Zoho Projects

    • Managing independent contractors

      We use Independent contractors as our field techs. I'm looking for a solution to be able to manage there timesheets, payments, and etc

    • License Issue

      Hello Campaigns Team, we have 17 ZOHO One licenses but in campaigns I get the message that we have a free plan. How can we use ZOHO campaigns? As far as I understood campaigns is a part of ZOHO One. THX Niels

    • Count Function in Zoho Table

      Hi Zoho Team, Hope you're doing well. We would like to request the addition of a Count function in Zoho Table, similar to what exists in Excel, Google Sheets, and Airtable. Currently, there is no built-in way to count the occurrences of unique values

    • Leads, Prospects, Contacts, Clients and Me

      I'm trying to adapt my process to Zoho and am getting confused. Here's how I did it before. Maybe it's just terminology...? Leads - anyone from any source that I might want as a client. Leads include people that I am sending emails or tapping in social

    • Contacts, Leads, Prospects, Opportunities, Deals, Oh my!

      I've implemented three CRM's in previous companies, and led the effort for two of those.  I'm pretty familiar with sales pipeline management. For my current company, I chose Zoho One because of the broad range it includes for other business processes, and am generally quite happy.  So now it's time to dig into Zoho CRM And I find myself a bit befuddled. In other implementations, a "Contact" was just that- a name and some contact information.  Could be anybody whether interested in doing business

    • Visual Sync Status Indicator

      I think I've see in documentation that WorkDrive had the industry standard of indicating a sync status for individual files and folders. I'm just starting with WD and synced my first test folder, but there was no way to tell what's happening by just looking

    • Is there a way to make the new tab button open a new cloud document?

      I'm coming from Google Docs. Plus, is there any way to have a set a default font? I'm using the desktop app and I prefer the options I can get through the cloud documents. I want the convenience of just opening a new cloud document rather than doc that's

    • Tip 26: How to hide the "Submit" button from a form

      Hi everyone, Hope you're staying safe and working from home. We are, too. By now, we at Zoho are all very much accustomed to the new normal—working remotely. Today, we're back with yet another simple but interesting tip--how to hide the Submit button from your forms. In certain scenarios, you may want to hide the submit button from a form until all the fields are filled in.  Use case In this tip, we'll show you how to hide the Submit button while the user is entering data into the form, and then

    • Trying to export a report to Excel via a deluge script

      I have this code from other posts but it gives me an error of improper statement, due to missing ; at end of line or incomplete expression. Tried lots of variations to no avail. openUrl(https://creatorapp.zoho.com/<username>/<app name>/XLSX/#Report:<reportname>,"same

    • Bigin_Email Notification not being sent when a new lead is created

      I have a workflow in BIGIN set to send an email notification when a new lead is created via a webform (integrated with ZohoForm) The trigger is whenever a contact is "Create or Edit". Conditioning was applied for contacts which source is "Website" If

    • ZOHO CRM API Python SDK Convert Quote to Sales Order

      I can see footprints that this may be possible through Inventory Conversion. But I am unable to locate any specific details or samples on how to do this. I am using the most current Python SDK. Any support or even sample code would be much appreciated

    • Displaying Notes/Description Columns in "All Expense" showing Tabs

      It's surprising to see there is no option to view description columns in tab showing all expenses. There are provisions for Reference# and Status, but why not the description/notes. Please Add. Thank You.

    • Zoho Books - uploading company logo squashed

      I am trying to upload my company logo with the following dimensions - 240 x 240 pixels and a file size of 106Kb. When I look at the logo in my invoices, it is squashed and not the right size. Any idea what is going on? I've tried uploading jpeg and png

    • Petty cash discrepancy

      How do I record a petty cash discrepancy? We had money go missing and need to document that in the books, but I'm not sure how to put that in. It's not an expense, just a loss of cash.

    • Zoho Workdrive file versions

      Hello. I have Workdrive setup to sync files offline to an external hard drive. The off line sync folder currently shows at 1.42 TB. I have a 5 TB storage limit in Workdrive. The cloud version of Workdrive says that I have used all 5 TB! I have 27, 285

    • Integration with Zoho CRM?

      Will it be possible to integrate WorkDrive with CRM similar do Zoho Docs?

    • How to control the # of version of files to keep?

      Currently most of the WorkDrive Storage comprise of the many versions of files saved. How do I save some space and reduce the number of version of date or files saved in WorkDrive? Thanks

    • Should I Use Zoho Mail Calendar, or Zoho CRM Calendar, or Zoho Calendar?

      After a couple of dozens Zoho solopreneur products that I transitioned to after becoming a Zoho One enthusiast 5 years ago, I am finally preparing to conquer the remaining two bastions: Mail and WorkDrive (using Google Workspace at the moment). A NYC

    • Add comments to a form

      Hello, I'm trying to add comments to a form using a subform with one field named comment, but I don't want prior comments to be editable or deleteable by anyone (except the admin).  Is there a way to only display prior comments (with a datetime, user and comment field preferably) but still be able to add new ones when editing the main form?  I'm not tied to subforms if there is an easier was to do this?

    • Zoho Books | Product updates | March 2025

      Hello users, We have rolled out new updates in Zoho Books to enhance your accounting experience. These include the ability to create workflow rules for manual journals and Multi-Factor Authentication (MFA) for customer and vendor portals. Explore these

    • Work Drive Tray Icon Missing

      How can I get the tray icon back? The app froze, had to restart PC and then it's been gone since...  I've re-installed the windows program and restarted my machine twice now.

    • cant receive emails

      I have checked the Dns and everything seems to be fine pls check the print screens attached below help me cause i need to solve this fast

    • Retainer invoice in Zoho Finance modlue

      Hello, Is there a way of creating retainer invoices in the Zoho Finance module? If not can I request this is considered for future updates please.

    • iOS Widget Not Working

      It appears that the iOS widget is not working, displaying a blank white screen instead of a selected note. I’m using app version 6.5.12 and iOS 18.3.1.

    • Two Problems With Data Imported to Notes

      Occasionally I want to create a note by copying and pasting a few paragraphs from an article on line. When I create a new note and paste in the section the newly created note winds up with each paragraph in white text on a dark background rather than

    • Workdrive on Android - Gallery Photo Backups

      Hello, Is there any way of backing up the photos on my android phone directly to a specific folder on Workdrive? Assuming i have the workdrive app installed on the phone in question. Emma

    • Generate a link for Zoho Sign we can copy and use in a separate email

      Please consider adding functionality that would all a user to copy a reminder link so that we can include it in a personalized email instead of sending a Zoho reminder. Or, allow us to customize the reminder email. Use Case: We have clients we need to

    • How to associate a document sent in Zoho Sign with an deal in the CRM?

      Hi, often documents are loaded in Zoho sign and sent for signature. These sometimes are linked to a deal in the Zoho CRM and would be nice to see the status of the document within the CRM. I am aware of the integration, but that assumes that the document

    • Preventing auto-redirect to Parent Record on Save...

      Our users often create records from the related list on th left side of the screen. They click the blue "plus" button to create the record. This is handy, but for some modules, or situations, they would like to remain on the record AFTER clicking "Save",

    • CRM Portal Help

      Hello, I am trying to set up a portal to connect with our referring doctors to keep patient cases organized. I set up the accounts module as office, the contacts as doctors, the leads as patients, and the deals as treatments. Everything seems to work

    • Zoho Books (UK) needs to be able to submit a CT600 CTSA return

      As well as a VAT Return, most (if not all) small businesses have to submit a CT600 Corporation Tax Self-Assessment. There are many providers who do this (like Xero) bujt not Zoho. Can you add this to the request list please? Many thanks Steve

    • No image image comes out in the recipient when I sent an email

      Hello to the entire forum, when I send an email from Zoho, my profile picture does not come out. On the other hand, if you do, using Gmail accounts. How is it configured to leave ??? Thank you Greetings !!

    • Zoho Desk & Tasks

      Hi, I'd like to be able to create a set of tasks each time a customer request comes in, as I understand it, currently each would need to be create manually. Project is too much of an overhead for what we want to use. Effectively in various use cases we

    • zet pack not working

      We are using the zet pack command to package our Zoho extension. However, after running the command, the extension gets packed, but the resulting package is empty. We've attached a screenshot for reference. Could you please assist us with resolving this

    • While retrieving the Balance Sheet Report, there is always this "COST OF GOODS SOLD", This is not editable.

      Hi Zoho & Readers, While retrieving the Balance Sheet Report, there is always this "COST OF GOODS SOLD", which is reduced from the Sales to arrive at the gross profit. The issue I face here is that Service Oriented Companies don't incur any COGS, hence

    • Changing salesorder_number via zoho flow

      For some reason updating salesorder_number via zoho flow does not stick. Flow is triggered by new sales order filtered by sales channel update sales order: PO#: CX${trigger.reference_number} Salesorder_number: CX${trigger.reference_number} PO# successfully

    • Working with Products that are non-tangible

      How does one create a 'service' in products? Is there a way to disable inventory functions for things like Sofware as a service? The services module doesn't look to be much help either. Not sure how to do this in CRM

    • Next Page