Optimize your Knowledge Base for better visibility by allowing search engine crawling and indexing

Optimize your Knowledge Base for better visibility by allowing search engine crawling and indexing

All you need to know about no-follow and no-index in KB.


What are article crawlability and indexability?  

Crawlability and indexability are vital for making an article visible and accessible to search engines. When a search engine crawls an article, it reviews the site's content to assess its topic and relevance to specific search queries. If a site lacks crawlability, meaning search engines struggle to scan it effectively, the content will not be indexed and will consequently not appear in search results.
 
Additionally, a site’s content must be appropriately optimized for search engine crawlers, allowing it to be indexed accurately and appear in relevant search outcomes. Ensuring every article is crawlable and indexable can significantly enhance its visibility and improve its search engine rankings.

How do you block search indexing?

There are two methods to apply the no-index directive: using a <meta> tag or via an HTTP response header. Both methods achieve the same result, so select the one that suits your website and content type best. Additionally, you can pair the no-index directive with other indexing rules. For instance, you can combine a no-follow directive with an index like this: <meta name="robots" content=" index, nofollow" />.
Quote<meta> tag 

To prevent all search engines that support the no-index rule from indexing a page on your site, place the following <meta> tag into the<head> section of your page:

Quote<meta name="robots" content="noindex"> 
Notes
Certain search engines may interpret the no-index directive in various ways. Therefore, your page could still appear in other search engines' results.

Usage of the no-index tag 

The no-index rule can be implemented using a <meta> tag or an HTTP response header to restrict search engines that recognize the no-index directive, like Google, from indexing certain content. When Googlebot comes across a page with this tag or header, it will remove that page entirely from Google Search results, regardless of whether there are links from other websites.

Notes
To ensure the index rule works, the page or resource must be accessible and not blocked by a robots.txt file. If blocked, the crawler won't see the index rule, though the page might still show in search results if linked from other pages. An index is beneficial if you lack root server access, as it lets you manage page access individually.

What are no-follow links? 

Typically, it's best to allow robots to follow all links on a webpage. Being overly strict in indicating which links should be followed or marked as no-follow may give the impression that the site is trying to influence how a robot views it.

 

Page sculpting is the practice of using no-follow commands to influence the transfer of signals between pages. In the best-case scenario, these strategies to manipulate bots are ineffective. In the worst-case scenario, trying to control bots with nofollow may result in penalties.

When should you use meta no-follow?

 Consequently, there are limited scenarios in which using meta robots no-follow on websites is appropriate. Typically, encountering meta robots no-follow during SEO audits may indicate that a website has been over-optimized.

What are dofollow links?

 Dofollow links, often called "follow" links, are typical online hyperlinks. Unlike links with specific attributes such as “nofollow” or “sponsored,” these links do not carry unique identifiers. As a result, the following links contribute to the ranking control of the pages they connect to.

Experimenting with crawlability and indexability

Before a website's pages can be displayed in search results, search engine bots must first discover all the pages on the site and then analyze them to determine their rankings. The initial phase of finding pages is called crawling, while the subsequent analysis is called indexing.

 

Crawling starts with bots identifying all the URLs of a website's pages. They primarily find these URLs through internal links within the site or through backlinks from external sites. Once a bot identifies a page’s URL, it retrieves content, including the title, text, images, and additional data, such as the date it was last updated. Restrictions can dictate which files and pages a bot can crawl.

 

Indexing occurs after the crawling process. During this stage, bots assess all the information collected from the crawl. They evaluate whether the content is valuable and authoritative and identify the topics associated with the page and how it compares to other relevant pages. Additionally, search engine bots will determine which search results a page might appear in, if any, and its position within those results.

No-follow vs no-index

Let me give you a typical example of an ad campaign landing page designed solely for traffic from typical ads. It's not ideal to prevent organic search traffic from reaching these pages. Implementing meta robots and a no-index tag will keep search engines from ranking them. 

Summing up

  • No-index tells a robot not to index the page. It should be used to keep pages out of search results or to help with low-quality content issues.
  • Meta nofollow tells a robot not to follow a specific link or all of the links on a page. This should typically not be used.
  • Robots should be allowed to determine what gets indexed and shown in search results. If a page is not intended to be visible in search results, it often shouldn't be part of the website.

 

Please watch this space for more detailed use cases of SEO tags that you can apply to your organisation's knowledge base.

 

Cheers,

 

Kavya Rao,

The Zoho Desk Team



      Zoho Campaigns Resources


        • Desk Community Learning Series


        • Digest


        • Functions


        • Meetups


        • Kbase


        • Resources


        • Glossary


        • Desk Marketplace


        • MVP Corner


        • Word of the Day


        • Ask the Experts


          • Sticky Posts

          • Register for Zoho Desk Beta Community

            With the start of the year, we have decided to take a small step in making the life of our customers a little easier. We now have easy access to all our upcoming features and a faster way to request for beta access. We open betas for some of our features
          • Share your Zoho Desk story with us!

            Tell us how you use Zoho Desk for your business and inspire others with your story. Be it a simple workflow rule that helps you navigate complex processes or a macro that saves your team a lot of time; share it here and help the community learn and grow with shared knowledge. 
          • Tip #1: Learn to pick the right channels

            Mail, live chat, telephony, social media, web forms—there are so many support channels out there. Trying to pick the right channels to offer your customers can get pretty confusing. Emails are most useful when the customer wants to put things on record. However, escalated or complicated issues should not be resolved over email because it's slow and impersonal.  When you need immediate responses, live chat is more suitable. It's also quick and convenient, so it's the go-to channel for small issues. 
          • Welcome to Zoho Desk Community - Say hello here!

            Hello everyone! Though we have been here for a while, it’s time to formally establish the Zoho Desk Community; we’re really happy to have you all here! This can be the place where you take a moment to introduce yourself to the rest of the community. We’d love to hear all about you, what you do, what company or industry you work for, how you use Zoho Desk and anything else that you will like to share! Here’s a little about me. I am Chinmayee. I have been associated with Zoho since 2014. I joined here
          • Webinar 1: Blueprint for Customer Service

            With the launch of a host of new features in Zoho Desk, we thought it’ll be great to have a few webinars to help our customers make the most of them. We’re starting off with our most talked about feature, Blueprint in Zoho Desk. You can register for the Blueprint webinar here: The webinar will be delivered by our in-house product experts. This is a good opportunity to ask questions to our experts and understand how Blueprint can help you automate your service processes. We look forward to seeing

          Zoho CRM Plus Resources

            Zoho Books Resources


              Zoho Subscriptions Resources

                Zoho Projects Resources


                  Zoho Sprints Resources


                    Zoho Orchestly Resources


                      Zoho Creator Resources


                        Zoho WorkDrive Resources



                          Zoho CRM Resources

                          • CRM Community Learning Series

                            CRM Community Learning Series


                          • Tips

                            Tips

                          • Functions

                            Functions

                          • Meetups

                            Meetups

                          • Kbase

                            Kbase

                          • Resources

                            Resources

                          • Digest

                            Digest

                          • CRM Marketplace

                            CRM Marketplace

                          • MVP Corner

                            MVP Corner




                            Zoho Writer Writer

                            Get Started. Write Away!

                            Writer is a powerful online word processor, designed for collaborative work.

                              Zoho CRM コンテンツ




                                ご検討中の方

                                  • Recent Topics

                                  • Layout Adjustment.

                                    Hello Latha, Hope you are doing well. I wanted to check with you about adjusting the layout columns in the Service Report from the backend, as currently there is no capability for us to customize the layout from our end. Below is the output that I am
                                  • Invoice emails repeatedly "Bounced back" from perfectly good email address of clients.

                                    Hi For a couple of months now I am getting tens of invoice email invoices "bounced back" from perfectly sound email addresses when they are sent via Zoho Books. (see attached or an example) It is clearly an issue with Zoho books but as yet they have failed
                                  • Newby Questions - Vendors, Customers, and Income vs. Other Income Clarifications

                                    Q1. For Deposits, there does not seem to be an option for "Income". "Other Income" is an option however. What is the process to add the option to assign a deposit to type "Income"? Q2. In many cases Vendors and Customer are the same. Vendors may purchase
                                  • yodlee bank fees

                                    My bank feeds have been pretty much error-free until this recent Yodlee update.  I have configured it as per the information however nothing comes through. It says that bank feeds have been refreshed as of 30th June, 2022. It is the end of the financial
                                  • Deleting, Adding and Changing a Zoho Books Entry

                                    Zoho Books Community & Support, If a bulk upload of entries were added to an account, what is the process to: - Delete an entry - Change the amount of an entry - Is there a means to manually add an entry other than upload?
                                  • Create Item group from a composite Item

                                    I have applied my mind for hours but cannot figure this out. Can you have a composite item in an item group?. E.g. We bundle different color and size SKU's together as composite items. Also Using composite items as Bill of Materials. We want to create
                                  • Better use of contacts

                                    Zoho inventory has the ability to add multiple contacts to customers. However Zoho inventory doesn't currently provide a way to link a contact to objects like sales orders. This means that while you can tell what company has placed a sales order you can't
                                  • Amazon.in FBA multiple warehouse integration with Zoho Inventory

                                    My organisation subscribed to Zoho One looking at the opportunity to integrate Amazon.in with Inventory. But during the configuration, we understood the integration has severe limitations when it involves multiple warehouses in the same Organisation.
                                  • Feature Request - Make Available "Alias Name" Field In Item List View

                                    Hi Zoho Inventory Team, I have noticed that the "Alias Name" field does not appear on the list of selectable columns in the Customise Columns feature in the Items module. This would be very useful to see for businesses who are using the Alias Name field
                                  • Feature Request - Option To Hide Default System Fields on Items

                                    Hi Zoho Inventory Team, As far as I know it is not possible to hid some of the defult system fields on Items, such as UPC, MPN, EAN, ISBN. A good use case is that in many cases ISBN is not relevant and it would be an improved user experience if we could
                                  • OpenAPI Specs are just plain wrong

                                    The provided yml files for generating the OpenAPI specs are absolutely riddled with errors and inconsistencies. From missing fields on the objects, to just incorrectly named resource objects. I'm having to go through and manually changing the spec to
                                  • Zoho Inventory - Composite Items - Assembly - Single Line Item Quantity of One

                                    Hi Zoho Inventory Team, Please consider relaxing the system rules which prevent an assembly items from consisting of a single line item and outputting a quantity of 1. A client I'm currently working with sells cosmetics and offers testers of their products
                                  • Categorize Items with Item Headers

                                    Hello customers, Did you ever want to classify items based on specific categories to help your customers understand your invoice better? With the new Item Header feature, you can easily categorize items in your invoices and estimates and give them a common title.  Item Headers are available in the Invoices, the Recurring Invoices and the Estimates module. It can be carried forward from estimates to invoices at the time of converting the estimates.  To add an item header: Go to the Estimates, Invoices
                                  • Push Forms to CRM

                                    Hello, I created a Form but forgot to integrate to CRM before many of the Forms were filled out. Is it possible to push those entries to CRM or should I just export a CSV then upload to CRM?
                                  • Zoho Creator Developer Console | Improved Distribution and Lifecycle Management for apps

                                    Hello everyone, We're excited to introduce new enhancements now in the Zoho Creator Developer Console. These updates strengthen private app distribution through licensing controls and extend environment support across all installed apps, helping teams
                                  • Logged out

                                    Hi, just been working on a sheet when a pop up box appeared telling me I'm going to be logged out in x number of seconds and if I reload I may lose any edits, or words to that effect. It did indeed log me out and I did indeed lose my last edits. Any idea
                                  • 554 5.1.1 – Mail sending blocked for the domain(s): [gmail.com]

                                    Here's your corrected text: Hello, I hope you are doing well. I was unable to send a message and received the following error: "554 5.1.1 – Mail sending blocked for the domain(s): [gmail.com]" I tried to send and deliver an email but got this error. I
                                  • You have reached the maximum limit of bank accounts that can be connected to Zoho Books through token.

                                    I can no longer connect to my bank account to download transactions into Zoho Books. I egt the error message: "You have reached the maximum limit of bank accounts that can be connected to Zoho Books through token. To connect more accounts, write to us
                                  • First Name in Mail

                                    While sending a mail/message to the user, I want only the first name to be displayed—for example: “Hi John” instead of the full name using "Hi ${Name_Field}"
                                  • Can you import projects into Zoho Projects yet?

                                    I see some very old posts asking about importing project records into Zoho Projects. But I can't find anything up to date about the topic. Has this functionality been added? Importing tasks is helpful. But we do have a project where importing projects
                                  • Connectivity issues with Google Calendar and third-party integrations

                                    Description: We are currently experiencing a critical failure with Zoho CRM third-party connections. This issue is heavily affecting our primary workflow. Symptoms: Sync Failure: Existing Zoho CRM to Google Calendar connections have been failing for approximately
                                  • Add Reauthentication Option for Zoho Bug Tracker Integration in Zoho Desk

                                    Hello Zoho Desk Team, We hope you're doing well. We would like to request an enhancement to the Zoho Bug Tracker integration within Zoho Desk. Current Limitation: At the moment, there is no option to reauthenticate the Zoho Bug Tracker integration in
                                  • Newby Questions

                                    Q1. The top bar of Zoho Books has a "Search in Banking (/) " field. What is the proper use of this text box? - Searching for Amazon for example has no results but there are transactions. - Is the search case sensitive? - Are regular expressions allowed?
                                  • Transaction Rule Matching

                                    Q1. Is there a method to have the Transaction Rules applied across multiple Bank Accounts? Q2. To match "contains" text in Transaction Rules, is a wildcard or regular expression required? Is it case sensitive? Would searching for "issue" in the following
                                  • Introducing parent-child ticketing in Zoho Desk [Early access]

                                    Hello Zoho Desk users! We have introduced the parent-child ticketing system to help customer service teams ensure efficient resolution of issues involving multiple, related tickets. You can now combine repetitive and interconnected tickets into parent-child
                                  • Three Zoho Billing Limitations Blocking Standard Subscription Operations

                                    After working through Zoho Billing support for over a year on these three issues without resolution, we wanted to flag them to the broader community. We are curious whether other businesses are running into the same walls. 1. Cannot Prepone (Move Earlier)
                                  • Automation #3 - Auto-sync email attachments to tickets

                                    This is a monthly series where we pick some common use cases that have been either discussed or most asked about in our community and explain how they can be achieved using one of the automation capabilities in Zoho Desk. Most of our customers use email
                                  • Can't add a sender adress from zoho campaigns

                                    hi, I need to change the sender address for a campaign.  When i try to add it i get a message to say 'duplicated email address found while adding your sender address'.  This is the first campaign i'm sending so I don't understand why this message is displayed? Thanks Jane 
                                  • Using a custom single line External ID form as merge fields in templates

                                    Hey everyone, We're looking to integrate a few external systems better with our Zoho CRM, and we had hoped to use external fields for this purpose. In this case, it would mean being able to use our own inoice system's invoice numbers are a direct id compatible
                                  • Introducing Workqueue: your all-in-one view to manage daily work

                                    Hello all, We’re excited to introduce a major productivity boost to your CRM experience: Workqueue, a dynamic, all-in-one workspace that brings every important sales activity, approval, and follow-up right to your fingertips. What is Workqueue? Sales
                                  • Zoho CRM - Feature Request - Analytics Components Group By Week Alternative Formats

                                    On the Zoho CRM Analytics Components, please consider adding an option to allow group by week format to be changed to other formats such as Week Commencing or Week Ending dates, rather than the current Week Number. This would provide improved usability
                                  • Dashboard target enhancements

                                    Often individuals in IT are creating dashboards for their sales team. The ability to create a single dashboard that can be used by multiple people is key. A components for a dashboard have the ability to filter by logged in user which is great. However
                                  • Zoho Campaigns - Feature Request - Re-Send Existing Email Action in Automations/Journeys

                                    Hi Zoho Campaigns and Zoho Marketing Automation teams, I would like to suggest a feature that would make building and managing complex automations significantly easier. The Feature Introduce a “Re-Send” or “Send Existing Email” action within Automations
                                  • Can I write a check in Zoho Books with no associated bill?

                                    This currently does not seem possible, and I have a client that desperately needs this function if I am able to convert them with Quickbooks. Thank you in advance for your reply. 
                                  • ISO 3166-2 Compliant Country and Regions

                                    Zoho should consider implementing ISO 3166-2 standards for countries and regions in address dropdowns. It adds a lot of noise in the data when items added programmatically conform to ISO 3166-2 (e.g., TX for Texas, CA for Canada, etc.), but the few items
                                  • US State abbreviations in Address fields

                                    In regards to all Address fields within Zoho, Is there a way to change the State field to be the 2 letter abbreviation vs the full spelled out US State name? Example: "Washington" should be WA. I am able to type in the abbreviated state, but it's not
                                  • Zoho Booking - TIN vs ATIN & ITIN

                                    Zoho Booking Vendors allows for TAX ID values of SSN, EIN, ATIN an ITIN. There is no option for TIN. What is the method to properly add TIN to the list of taxable values for companies? For reference: Social Security Numbers (SSN) Individual Taxpayer Identification
                                  • How to see Statement Details Shown in Unclassified Transactions in All Transactions?

                                    All, The list of Unclassified Transactions show the Statement Details and Descriptions. What is the method to see that column in the All Transactions list? -Thanks!
                                  • Zoho Books | Product updates | October 2025

                                    Hello users, We’ve rolled out new features and enhancements in Zoho Books. From iOS 26 updates to viewing reports as charts, explore the updates designed to enhance your bookkeeping experience. Zoho Books Updates for Apple Devices At WWDC 2025, Apple
                                  • Bill sent for Payment Approval

                                    Zoho Books currently supports three levels of transaction approval for bills: Draft, Submit, and Approved. We are looking to add a layer of approval like a bill marked for payment before initiating payment requests through HSBC Bank. Can you please guide
                                  • Next Page