Optimize your Knowledge Base for better visibility by allowing search engine crawling and indexing

Optimize your Knowledge Base for better visibility by allowing search engine crawling and indexing

All you need to know about no-follow and no-index in KB.


What are article crawlability and indexability?  

Crawlability and indexability are vital for making an article visible and accessible to search engines. When a search engine crawls an article, it reviews the site's content to assess its topic and relevance to specific search queries. If a site lacks crawlability, meaning search engines struggle to scan it effectively, the content will not be indexed and will consequently not appear in search results.
 
Additionally, a site’s content must be appropriately optimized for search engine crawlers, allowing it to be indexed accurately and appear in relevant search outcomes. Ensuring every article is crawlable and indexable can significantly enhance its visibility and improve its search engine rankings.

How do you block search indexing?

There are two methods to apply the no-index directive: using a <meta> tag or via an HTTP response header. Both methods achieve the same result, so select the one that suits your website and content type best. Additionally, you can pair the no-index directive with other indexing rules. For instance, you can combine a no-follow directive with an index like this: <meta name="robots" content=" index, nofollow" />.
Quote<meta> tag 

To prevent all search engines that support the no-index rule from indexing a page on your site, place the following <meta> tag into the<head> section of your page:

Quote<meta name="robots" content="noindex"> 
Notes
Certain search engines may interpret the no-index directive in various ways. Therefore, your page could still appear in other search engines' results.

Usage of the no-index tag 

The no-index rule can be implemented using a <meta> tag or an HTTP response header to restrict search engines that recognize the no-index directive, like Google, from indexing certain content. When Googlebot comes across a page with this tag or header, it will remove that page entirely from Google Search results, regardless of whether there are links from other websites.

Notes
To ensure the index rule works, the page or resource must be accessible and not blocked by a robots.txt file. If blocked, the crawler won't see the index rule, though the page might still show in search results if linked from other pages. An index is beneficial if you lack root server access, as it lets you manage page access individually.

What are no-follow links? 

Typically, it's best to allow robots to follow all links on a webpage. Being overly strict in indicating which links should be followed or marked as no-follow may give the impression that the site is trying to influence how a robot views it.

 

Page sculpting is the practice of using no-follow commands to influence the transfer of signals between pages. In the best-case scenario, these strategies to manipulate bots are ineffective. In the worst-case scenario, trying to control bots with nofollow may result in penalties.

When should you use meta no-follow?

 Consequently, there are limited scenarios in which using meta robots no-follow on websites is appropriate. Typically, encountering meta robots no-follow during SEO audits may indicate that a website has been over-optimized.

What are dofollow links?

 Dofollow links, often called "follow" links, are typical online hyperlinks. Unlike links with specific attributes such as “nofollow” or “sponsored,” these links do not carry unique identifiers. As a result, the following links contribute to the ranking control of the pages they connect to.

Experimenting with crawlability and indexability

Before a website's pages can be displayed in search results, search engine bots must first discover all the pages on the site and then analyze them to determine their rankings. The initial phase of finding pages is called crawling, while the subsequent analysis is called indexing.

 

Crawling starts with bots identifying all the URLs of a website's pages. They primarily find these URLs through internal links within the site or through backlinks from external sites. Once a bot identifies a page’s URL, it retrieves content, including the title, text, images, and additional data, such as the date it was last updated. Restrictions can dictate which files and pages a bot can crawl.

 

Indexing occurs after the crawling process. During this stage, bots assess all the information collected from the crawl. They evaluate whether the content is valuable and authoritative and identify the topics associated with the page and how it compares to other relevant pages. Additionally, search engine bots will determine which search results a page might appear in, if any, and its position within those results.

No-follow vs no-index

Let me give you a typical example of an ad campaign landing page designed solely for traffic from typical ads. It's not ideal to prevent organic search traffic from reaching these pages. Implementing meta robots and a no-index tag will keep search engines from ranking them. 

Summing up

  • No-index tells a robot not to index the page. It should be used to keep pages out of search results or to help with low-quality content issues.
  • Meta nofollow tells a robot not to follow a specific link or all of the links on a page. This should typically not be used.
  • Robots should be allowed to determine what gets indexed and shown in search results. If a page is not intended to be visible in search results, it often shouldn't be part of the website.

 

Please watch this space for more detailed use cases of SEO tags that you can apply to your organisation's knowledge base.

 

Cheers,

 

Kavya Rao,

The Zoho Desk Team


    • Sticky Posts

    • Register for Zoho Desk Beta Community

      With the start of the year, we have decided to take a small step in making the life of our customers a little easier. We now have easy access to all our upcoming features and a faster way to request for beta access. We open betas for some of our features

    • Share your Zoho Desk story with us!

      Tell us how you use Zoho Desk for your business and inspire others with your story. Be it a simple workflow rule that helps you navigate complex processes or a macro that saves your team a lot of time; share it here and help the community learn and grow with shared knowledge. 

    • Tip #1: Learn to pick the right channels

      Mail, live chat, telephony, social media, web forms—there are so many support channels out there. Trying to pick the right channels to offer your customers can get pretty confusing. Emails are most useful when the customer wants to put things on record. However, escalated or complicated issues should not be resolved over email because it's slow and impersonal.  When you need immediate responses, live chat is more suitable. It's also quick and convenient, so it's the go-to channel for small issues. 

    • Welcome to Zoho Desk Community - Say hello here!

      Hello everyone! Though we have been here for a while, it’s time to formally establish the Zoho Desk Community; we’re really happy to have you all here! This can be the place where you take a moment to introduce yourself to the rest of the community. We’d love to hear all about you, what you do, what company or industry you work for, how you use Zoho Desk and anything else that you will like to share! Here’s a little about me. I am Chinmayee. I have been associated with Zoho since 2014. I joined here

    • Webinar 1: Blueprint for Customer Service

      With the launch of a host of new features in Zoho Desk, we thought it’ll be great to have a few webinars to help our customers make the most of them. We’re starting off with our most talked about feature, Blueprint in Zoho Desk. You can register for the Blueprint webinar here: The webinar will be delivered by our in-house product experts. This is a good opportunity to ask questions to our experts and understand how Blueprint can help you automate your service processes. We look forward to seeing

    • Recent Topics

    • Make Quick Edits to Images Before Attaching

      Hello everyone, We have enhanced how attachments are handled in tickets to help agents preview and share files more efficiently in Zoho Desk. Agents can preview image attachments before adding them to tickets and edit them using attachment annotator.

    • 3/18 オンライン勉強会のお知らせ Zoho ワークアウト （無料）

      ユーザーの皆さま、こんにちは。コミュニティチームの中野です。 3月開催のZoho ワークアウトの開催が決定しましたのでご案内します。 今回はZoomにて、オンライン開催します。 ▶︎参加登録はこちら（無料） https://us02web.zoom.us/meeting/register/BoNTN7zYR8OvOPGShqBY0A ━━━━━━━━━━━━━━━━━━━━━━━━ Zoho ワークアウトとは？ Zoho ユーザー同士で交流しながら、サービスに関する疑問や不明点の解消を目指すイベントです。

    • Extend color coding to custom picklist fields

      Objectively, Projects has the best UI of any Zoho app — clean, intuitive, and never feels bloated. Big props to whoever owns the design. Feature request: color coding for custom picklist field values in field customization. You've already done it in two

    • New in Office Integrator: In-sheet text translation

      Hi users, We're pleased to introduce translation capability in the spreadsheet editor in Zoho Office Integrator. This allows you to translate the text in your spreadsheet's cells into 70+ languages from within your web app. Office Integrator's spreadsheet

    • Streamline email communication with Out of Office configuration

      Managing user communication effectively is the key to ensuring timely responses and consistent messaging. However, when users are unavailable, the absence of an Out of Office response can lead to delays and missed expectations. Managing these settings

    • Changing settings for auto logoff

      I've noticed that when I haven't used Cliq for a while, I have to re-enter my password. That is really clumsy, especially if you have a complicated password. Because it won't be filled in automatically. Is there a way to change that behaviour? We are

    • A few Issues when using "Pay Bill via Check"

      We have quite a bit of issues with how paying for Bills via Check works. Would love some feedback from the Zoho team in case we are doing something incorrectly. 1. When we go from a vendor and select "Pay Bill via Check" option, we see ALL the outstanding

    • Prevent tracking users from specific countries

      Currently, I’m receiving many bot visits from the United States and Malaysia. I would like these visits not to be recorded in SalesIQ. I already enabled the option to exclude traffic from cloud service providers, but I’m still receiving bot visits. Ideally,

    • My client requires me to have custom pdf file names to except payment for invoices, how can I customize this before emailing.

      Hello! I love the program so far but there are a few things that are standing in the way. I hope you guys can code them in so I can keep the program for years to come. My client requires I customize the pdf file names I send in for billing. Can you please

    • Edit Project Number?

      Hi all: We just signed up for a trial of zoho one, which includes ZoHo Projects. We've noticed there was a 'dummy project' preloaded in projects to help familiarize yourself with the software. We've created a couple of our own projects now but noticed since the dummy project was preloaded, our projects start with number 2 then 3, sequentially. Since it seems we will be keeping zoho past the trial, If we delete the dummy project, how do we get our own projects renumbered, beginning with 1? We'd like

    • Download pricebook products & details - not just pricebook creation date & name

      We're looking to download a copy of a pricebook and its associated products & book prices (as we have several offices in different countries selling the same products), however, when using the export feature under Data administration it only gives me

    • 554 5.1.1 – Mail sending blocked for the domain(s): [gmail.com]

      Here's your corrected text: Hello, I hope you are doing well. I was unable to send a message and received the following error: "554 5.1.1 – Mail sending blocked for the domain(s): [gmail.com]" I tried to send and deliver an email but got this error. I

    • Outgoing Mail Blocked – Suspicious Login Activity (Need Clarification and Solution)

      Hello, I’m currently facing an issue where my Zoho Mail account has been blocked due to “suspicious login activity,” and outgoing emails are restricted. Here are the details shown: Block type: Outgoing mail blocked Reason: Suspicious login activity A

    • Assign Meeting in records

      It would be nice to be able to "call and assing" meetings from a record, for example from a Deal. Right now - calendar is synced with CRM - meetings show in calendar - you can go in each meeting and assign it to a record It would be nice to be able to

    • Allow Global Admin to access/edit all forms without changing owners

      Hi there, Please consider adding a feature where the Global Admin of the account an automatically access/edit any form in the Company Account. I'm the Global Admin on our Zoho One plan, and we have multiple users that use/create forms. But for me to access

    • ERROR: "Please enter a valid Phone"

      WHAT IS THE PHONE FORMAT? There is nothing ANYWHERE to define the format. At least the error should either show the correct format or provide a link to the help file I enter a valid phone number in as many formats as I can think of and none of them allow me to save the number to CRM Nothing works! No matter what format I enter I keep getting a red error "Please enter a valid Phone" The international format for MY mobile is +61414652366 (or +61 414 652 366) Local format is 0414652366 I call all over

    • Introducing Zoho Sprints 3.0

      Zoho Sprints is consistently evolving in steady increments. The introduction of the latest version, with its enterprise level solutions, brings to you advanced capabilities that propel your agile efforts in the right direction. Here's a quick glimpse

    • Customize Sign-out Button

      Are there some url parameters I can use to make a form button sign-out the user from the app? The sign-out link on the top right is small. Here's and example of the url for the top right sign-out: https://creator.zoho.com/logoutpage.jsp?sharedBy=niskypto&appID=212085000006568003&appLinkName=MYAPP&signOutUrl=niskypto/MYAPP/view-login/SOMEPAGE Note: In my account, the sign-out is set to redirect users to my website. Can I also override this with some url parameters? John M. Whitney

    • Please can the open tasks be shown in each customer account at the top.

      Hi there This has happened before, where the open tasks are no longer visible at the top of the page for each customer in the CRM. They have gone missing previously and were reinstated when I asked so I think it's just after an update that this feature

    • Tip #65 - Exploring Technician Console: Short Keys - 'Insider Insights'

      Hello Zoho Assist Community! Have you ever been in the middle of a remote support session, trying to pass a key combination onto the remote machine, only to find it's reflecting on the technician's computer. The Short Keys feature in Zoho Assist is here

    • Reading from and writing to Zoho Projects Custom Module with Deluge

      Does anyone know if there is a way to read from and write to the Custom Modules that Zoho now supports. I would love to be able to loop through a set of data and create the entities I need to for this new custom module I'm looking to put together.

    • Disappointment with Zoho Payments

      Dear Gowdhaman, I am writing to inform you that I am removing Zoho Payments from my website. I cannot continue to disappoint my customers due to the lack of UPI support, as has been the case with my experience so far. Please note that the 0.5% transaction

    • Evolução do modelo de ambientes: Dev, Homologação e Produção com pacotes versionados

      Hoje o Zoho CRM já oferece Sandbox, o que é um avanço importante para organizações que trabalham com customizações mais complexas. No entanto, na prática, o modelo atual ainda apresenta limitações significativas quando múltiplas equipes ou consultorias

    • Permissões granulares por usuário além do modelo baseado exclusivamente em perfis

      Atualmente, o modelo de segurança do Zoho CRM é fortemente baseado em perfis. Embora funcional, esse modelo apresenta limitações quando equipes possuem variações individuais de acesso dentro do mesmo grupo operacional. Em cenários reais, é comum que usuários

    • Add or update lookup field values during Blueprint transition

      Hello everyone, During blueprint transition users can add or modify the value of a lookup field. For instance, if the Tickets module includes a lookup field that connects it to records in the Assets module, agents can link the ticket to the correct asset

    • Can you limit SEO penalties by delaying the appearance of a pop-up?

      Google is not keen on pop-ups and has got even less keen on them with the new Core Web Vitals updates. But I like using pop-ups. If you delay the pop-up so it only appears 10 seconds after the page loads, do you avoid Google’s penalties? 

    • Adding bank details to the contact through API

      How to add bank-related information to the contact while creating it using API? The account number needs to be encrypted before sending it through API but not sure how to encrypt and get those values. Please guide me in this.

    • Restrict Payment Methods

      Allow us to restrict certain payment methods specific for each customer.

    • Clone Banking Transaction

      Why is there no option to CLONE a Transaction in the Banking module?? I often clone Expenses (for similar expense transactions each month) so I would also like to clone Income transactions. But there is no option in Banking to clone an existing Income

    • PDF limit

      Hello everyone, We have received an e-mail that we have reached our PDF limit. (see screenshot) However, I cannot find any reference to a PDF limit in our tariff plan (Premium). (see screenshot). What is the maximum number of PDFs that can be generated

    • How can I see content of system generated mails from zBooks?

      System generated mails for offers or invices appear in the mail tab of the designated customer. How can I view the content? It also doesn't appear in zMail sent folder.

    • Credit Card Pre-Authorization with later Capture/Settlement

      We really enjoy the convenience of being able to pay off a customer's invoice using our Auth.Net integration with Zoho Books. Unfortunately, we can only take advantage of this feature with a small percentage of our customers as it leaves a gaping hole

    • Accessibility in Zoho CRM: Not just a feature—a way to empower

      For instructions on setting up these controls, please check this help document: Configuring accessibility controls. Hello everyone, Today (December 3, 2024), on the International Day of Persons with Disabilities, we begin our journey towards a CRM that

    • Automation Series #1: Round Robin vs Direct Assignment in Zoho Desk

      Direct Assignment vs Round Robin: Choosing the right routing method in Zoho Desk This post is part of the "Desk Automation Series," Chapter 1. Through this series, we will help you choose the right automation type in Zoho Desk by comparing commonly confused

    • Invalid tax authority ID.

      How do I correct this ?

    • Pay Pal Paylater button

      I am testing the paypal setup to have my customers pay invoices with paypal and credit cards. But it seems to have two options. Paypal and Pay later. I don't want my customers access to pay later feature. How can I turn that part off.. My other integration

    • Zoho Workshops are coming to the Netherlands - Join us on 14-16 April in Amsterdam!

      Dear Zoho Community Members, After succesful recent editions in the UK and Scandinavia, we’re pleased to invite you to the upcoming Zoho Benelux Workshop 2026, taking place 14–16 April 2026 in Amsterdam. This three-day, in-person event at the Park Plaza

    • How to install Widget in inventory module

      Hi， I am trying to install a app into Sales Order Module related list, however there is no button allow me to do that. May I ask how to install widget to inventory module related list?

    • Sub form auto field population based on parent form

      I have a parent form called "Sites" with a text field called "Site". I have a subform called "Design Comments" (actual form name "Review Comments") with a lookup field name "Sites1" that looks up from the Sites form. I want the Sites1 lookup field to

    • Building Toppings #7 - Using schedules and workflow functions

      Hello Biginners, In our previous forum post, we explored install and uninstall actions and learned how to trigger custom logic the moment a topping is added or removed from an organization. In this post, we'll look at how to automate actions during regular,

    • Next Page