How to Stop Search Engines from Crawling a WordPress Website

Prevent search engines from crawling a WordPress website

Whenever a particular topic is being searched on search engines, the search engines check various contents on different websites to present the most relevant ones to the viewers.

This is the reason why websites take special attention to ensure that their websites stay updated and found easily on search engines.

The various functions that search engines perform to find relevant contents are crawling, indexing, and ranking.

Crawling in the process where software such as web crawlers or spiders is used to identify the most recent content of a website.

However, as they often do not check whether updates are being made on existing pages or whether any new page is created, they might avoid visiting a website page for a month or two.

Indexing is the process by which search engines process and store information of a website they consider to be of relevance on a huge database.

This information can be obtained from the index when a string of keywords is entered.

The ranking is the method through which search engines present results in order of relevance when a topic is being searched.

By locating a website, search engines help in improving the organic traffic to a particular website.

Therefore, one might wonder why should one opt for search engines to stop crawling a WordPress website.

The reason can be various and they are discussed below.

Staging site

When the website is still ‘under working’, site owners will want to remove their website from the search engine results page or SERPs till their website is ready for their audience.

Private Blogs

Many people use WordPress to build up private blogs. Since they are private, they will prefer to prevent their websites from being crawled or indexed on search engines.

Intranet

WordPress is often used for intranet or project management.

To prevent the internal documents from getting accessed by the public, users might want to prevent it from crawling.

Many people have this conception that if links are not added to the website, then the search engines cannot locate them.

However, this is not true as there are various ways through which search engines can locate a website.

Therefore, it is important to take necessary actions to prevent search engines from crawling a website.

Preventing Search Engines from indexing WordPress Website

Search engines can stop crawling a WordPress site when the users take the necessary actions involved in it.

Following methods will prevent search engines from indexing or crawling a website on WordPress. Let us discuss them.

Method 1: By making use of the inbuilt feature of a WordPress website

In this method, site owners can use the inbuilt feature of a WordPress site to prevent the search engines from crawling or indexing a website.

The steps involved in this process are as follows:

Step 1: Visit the admin area of the WordPress site and click on ‘Settings’. From the drop-down list that appears below ‘Settings’, click on ‘Reading’.

Step 2: An option named ‘Search Engine Visibility’ will appear. Next, to it, click on the box that mentions ‘Discourage search engines from indexing this site’. This will cause WordPress to edit the robots.txt file and apply the rules that will block the search engines from indexing or crawling the website.

Step 3: Click on the ‘save’ button to confirm the changes.

Method 2: Editing the robots.txt file

If the users opt for preventing indexing or crawling through the manual process, they can do so by editing the robots.txt file.

This file is located at the root directory of a website and this is a powerful method that will result in search engines stop crawling a WordPress site.

The steps involved in this method are discussed below.

Step 1: Use the ‘File Manager’ or ‘FTP client’ to access the website files.

Step 2: Find the robots.txt file. Generally, it could be found in the same folder as WordPress which is usually ‘public_html’ folder.

However, if the file couldn’t be located, one can create a new blank file.

Step 3: Enter the following syntax

User-agent:*
Disallow: /

This syntax instructs the web crawlers to not index the website pages.

Though, most of the search engines respect this instruction, however, some random images or pages in the website might get indexed.

Password Protecting the WordPress Website

Users can protect their website from search engines stop crawling a WordPress site, by providing it password protection. This can be done by following methods.

Method 1: Password protecting the website using the hosting Control Panel

In this method, websites can be protected with a password through accessing Control Panel or cPanel.

For Hostinger client

The following steps will help in password protecting the website

Step 1: Visit Hostinger control panel and click on the ‘Password Protect Directories’ icon

Step 2: In this step select the directory one wants to protect, which the place where WordPress is installed.

Step 3: After selecting the directory, enter the username and password and click on the ‘protect’ button.

Using cPanel

The following steps are required to carried out while using cPanel

Step 1: Login to the cPanel account and click on ‘Directory Privacy’

Step 2: Select the folder where WordPress is installed. Generally, it is ‘public_html’

Step 3: Click on ‘Password Protect this Directory’ and enter the name of the directory that one wants to protect. Click on the ‘Save’ button and fill-up the form of ‘Create User’ that can access the protected directory. Click on the ‘save’ button to password protect the WordPress website.

Method 2: Password Protect WordPress by using Plugin

Sometimes users might not have access to cPanel if they are using a managed WordPress hosting solution.

In that case, they might use plugins to password-protect their website.

This method will cause search engines to stop crawling a WordPress site.

Some of the popular plugins used for this purpose are Password Protected plugin, WordFence and others.

However, before selecting an appropriate plugin, users must ensure that they are recently updated.

Once they are being installed, they need to go to the setting and set a new password for the site.

Thus, if the site owners want to protect their websites from being indexed or crawled by search engines, they can use the above-mentioned methods.

BTW if you are searching for steps to improve search engine presence of your WordPress website?. You should check the link shared.