Disallow Google’s Robot from Indexing an FTP Folder

raccoon-mascot

When you create products, such as ebooks, that you wish to sell to your website visitors, it is important to lock those products away. In other words, to make them invisible to all those who did not purchase them.

One of the easiest ways to look up an ebook is via a Google search.

Let’s say that you created an ebook called My-Fantastic-Ebook.pdf and added it to your WordPress installation’s /wp-content/uploads/ folder.

Now, if you do this and nothing else, a simple Google search containing your website’s URL address and the keyword pdf will reveal to anyone the location of your ebook.

robots.txt

To hide the PDF file containing your product from non-paying visitors, you need to make use of the robots exclusion standard or, as it is better known, the robots.txt file.

Create a plain-text file and name it robots.txt. Add the code disallowing search robots from crawling certain areas of your website and then upload the robots.txt file into your hosting server’s root FTP folder.

To make search robots skip a specific folder on your site from crawling, add the following lines to your robots.txt file:

You can list several folders, too:

Important Note

Of course, you need to list those specific folders on your server that search engine robots and crawlers shouldn’t index.
 
The “User-agent: *” line defines that the settings are valid for all search robots.

If you want to disallow search engine robots to crawl your whole WordPress installation, add only this one line to your robots.txt file:

.htaccess

Another way, and an even stronger way, to make certain files hidden from search robots is to include the following lines int your root .htaccess file:

This code tells all robots not to index any files of the doc, pdf, or txt format.

These two simple tricks can save you a lot of headache down the road.

If you plan to sell digital items from your website, definitely use both robots.txt and set a X-Robots-Tag header within your .htaccess file.

Liked this post?

Subscribe to our newsletter to receive early notification of new posts and deals:

Next Post »« Previous Post

Leave a Reply

Your email address will not be published. Required fields are marked *