What is Robots.txt and The best way to Optimize It

by ThinkCode | Sep 11, 2015 | Search Engine Optimization

Basically, robots.txt is a transmission between a website and the web search engine. It traces the guidelines for indexing your site and presents them to the web crawler. So fundamentally it defines that which element of your website Google’s permitted to index and which to be avoided.

It is very important don’t forget however in which some reliable search engines like Google will certainly realize along with respect the directives inside the robots.txt  file, many destructive or perhaps low-quality crawlers may possibly neglect them completely.

Right now, if you’re wondering whether robots.txt is can be an overall must, the solution isn’t any. Your website will be indexed by the major search engines paying little mind to whether you have it or not.

Step by step instructions to create robots.txt for WordPress

Making robots. txt document is actually much simpler than one may believe. You should simply make a content document, name it robots.txt and transfer it into your site’s root index. Including the substance into the document is not difficult. You will find simply 6 instructions to create robots. txt:

User-Agent – decides the particular search engine crawler

Disallow – denies the crawler to index specific files or perhaps directories on the website

Allow – particularly permits crawling an item on the website

Sitemap – indicates the crawler the sitemap

Crawl-delay – specifies any time period of time in between search engines’ demands to your server.

Take note, if you want to arrange the same guidelines for those search engines like yahoo, bing utilizes asterisk (*) after the user-agent command

For example:

User-Agent: *

Allow: /wp-content/uploads/

Disallow: /wp-content/plugins/

Sitemap: https://thinkcode.co.in/sitemap.xml

If you use the robots.txt file properly, it helps search engine away from the content you don’t want to be seen. On the whole, you must remember, that there are no general standards on how to prepare  the ideal robots.txt file and its particular content should be taking into account on the type of website you have.

What happen if we place robots.txt file in sub directory?

by ThinkCode | Aug 25, 2015 | Search Engine Optimization

It will apply to that directory only – not sure if it would get crawled there. It would be best to implement in ur root folder and disallow any sub-directories you don’t want to be crawled. It’s all up to the crawlers to respect your directives then.
Ideally you should have it in the root file but if you have to place it in the sub-directory, then this is what we would recommend.

1. Specify the path in Google Webmasters

2. 301 Redirect the conventional path – domain.com/robots.txt to the new path.
If you have a multi-language website you should put every robots.txt in the first directory for every language sections of the website and not in the different sub-directory.