How To Add & Test Custom Robots Txt On Blogger SEO

Are you looking for a perfect custom robots.txt for your Blogger blog? You are in the right place. In this tutorial we will guide you step-by-step to how to add and test Custom Robots Txt on Blogger.

 

Following topics will be covered:

• What Is Custom Robots.txt and Its Importance.
• How To Add a Custom Robots.txt In Legacy(Old) Blogger
• How To Add  a Custom Robots.txt In New Blogger.  
• How To Test The Custom Robots.txt.

What is Custom Robots.txt?

Blog and website owners use the robots.txt file to give information about their site to web robots. Web Robots are also known as Spiders or Crawlers, are programs that traverse the web automatically. Search engines such as Google use them to index the web content.

It works like, for instance, if a robot visits your website it will first go to custom robots.txt and will check which pages, type of pages , folders or URLs are allowed to be traversed or which are not because sometimes we don't want robots to traverse or crawl unnecessary pages or non-public folders and directories like admin side URLs or directories. 

How To Add A Custom Robots Txt In Legacy Blogger?

• Go to Blogger dashboard.
• Click Settings.
• Click Search preferences.
• Under Crawlers and indexing section and click Edit in front of Custom robots.txt ? label.

• Select Yes, to enable custom robots.txt.
• Click Save changes.

• Copy & paste the below code as it is: replace meersworld.net with your domain name.
• The "User-agent: *" means this section applies to all robots.
• Disallow: /search means ignore all those URLs having keyword /search/ like https://www.meersworld.net /search/ label/Blogger
• The allow directive is used to override disallow directives in the same robots.txt file. Allow:/ means your home page is allowed to be crawled or traversed.
• Sitemap tells the range of posts to be indexed on search engines. Here we have added 7 lines of sitemap which define the range from 1 to 3000 posts. If your posts exceed 3000 you can add another line in the same manner. It will be like, start-index= 3501&max-results=500

------------------------------------------------------------

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=501&max-results=500
Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=1001&max-results=500
Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=1501&max-results=500
Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=2001&max-results=500
Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=2501&max-results=500
Sitemap: https://www.meersworld.net/atom.xml?redirect=false&start-index=3001&max-results=500

---------------------------------------------------------------

• Click Save changes.

How To Add A Custom Robots Txt In New Blogger?

• Sign in to Blogger.
• Click Settings.
• Scroll down to "Crawlers and Indexing Section".
• Enable the "custom robots.txt" by turning the slider towards right.
• Once you enable the "custom robots.txt" the "Custom robots.txt" will get active.
• Click "Custom robots.txt".

• Now Paste the above code in the pop-up box.
• Click Save.

How To Test Robots Txt?

• Now it is the testing phase of you robots.txt file.
• Go to robots.txt tester on Google Search Console.
• Click Add property now, means your blog URL.
• It will ask you to enter your blog URL.
• Click Continue.

• It will check for errors and will display your robots.txt.

• You can open your live robot.txt as well.
• Enter /robots.txt at the end of your blog URL like https://www.meersworld.net /robots.txt

 

• If you like this post then don't forget to share with your friends. Share your feedback in the comments section below.