How to add robots.txt file in blogger website - Peak Fiction

How to Set Up Custom Txt Robots in the Latest Blogger

In this guide, we'll explore the essentials of custom robots.txt and walk you through the steps to set it up seamlessly on your Blogger website. Additionally, we'll cover adding a sitemap to enhance your site's visibility.

Robots Custom Txt in Blogger

What is custom robot.txt?

Custom robots.txt is a file that site owners use to instruct web robots, such as spiders or crawlers, on which URLs, directories, and pages of their site should be crawled. This is crucial for controlling what content gets indexed by search engines like Google and Bing.

When a web robot visits your site, it checks the custom robots.txt file to determine which pages or directories are allowed to be crawled. This is particularly useful when you want to prevent robots from crawling unnecessary or non-public content.

Custom Robot.txt Code

Here's the custom robots.txt code for your Blogger blog. Copy and paste this code into the Custom Robots.txt section, replacing the example sitemap with your blog's domain name.

User-agent: *
Disallow: /search
Disallow: /category/
Disallow: /tag/
Allow: /
Sitemap: https://www.peakfiction.top/sitemap.xml

Explanation of the code:

• "User-agent: *" applies to all robots.
• "User-agent: Mediapartners-Google & Disallow" allows Google AdSense ads to show on any page.
• "Disallow: /search" ignores URLs with the keyword /search/.
• "Allow: /" permits web robots to crawl any page/URL except those explicitly disallowed.

How to Set Up Custom Txt Robots in Blogger

1. Sign in to your Blogger account.
2. Navigate to Settings in the Sidebar.
3. Scroll down to the Crawler and Indexing section.
4. Enable "Enable custom robots.txt."
5. Click "Custom Robots.txt."
6. Copy and paste the custom robots.txt code here.
7. Click "Save."

Testing Your Custom Robots.txt

To ensure everything is set up correctly, add /robots.txt to your blog URL and press enter. For example:

https://www.abc.com/robots.txt

You should see the same code you added to your custom robots.txt file.

Conclusion

Now you've successfully set up custom robots.txt on your Blogger site. If you found this guide helpful, share it with your friends. Feel free to leave your feedback in the comments section below. For more advanced robot.txt options in the future, refer to our article on optimizing SEO using robots.txt on Blogspot.