How to Write a Robots.txt File in Yoast SEO

Search engine crawlers are automated programs that scan the web, following links from one page to another, to index the websites’ content for search engines. One of the ways site owners can communicate with these crawlers is by using a robots.txt file. This text file, placed in the root directory of a website, tells crawlers which pages or sections of a site should not be indexed for search results.

In this article, we’ll discuss the step-by-step method for creating a robots.txt file in Yoast SEO, a popular plugin for WordPress. We’ll also discuss the importance of this file, the default directives set by Yoast SEO, and best practices for editing and using the file.

What is a Robots.txt File?

A robots.txt file is a simple text file that is placed in the root directory of a website. It is used to communicate with web crawlers, also known as “bots” or “spiders,” to tell them which pages or sections of a site should not be indexed. This file is not mandatory, but having one in place is a good practice.

This file can be used for various reasons, such as blocking access to sensitive files on your server or preventing duplicate or non-public pages from being indexed. It can also be used to indicate the location of your XML sitemap (sitemap_index.xml) to the crawlers.

Why is a Robots.txt Important?

A robots.txt file is an important tool for site owners to have control over how search engines interact with their websites. Without it, search engine crawlers would index all pages on your site, including those that you may not want to be publicly available.

For example, you may have a development or staging site that you want to avoid being indexed or pages that are password protected or under construction. Using the robots.txt file, you can block access to these pages and prevent them from appearing in search engine results.

A robots.txt file can also be used to manage the crawl rate, the speed at which search engines crawl your site. By setting a crawl-delay directive, you can slow down the rate at which crawlers access your site, which can help to conserve server resources and prevent your site from being overwhelmed.

The Yoast SEO Default Directives

When you install the Yoast SEO plugin on your WordPress site, it generates a robots.txt file with some default directives. These directives include:

  • User-agent: * (applies to all crawlers)
  • Disallow: /wp-admin/
  • Disallow: /wp-includes/
  • Disallow: /wp-content/plugins/
  • Disallow: /wp-content/cache/
  • Disallow: /wp-content/themes/
  • Disallow: /trackback/
  • Disallow: /feed/
  • Disallow: /comments/
  • Disallow: /category/
  • Disallow: */trackback/
  • Disallow: */feed/
  • Disallow: */comments/
  • Disallow: /?
  • Disallow: /*?
  • Sitemap:

These directives block access to the wp-admin, wp-includes, and wp-content folders, as well as trackbacks, comments, and feed pages. Additionally, it disallows pages with query strings and category archives and includes a directive to the XML sitemap.

How to Create a Robots.txt File in Yoast SEO

Creating a robots.txt file in Yoast SEO is a simple process. Follow these steps:

  1. Log in to your WordPress website
  2. Click on ‘Yoast SEO’ in the admin menu
  3. Click on ‘Tools’
  4. Click on ‘File Editor’
  5. Click the Create robots.txt file button
  6. View (or edit) the file generated by Yoast SEO
  7. Save changes to robots.txt

It’s important to note that while Yoast SEO generates a robots.txt file with default directives, you can edit the file to suit your specific needs. This can include adding directives for specific user agents, such as “User-agent: Bingbot”, or blocking access to specific pages with the “Disallow” directive.

Robots.txt Best Practices

When editing your robots.txt file, it’s important to follow best practices to ensure that it is effective and does not cause any issues for your website.

  1. Block duplicate and non-public pages. Use the robots.txt file to block access to pages that are either duplicates or not meant for public consumption, such as development or staging sites.
  2. Use the correct syntax. The robots.txt file uses a specific syntax that must be followed. You must use the correct format for the “User-agent” and “Disallow” directives.
  3. Use the “Noindex” directive. If you don’t want a page to be indexed, use the “Noindex” meta tag rather than blocking access with the robots.txt file.
  4. Utilize the “Crawl-Delay” directive. If your site is experiencing high traffic or server strain, use the “Crawl-delay” directive to slow down the rate at which crawlers access your site.
  5. Use the “Allow” directive. If you want to allow access to a specific page or folder that is blocked by a more general disallow directive, you can use the “Allow” directive to do so.
  6. Keep it up-to-date. Regularly review and update your robots.txt file to ensure that it is still relevant and effective.
write robots.txt file

Types of Robots.txt Directives

The User-Agent Directive

The “User-agent” directive is used to specify which search engine crawler the directive applies to. The default value is “*”, which applies to all crawlers. You can also specify specific crawlers, such as “User-agent: Googlebot” to only apply the directive to the Googlebot crawler.

The Disallow Directive

The “Disallow” directive blocks access to specific pages or folders on your site. For example, “Disallow: /wp-admin/” would block access to the wp-admin folder.

The Allow Directive

The “Allow” directive overrides a more general “Disallow” directive and allows access to a specific page or folder. This directive should be used in conjunction with the “Disallow” directive.

Noindex Directive

The “Noindex” directive, also called the “meta robots” tag, indicates to search engines that a page should not be indexed. This is different from the “Disallow” directive in robots.txt, which altogether blocks access to the page.

Crawl-Delay Directive

The “Crawl-delay” directive is used to specify the number of seconds that a search engine should wait between requests to your site. This can be useful for managing server resources and preventing your site from being overwhelmed by high traffic or frequent crawling. It’s important to note that not all search engines honor this directive, so it should be used in conjunction with other methods to manage crawl rates.

Robots.txt: Final Thoughts

A robots.txt file is an essential tool for site owners to have control over how search engines interact with their websites. The Yoast SEO plugin makes it easy to create and edit this file and provides default directives to block access to sensitive files and sections of your site.

When editing your robots.txt file, you are following best practices, and using the correct syntax is important. Additionally, use the “Noindex” directive for pages that should not be indexed, and utilize the “Crawl-Delay” directive to manage the crawl rate.

Top SEO FAQs and Questions

Welcome to the Digital Results guide to the top SEO FAQs and questions. We will provide you with valuable insights and tips to optimize your website and boost your online visibility. Whether you’re a beginner or an experienced marketer, these guides will help you understand SEO and address some of the most common questions and concerns.

Search engine optimization, or SEO marketing, is improving the visibility and ranking of a website in search engine results pages through various techniques and strategies. SEO marketing can include keyword research, creating high-quality content, and optimizing website design and structure to improve user experience. It aims to increase website traffic and visibility by making it easier for search engines to understand and recommend the website to users.

SEO is important to businesses because it helps to increase visibility and drive traffic to a website. By ranking higher in search engine results pages, a website is more likely to be seen by potential customers. SEO helps to establish credibility and trust by making a website easy to navigate and providing high-quality content. It can lead to more conversions, sales, and, ultimately, more revenue for a business. SEO is an ongoing process that helps keep a business’s website relevant and competitive in the ever-changing digital landscape.

SEO is still worth it for small businesses because it can help to level the playing field with larger competitors by increasing visibility and driving traffic to their websites. SEO can also help small businesses establish credibility and trust with potential customers. Overall, SEO can be a cost-effective way for small businesses to reach their target market and compete in the online marketplace.

Some of the best SEO Chrome extensions include SEOquake, SEO Minion, and Google Analytics. These can make your life easier by providing you with essential SEO data within the Google Chrome user interface. These tools are widely used by SEO experts and can effectively improve your website’s SEO.

Identifying keywords for SEO is an essential step in optimizing a website for search engines. Keywords are the phrases or terms people use to search for products or services on search engines. Including these keywords on a website makes it easier for search engines to understand what the website is about and match it to relevant search queries. There are several different methods for identifying keywords, including using keyword research tools, analyzing competitors’ keywords, and even Google SERPs.

On-page SEO refers to optimizing elements within a website that can affect its ranking on search engine results pages. It includes title tags, meta descriptions, header tags, and keyword usage throughout the content. On-page SEO also includes ensuring that the website is well-structured, easy to navigate, and has a good user experience. By optimizing these elements, a website can improve its visibility and ranking in search results, making it more likely to be seen by potential customers. Proper use of on-page SEO can also increase the chances of getting better results with less effort than off-page SEO.

If you want your website to be seen by potential customers, writing SEO content is a must. Not only does it improve your visibility and ranking in search results, but it also provides value to your audience. By incorporating relevant keywords, structuring your text with header tags, and including meta descriptions and title tags, you increase the chances of your content being found by search engines. In addition, by making sure your content is easy to read and engaging, you’ll keep visitors on your website longer, which can lead to more conversions. Writing SEO content may take a bit more effort, but the long-term benefits are well worth it. Take advantage of potential customers; make sure your website’s content is SEO optimized.

YouTube SEO is optimizing your YouTube videos to make them more easily discoverable on YouTube and other search engines. By using relevant keywords, creating a compelling title and description, and including closed captions, you increase your chances of potential viewers’ finding your videos. Investing in YouTube SEO can reach a wider audience, boost views and watch time, and ultimately grow your channel. Take advantage of the opportunity to reach millions of people on the second-largest search engine.

When it comes to SEO, WordPress and Shopify have their strengths and weaknesses. One offers more flexibility and control over the structure and design of the website, allowing for better optimization of content, images, and videos. The other has more out-of-the-box SEO features. To make an informed decision, it’s essential to read more about the pros and cons of each platform and see which one aligns with your business needs.

Ready to Grow Your Search Engine Results?

Let Digital Results assist you in your SEO strategy and help
deliver the search engine results you need.