Support Log in

19. Robots.txt

Developer Guide
Prefer video? Watch the Rank Forge tutorials 9 short guides covering every feature Watch

Rank Forge builds robots.txt from the rankforge_robots_txt option. It answers /robots.txt itself on template_redirect and also hooks the WordPress robots_txt filter. When the settings are saved it writes the robots.txt it builds to a robots.txt file in the site root, where WordPress can write to it (the file starts with # Managed by Rank Forge). A web server that finds a robots.txt file sends it directly, without WordPress. FREE feature.

How It Works

  • Answers /robots.txt itself on template_redirect (priority 0) and also hooks the robots_txt WordPress filter at priority 10; both give the same text
  • Uses the text of the rankforge_robots_txt option if set; otherwise the default rules of Rank Forge (User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php). When the site discourages search engines (Settings > Reading), the text is User-agent: * with Disallow: /
  • Adds a Sitemap: line pointing to your sitemap index when the text has none
  • Writes the same text to robots.txt in the site root when the option is saved, where WordPress can write to it

Customization

php
// Set custom robots.txt content
update_option( 'rankforge_robots_txt', "User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /" );

// Remove custom robots.txt (revert to default + sitemap)
delete_option( 'rankforge_robots_txt' );

Adding rules from code

A robots_txt filter of your own does not change what the site sends. Rank Forge answers /robots.txt on template_redirect, before WordPress would run that filter, and it builds the file it writes on save without the filter. When a robots.txt file is in the site root, the web server sends the file and WordPress does not run at all.

To add rules from code, change the rankforge_robots_txt option as in the example above. Saving the option also rewrites the file that Rank Forge manages, where WordPress can write to it. To keep every crawler out of a staging site, turn on the WordPress setting that discourages search engines (Settings > Reading): Rank Forge then serves User-agent: * with Disallow: /.

—

Forge AI Assistant Online

Hi! I'm the Rank Forge AI assistant. Ask me anything about the plugin — setup, features, troubleshooting, or development.

Just now
Powered by Forge AI · Browse docs