Rank Forge builds robots.txt from the rankforge_robots_txt option. It answers /robots.txt itself on template_redirect and also hooks the WordPress robots_txt filter. When the settings are saved it writes the robots.txt it builds to a robots.txt file in the site root, where WordPress can write to it (the file starts with # Managed by Rank Forge). A web server that finds a robots.txt file sends it directly, without WordPress. FREE feature.
How It Works
- Answers
/robots.txtitself ontemplate_redirect(priority 0) and also hooks therobots_txtWordPress filter at priority 10; both give the same text - Uses the text of the
rankforge_robots_txtoption if set; otherwise the default rules of Rank Forge (User-agent: *,Disallow: /wp-admin/,Allow: /wp-admin/admin-ajax.php). When the site discourages search engines (Settings > Reading), the text isUser-agent: *withDisallow: / - Adds a
Sitemap:line pointing to your sitemap index when the text has none - Writes the same text to
robots.txtin the site root when the option is saved, where WordPress can write to it
Customization
// Set custom robots.txt content
update_option( 'rankforge_robots_txt', "User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /" );
// Remove custom robots.txt (revert to default + sitemap)
delete_option( 'rankforge_robots_txt' );Adding rules from code
A robots_txt filter of your own does not change what the site sends. Rank Forge answers /robots.txt on template_redirect, before WordPress would run that filter, and it builds the file it writes on save without the filter. When a robots.txt file is in the site root, the web server sends the file and WordPress does not run at all.
To add rules from code, change the rankforge_robots_txt option as in the example above. Saving the option also rewrites the file that Rank Forge manages, where WordPress can write to it. To keep every crawler out of a staging site, turn on the WordPress setting that discourages search engines (Settings > Reading): Rank Forge then serves User-agent: * with Disallow: /.
—