Rank Forge provides a virtual robots.txt via the WordPress robots_txt filter. FREE feature.
How It Works
- Hooks into the
robots_txtWordPress filter at priority 10 - Uses custom content from
rankforge_robots_txtoption if set; otherwise uses WordPress defaults - Automatically appends
Sitemap:directive pointing to your sitemap index
Customization
php
// Set custom robots.txt content
update_option( 'rankforge_robots_txt', "User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /" );
// Remove custom robots.txt (revert to default + sitemap)
delete_option( 'rankforge_robots_txt' );Extending via Filter
php
// Block AI training crawlers
add_filter( 'robots_txt', function ( $output, $public ) {
$output .= "nUser-agent: GPTBotnDisallow: /n";
$output .= "nUser-agent: ChatGPT-UsernDisallow: /n";
$output .= "nUser-agent: Google-ExtendednDisallow: /n";
return $output;
}, 20, 2 );php
// Block everything on staging
add_filter( 'robots_txt', function ( $output, $public ) {
if ( defined( 'WP_ENVIRONMENT_TYPE' ) && WP_ENVIRONMENT_TYPE === 'staging' ) {
return "User-agent: *nDisallow: /n";
}
return $output;
}, 20, 2 );—