WordPress Robots.txt Editor Without FTP
Rank Forge lets you edit your WordPress site’s robots.txt from the admin, in Rank Forge > Settings, without an FTP client. The text you save is kept in the WordPress database. Rank Forge builds the robots.txt of your site from it, and WordPress answers /robots.txt with that. When you save, Rank Forge also writes the robots.txt it builds to a robots.txt file in the site root, where WordPress can write to it.
What robots.txt does
The robots.txt file sits at the root of your site (example.com/robots.txt) and tells search engine crawlers which parts of your site they’re allowed to crawl.
A correct robots.txt blocks crawlers from wasting your crawl budget on pages that shouldn’t be indexed (admin pages, search result pages, feed URLs, parameter-heavy URLs). A wrong one accidentally blocks pages you do want indexed — which can keep those pages out of Google.
How Rank Forge serves robots.txt
Edit it in the admin
You edit the text in the Robots.txt card of Rank Forge > Settings. When you save, Rank Forge writes the robots.txt it builds from your text to robots.txt in the site root through the WordPress file API. It does this when no robots.txt file is there yet, or when the file is one that Rank Forge wrote earlier (it starts with the line # Managed by Rank Forge). When the folder of your WordPress files is not the folder your web server serves, it writes the file in both folders.
Rank Forge writes where WordPress can write to the site root directly. On a host that would ask for FTP details, Rank Forge writes nothing, and the settings screen tells you when a file that was already there could not be updated.
A file that Rank Forge did not write
If a robots.txt that Rank Forge did not write is in the site root (some hosts put one there), the web server sends that file and the request never reaches WordPress. Rank Forge leaves such a file alone. The Robots.txt card turns read-only and says why. Delete the file with FTP or your hosting control panel, and Rank Forge serves the text from your settings.
What is kept in the database
The text is stored in the rankforge_robots_txt option. If the file in the site root is removed, WordPress builds /robots.txt again from that option. After a site is cloned, a file that Rank Forge wrote earlier is brought up to date where WordPress can write to it, so that its Sitemap line points to the address of the new site.
The Sitemap line
When your text has no Sitemap: line, Rank Forge adds Sitemap: https://example.com/sitemap_index.xml, the address of the sitemap index of your site. If your text has a Sitemap line that points to a sitemap_index.xml, Rank Forge rewrites it to the address of this site. Other Sitemap lines that you wrote stay as they are.
What the Robots.txt card checks
The card also looks at what your site really sends at /robots.txt. When that is different from your settings, for example because a CDN, a proxy or the hosting control panel answers instead of WordPress, the field turns read-only, a “server override detected” label appears, and a panel shows what visitors get.
The Robots.txt card
The Robots.txt card is in Rank Forge > Settings, on the Search Appearance tab. It shows:
- The text of your robots.txt in a plain text field
- A link to your live robots.txt and a link to Google’s robots.txt tester
- A read-only label when a file that Rank Forge did not write is in the site root, and a “server override detected” label when something else answers at
/robots.txt
The card has no rule checker. You save with the save button of the Settings screen. Saving clears the WordPress object cache, and it asks LiteSpeed Cache to purge the robots.txt address and WP Rocket to clear its cache, when those plugins are active.
Default rules
With an empty field, Rank Forge serves:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlThis asks crawlers to stay out of the admin area and still allows admin-ajax.php, which some themes and plugins call from the front end. If the WordPress setting that discourages search engines is on (Settings > Reading), the file is User-agent: * with Disallow: /, plus the Sitemap line, whatever you saved.
Common robots.txt mistakes
Rank Forge saves the text you write and does not check the rules in it, so look for these before you save.
Blocking uploads
A classic mistake: Disallow: /wp-content/uploads/. This keeps Google from crawling your images, so they cannot show up in image search.
Blocking the whole site
Disallow: / blocks everything.
Missing sitemap
If your text has no Sitemap: line, Rank Forge adds one for you.
Blocking CSS and JavaScript
Google needs to render pages fully to understand them. Blocking /wp-content/themes/ or /wp-includes/js/ breaks rendering.
Case sensitivity mistakes
Paths in robots.txt are case-sensitive. Disallow: /Admin/ does not block /admin/.
Custom rules for specific crawlers
Block or allow specific crawlers by name:
User-agent: Googlebot
Disallow: /example-page/
User-agent: Bingbot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /Each company documents its own crawler names and what they do. A crawler is matched by its name in a User-agent: line, as in the example above. As of October 2026, Anthropic’s support site lists ClaudeBot, Claude-User and Claude-SearchBot. Read the documentation of a crawler before you block it.
Checking your robots.txt
After saving changes, test with:
- The robots.txt report in Google Search Consoleshows the robots.txt files Google found, their fetch status and any parsing issues (as of October 2026, per Google’s help page)
- URL Inspection in Google Search Consoletests whether Google can reach a URL
curl https://example.com/robots.txtconfirm the file is being served correctly- The View robots.txt link in the Robots.txt card: opens your live robots.txt
Ready to control your robots.txt?
The Robots.txt Editor is included in every version of Rank Forge, including the free one.