WordPress robots.txt Configuration

1. The Ultimate WordPress robots.txt Template

By default, WordPress generates a generic virtual robots file. For maximum SEO efficiency and crawl protection, you should replace it with this highly optimized production template:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Allow: /wp-includes/js/
Allow: /wp-includes/css/
Disallow: /search/
Disallow: *?s=
Disallow: */feed/
Disallow: */trackback/

Sitemap: https://webkernelai.com/sitemap.xml

2. Critical Directives Explained

Let's break down why this configuration is essential for your WordPress installation:

• Allow Admin-Ajax: Even though /wp-admin/ is disallowed, we explicitly Allow: /wp-admin/admin-ajax.php. WordPress plugins and themes require this endpoint to load dynamic front-end assets and interactive functions.
• Allow CSS and JS: Disallowing /wp-includes/ completely is a legacy mistake. Modern crawler bots render pages just like web browsers. Googlebot needs access to critical Javascript/CSS assets located inside /wp-includes/js/ and /wp-includes/css/to evaluate core web vitals and mobile responsiveness.
• Block Feeds and Search: Feed URLs (/feed/) and internal search strings (?s=) add no indexation value and pollute crawler queues. Blocking them prevents severe indexing waste.

3. Aligning robots.txt with your XML Sitemap

A common Search Console warning is "Submitted URL blocked by robots.txt". This happens when you instruct Google to ignore a directory in robots.txt, but subsequently submit URLs from that same directory inside your XML sitemap. Ensure your sitemap generates automatically and dynamically to exclude any path disallowed in your crawl policies.