=== ADG Markdown for AI Crawlers ===
Contributors: andyodua
Tags: ai, llms.txt, markdown, gptbot, crawler
Requires at least: 6.0
Tested up to: 7.1
Requires PHP: 7.4
Stable tag: 1.1.0
License: GPLv2 or later
License URI: https://www.gnu.org/licenses/gpl-2.0.html

Serve AI crawlers clean Markdown of your content on the same URL, and auto-generate a virtual llms.txt / llms-full.txt. No external services.

== Description ==

When a known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others) requests one of your posts or pages, this plugin returns clean **Markdown** of the main content on the **same URL** instead of the full theme HTML — no header, footer, sidebar or scripts. Regular visitors keep seeing your normal site.

It also generates a virtual **llms.txt** and **llms-full.txt** — a sitemap-style index of your content for AI tools — served at `/llms.txt` and `/llms-full.txt` without creating any physical files.

**What it does**

* Detects AI crawlers by User-Agent (the list is editable in the settings).
* Serves clean Markdown on the same URL with `Content-Type: text/markdown`, `Vary: User-Agent` and `X-Robots-Tag: noindex`.
* Works with any public post type — including WooCommerce products — plus a Markdown index for category, tag and custom-taxonomy archives.
* Configurable **YAML front matter** (title, url, date, author, tags, canonical) with per-field toggles.
* Include your own **custom fields / ACF** values in the front matter.
* Adds machine-readable Schema.org JSON-LD (title, dates, author, cover image, publisher).
* Generates virtual `/llms.txt` and `/llms-full.txt`.
* Caches the converted Markdown in post meta for fast responses; the cache is cleared automatically when a post is edited.
* Plays nice with page caches: sets `DONOTCACHEPAGE` (WP Rocket, W3 Total Cache, WP Super Cache, LiteSpeed) so bots are never served a stale HTML cache.
* Exclude content with a per-post "Hide from AI bots" checkbox or by category/tag slug.
* "View Markdown" link in the editor and post list — preview exactly what crawlers receive.

**Privacy and external services**

This plugin does **not** connect to any external service, send any data off your site, or load any remote scripts. All processing happens on your own server.

**Good to know**

* No .htaccess edits and no Composer — a built-in, zero-dependency HTML→Markdown converter is included.
* Does not interfere with SEO plugins, caches, or WooCommerce shop/cart/checkout pages.
* Private, password-protected and unpublished content is never served to bots — access rights are not bypassed.

**Pro version**

A Pro version is available with bot IP verification (anti-spoofing), bot-visit analytics with CSV export, a one-click self-test, and priority support. See the plugin's settings page for details. The free version is fully functional on its own and is not time-limited.

== Installation ==

1. Upload the `adg-markdown-for-ai-crawlers` folder to `/wp-content/plugins/`, or install the plugin through the WordPress "Plugins" screen.
2. Activate the plugin.
3. Go to **Settings → AI Bots & Markdown** to configure it.
4. On Nginx, if `/llms.txt` returns a 404, make sure your config falls through to `index.php` (the standard `try_files $uri $uri/ /index.php?$args;` block covers this). On Apache it works out of the box.

== Frequently Asked Questions ==

= Will private or draft pages be exposed to bots? =
No. Markdown is served only for published, non-password-protected, non-private posts of the allowed types. Access rights are never bypassed.

= Does it send my content to any external server? =
No. There are no outbound requests at all — detection is by User-Agent, and all conversion happens locally.

= Does it work with page builders like Elementor? =
Yes. Content is taken through `the_content`, so shortcodes and builder output are rendered before conversion. Testing on your specific setup is recommended.

= A bot still receives HTML instead of Markdown. Why? =
Most often the request is blocked at the server/CDN level (HTTP 403) or served from a page cache before the plugin runs. Make sure your cache honors `Vary: User-Agent` or excludes bot user-agents.

== Screenshots ==

1. The settings screen: content negotiation, editable AI-bot list, and llms.txt options.
2. The "Hide from AI bots" checkbox on the post editor.

== Changelog ==

= 1.1.0 =
* New: configurable YAML front matter (title, url, date, author, tags, canonical) with per-field toggles.
* New: include custom fields / ACF values in the Markdown front matter.
* New: Markdown index for category, tag and custom-taxonomy archives.
* New: WooCommerce products (and any public post type) can be served as Markdown and listed in llms.txt.
* New: exclude content by category/tag slug, in addition to the per-post checkbox.
* New: "View Markdown" preview link in the post editor and post list.
* Improved: page-cache compatibility — sets DONOTCACHEPAGE for WP Rocket, W3 Total Cache, WP Super Cache and LiteSpeed.
* Fixed: list item text (simple <li> entries) was dropped from the Markdown output.

= 1.0.0 =
* Initial release: content negotiation (Markdown for AI bots on the same URL), Schema.org/JSON-LD front matter, post-level Markdown cache, virtual llms.txt / llms-full.txt, and a "Hide from AI bots" metabox.

== Upgrade Notice ==

= 1.1.0 =
YAML front matter, custom fields/ACF, taxonomy archives, WooCommerce support, term-based exclusions, cache compatibility, and a Markdown preview link.

= 1.0.0 =
Initial release.