=== AI Crawler and SEO Manager by akbansa ===
Contributors: @akbansa
Tags: SEO, Analytics, AI Crawlers, Crawler Logger, LLMs.txt
Requires at least: 5.0
Tested up to: 6.8
Requires PHP: 7.4
Stable tag: 1.0.4
License: GPLv2 or later

== Short Description ==
Advanced AI-powered SEO and analytics for WordPress. Generates llms.txt for AI crawler discovery, logs AI bot visits, and provides comprehensive analytics dashboard.

== Description ==

The AI Crawler and SEO Manager by akbansa plugin generates an `llms.txt` file for crawler discovery and logs visits by AI-based crawlers such as GPTBot, ClaudeBot, and others. It also provides an analytics page in the WordPress admin dashboard to view crawler activity by path, URL, and timestamp.

### Key Features:
- Generates `llms.txt` file for crawler discovery.
- Removes `llms.txt` on plugin uninstall
- Schedules automatic generation of `llms.txt` daily at midnight (frequency configurable: daily, twice daily, hourly)
- Uses atomic file writes for `llms.txt` and `sitemap.xml` for safety
- Integrates with Yoast SEO, Rank Math, or All in One SEO for metadata (falls back to DB if not present)
- Admin panel to select/exclude categories and content types for `llms.txt`
- Logs every AI crawler request from the frontend (user agent, crawl info, page URL, IP, referrer)
- Filters crawlers using `config.json` and stores all details in a dedicated DB table
- Analytics dashboard in admin to view recent crawler activity
- Robust AJAX security and error handling
- Detects and logs visits from AI crawlers based on their User-Agent.
- Stores crawler logs in a database for analysis.
- Supports popular AI-based crawlers (e.g., OpenAI, Anthropic, GPTBot, etc.).
- Option to customize the list of crawlers via a `config.json` file.

== Installation ==

1. Upload the `akbansa-ai-seo-manager` plugin folder to your WordPress installation's `wp-content/plugins/` directory.
2. Activate the plugin through the 'Plugins' menu in WordPress.
3. Ensure the `config.json` file is in the plugin directory. You can update this file to include or exclude specific AI crawlers as needed.

== Usage ==

- After activation, the plugin will automatically start generating the `llms.txt` file and log visits from AI-based crawlers.
- The crawler logs are stored in the database for analysis.
- You can view analytics by navigating to **LLMS Crawlers** in the WordPress admin menu.

== Admin Analytics ==

The admin panel will display a list of the most recent crawler visits, showing the following information:
- Path
- URL
- Crawler Name
- Timestamp

== Changelog ==

= 1.3 =
* Added support for detecting and logging visits from a wider range of AI-based crawlers (e.g., GPTBot, ClaudeBot).
* Introduced `config.json` file for easy crawler list updates.
* Improved logging of crawler visits to a dedicated database table.
* Added admin analytics page for viewing crawler logs.

= 1.2 =
* Initial release.

== Frequently Asked Questions ==

= How can I add more AI crawlers? =
You can add more AI crawlers to the `config.json` file located in the plugin directory. Simply add the User-Agent strings of the crawlers you want to track.

= Can I exclude certain crawlers from being logged? =
Yes, you can remove any crawlers from the `config.json` file to prevent them from being detected and logged.

= How do I view the crawler logs? =
You can view the logs by navigating to the **LLMS Crawlers** section in the WordPress admin menu. This will display the most recent AI crawler visits.

== Support ==

If you encounter any issues, please feel free to reach out via the plugin support page or create an issue on the GitHub repository.

== License ==

This plugin is licensed under the GPLv2 or later.

== New Features ==

1. **Admin Settings Filter**: You can now filter active pages, products, categories, and posts for generating the `llms.txt`.
2. **Sitemap Handling**: If a `sitemap.xml` exists, it will be used to get the list of pages. If not, the database will be used as a fallback.
3. **SEO Plugin Integration**: The plugin integrates with Yoast SEO, All in One SEO, and Rank Math to fetch metadata for the `llms.txt` file.
4. **Frequency Setting**: You can choose the frequency for regenerating the `llms.txt` file (daily, weekly, monthly, hourly).
5. **Traffic Capture and AI Agent Logging**: A front-end script captures traffic, sends it to the backend, and logs AI agents based on a `config.json` file.
6. **Sitemap Update**: The `sitemap.xml` is automatically updated with the absolute path to the `llms.txt` and the updated timestamp.