TalkGenAI Bot. Crawler Info & Opt-Out

What is TalkGenAI-Bot?

TalkGenAI-Bot is a lightweight web crawler operated by TalkGenAI. It is used by our customers, businesses and content creators, to monitor publicly available article titles and URLs on domains they choose to track for competitive research.

The crawler only accesses public RSS feeds, XML sitemaps, and article pages. It reads titles and URLs only, it does not store or reproduce full article text.

How to opt out

To stop TalkGenAI-Bot from crawling your site, add the following lines to your robots.txt file:

User-agent: TalkGenAI-Bot
Disallow: /

The crawler re-checks robots.txt at least once every 24 hours, so any domain that adds a disallow rule will stop being crawled, typically within a day.

Crawler behaviour

Honors robots.txt, re-checking it at least once every 24 hours
Reads only public RSS feeds, XML sitemaps, and article metadata (title, URL)
Does not store full article content, images, or personal data
Crawls each domain on a low-frequency schedule (typically once per day; results are cached and shared so the same site is not re-fetched more than once every couple of hours)
Stops requesting a page on any non-success HTTP response (e.g. 429 or 503) rather than retrying

Contact us

If you have questions about TalkGenAI-Bot, or if your robots.txt opt-out is not being honoured, please contact us at [email protected] and we will investigate promptly.