AI crawler and llms.txt checker
Enter your website. Urlaunch reads its robots.txt and its /llms.txt and shows, for each AI crawler its operator documents, whether your rules let it read your home page. Crawlers that build AI search results or fetch a page for a person are listed apart from crawlers that collect training text.
How it works
- robots.txt is read the way the robots.txt standard (RFC 9309) reads it, for your home
page: a group that names a crawler decides for it, and the group for all crawlers (
*) applies only when none names it. - The crawlers are the ones their operators document, each read from the operator's own page. AI search crawlers build the results AI assistants link to, on-request crawlers fetch a page when a person asks an assistant about it, and training crawlers collect text that models learn from. Blocking a training crawler is your choice, so it is never reported as a problem.
- llms.txt is a proposal (llmstxt.org) for a plain-text file at /llms.txt that points AI assistants at a site's key pages. The check says whether the file exists and follows the proposal's form. Whether an llms.txt changes what assistants answer is unmeasured.
- A file that could not be read is reported as unread, never as missing.
Other tools
- Website grader: site health and measured topic coverage.
- Website keyword analysis: the measured searches each page could rank for.
- Keyword search volume: monthly searches for one topic.
- Landing page checker: whether a page answers its keyword.
- Ad copy checker: each line of ad text counted against its platform's character limit.
- Break-even ROAS calculator: what a sale and a click may cost in ads.
- UTM builder: links tagged for your analytics.