When someone asks an AI assistant "what's a good tool for X?", the answer isn't drawn from one place. Depending on the product and the setup, the assistant may search the web, read pages on demand, use structured data, or call a site's API through a tool. Each path has its own requirements, and none of them guarantees a mention.
This guide covers the main paths, what's documented about each, and what you can reasonably do as a maker. Platforms change their policies often, so check the linked sources for current details.
1. Web search and on-demand page fetches
Assistants that answer questions about current products often ground their answers in web results. Several publish how this works:
- Google. AI Overviews and AI Mode draw on Google Search. Google says the usual SEO best practices apply, there are no additional technical requirements, and a page has to be indexed and eligible to show with a snippet to appear as a supporting link.
- ChatGPT. OpenAI uses OAI-SearchBot for ChatGPT's search features, separate from GPTBot, which is used for training. Sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.
- Perplexity. PerplexityBot surfaces and links sites in Perplexity's search results, and Perplexity-User fetches pages when a user asks a question.
- Claude. Anthropic lists Claude-SearchBot for search quality and Claude-User for user-initiated fetches, separate from ClaudeBot, which is used for training.
What this means for you: being crawlable and indexable is the baseline. Search crawlers and training crawlers are controlled separately, so you can block training without disappearing from search answers. Check that your robots.txt, CDN, or firewall isn't blocking the search agents you actually want. Make sure the important facts about your product exist as text on the page, not only in images or client-rendered JavaScript.
2. llms.txt
llms.txt is a proposal for a markdown file, usually at /llms.txt, that gives agents a short overview of a site and links to the pages that matter. According to the proposal, it is mostly used at inference time, when an agent needs information while helping a user, and it's especially common in software documentation.
Support varies. Google says you don't need new machine-readable files or AI text files to appear in its AI features. Treat llms.txt as a cheap, useful summary for agents that read it, not as a ranking factor.
What this means for you: if you write one, keep it short and factual. Cover what the product is, who it's for, the key pages, and what it isn't (useful if your name resembles another product's).
3. Structured data (schema.org)
schema.org defines types such as SoftwareApplication and WebApplication for describing software in a page's markup, usually as JSON-LD. Google documents which SoftwareApplication properties it supports for rich results.
Two caveats. First, Google says no special schema.org markup is needed for its AI features. Second, structured data has to be a true representation of the page content, and ratings or reviews that don't come from actual users can lead to a manual action.
What this means for you: markup can help machines parse the basics (name, category, platform, whether it's free). Only mark up what's visible and true on the page.
4. Public APIs and MCP
Agents working inside editors and chat apps can also call tools directly instead of reading pages. The Model Context Protocol (MCP) is an open standard for connecting AI applications to external systems, and MCP's own docs list clients such as Claude, ChatGPT, VS Code and Cursor as supporting it. An API described with OpenAPI gives an agent exact fields, allowed values and error codes.
What this means for you: this matters most if your product is itself a developer tool or service. Then a documented API, or an MCP server, can be the most direct way for an agent to understand and use it. For discovery, it also helps to be listed on sites that expose their own data this way. For example, LaunchSignal describes itself in llms.txt and publishes its approved listings through a JSON API and an MCP server.
5. What other sources say about you
When an assistant answers from search results, it can only cite what pages actually say. If the only description of your product is your own homepage, that's all it has to work with. Accurate descriptions on independent pages, such as docs, articles, community threads and listings, give it more to cross-check.
This also cuts the other way. Google's spam policies cover tactics like keyword stuffing and scaled low-value content, and fabricated reviews are a liability under its structured data rules. Earn mentions honestly, and keep them consistent.
What this means for you: write one plain description of your product and reuse it everywhere. Contradictory descriptions across sites make it harder for any reader, human or machine, to know which one is true.
Maker checklist
- Homepage states in plain text what the product does, who it's for, and whether it's free, paid, or in beta
- Key pages work without JavaScript or a login wall (try fetching them with
curl) - robots.txt, CDN and firewall allow the search agents you want, and training crawlers are set the way you intend
- One consistent one-line description, reused across your site and listings
- Optional: a short, factual
/llms.txt - Optional: schema.org markup that matches the visible page, with no invented ratings
- If you offer an API: public docs or an OpenAPI file with clear error codes; consider an MCP server
- Listed accurately on a few relevant, reputable sites, and kept up to date as the product changes
- Periodically ask a few assistants about your product category and note what they get wrong
Sources
- Google Search Central: AI features and your website
- Google Search Central: Software app structured data and general structured data guidelines
- Google Search Central: Spam policies
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Anthropic: Does Anthropic crawl data from the web?
- The /llms.txt file proposal
- schema.org SoftwareApplication
- Model Context Protocol: introduction
- OpenAPI Initiative
Sources checked October 18, 2026.