Why Your Content Isn't Showing Up in AI Search

Someone asks ChatGPT a question your article answers better than anything else on the internet. It cites three other pages. You check Google and you’re position four for the same words. You check Perplexity and you’re nowhere.

So you search for the fix, and every result gives you the same three-item checklist: publish an llms.txt file, add schema markup, unblock the AI crawlers. One of those three is worth doing. Two of them have now been tested at scale, and the results don’t support the advice.

Here’s what actually makes a page invisible to AI search, in the order worth checking, and where the popular fixes fall apart.

First, make sure you’re actually invisible

Most “I’m not showing up in AI search” reports are a sample size problem.

LLM answers are non-deterministic. Ask the same question twice in the same session and you can get different citations, different phrasing, sometimes a different conclusion. Add memory, location, and account-level personalisation, and one test tells you almost nothing. I’ve watched a page appear in four of ten runs of an identical prompt.

Before you change anything, run the prompt eight to ten times in fresh sessions, on each engine you care about, and log which domains get cited. Two patterns matter. If you appear a few times out of ten, you’re in the candidate pool and you have a strength problem. If you appear zero times out of thirty across three engines while a competitor appears in most runs, something structural is wrong and the rest of this post applies.

Also be precise about which surface you mean. Google AI Overviews, Google AI Mode, ChatGPT, Perplexity, and Copilot select sources differently and from different indexes. “AI search” is not one system, and a fix for one is often irrelevant to another.

The thing that changed: ranking stopped being the ticket

For most of 2025, the honest answer to “how do I get cited” was “rank in the top ten.” Ahrefs analysed 1.9 million citations from a million AI Overviews in July 2025 and found 76.1% of cited pages ranked in the top 10 for the query that triggered the Overview.

They re-ran it. In March 2026, across 4 million AI Overview URLs from 863,000 keyword SERPs, that number was 37.9%. Ahrefs attributes the drop to query fan-out: rather than summarising the results for the query you typed, the system decomposes it into sub-queries, runs those, and pulls sources from the SERPs it generated internally.

That single number reframes the whole problem. “I rank number one and I’m not cited” used to be a symptom of something broken on your page. Now it’s the normal case, because you’re being evaluated against a set of SERPs you never saw. The queries that determine your citation are the sub-questions the engine invented, not the head term in your rank tracker. Which is why the People Also Ask cluster is more useful now than it was two years ago. It’s the closest publicly visible proxy for how a system decomposes a topic.

The blockers that genuinely make you invisible

Your content only exists after JavaScript runs

This is the highest-yield check and almost nobody runs it.

Vercel measured real crawler traffic across its network for The Rise of the AI Crawler and found that none of the major AI crawlers render JavaScript. ClaudeBot fetched JavaScript files in 23.84% of its requests and GPTBot in 11.50%, but neither executed them. They read the raw HTML response and nothing else. Googlebot does render, which is why a client-rendered page can rank perfectly well in classic search and be a blank shell to ChatGPT.

The test takes thirty seconds:

curl -A "GPTBot" -s https://yoursite.com/your-page | grep "a distinctive sentence from your article"

If your body copy isn’t in that response, no AI crawler has ever read it. This bites hardest on React and Vue marketing sites, and on individual components that get injected client-side even when the rest of the page is server-rendered. Review widgets, FAQ accordions, tabbed specs, and comparison tables are the usual offenders, and those are exactly the passages an answer engine would want to lift.

One caveat I want to be straight about: Vercel’s measurement is from late 2024, and I couldn’t find a comparably rigorous public re-measurement since. No vendor documents JavaScript execution for these bots either, so it remains the best available evidence rather than a settled law.

You’re blocking the crawlers, possibly by accident

Two things happened that put blocks on sites whose owners never chose them.

The first was the 2023 scraping panic, when a lot of teams pasted Disallow rules for GPTBot into robots.txt and then forgot. The second is infrastructure. Cloudflare changed its default on 1 July 2025 so that new domains on its network block AI crawlers unless the owner opts in. If your site launched after that date behind Cloudflare and nobody explicitly allowed the bots, you’re blocked and your robots.txt looks clean.

The trap underneath both is that training and retrieval use different user agents. GPTBot is OpenAI’s training crawler. OAI-SearchBot and ChatGPT-User are what fetch pages to answer a live question. Blocking one and expecting the other to behave is a common and completely silent failure. Check each bot by name, in robots.txt, in your WAF rules, and in your CDN’s bot management settings, then confirm against server logs that the retrieval bots are actually getting 200s from you. The log is the truth; the config file is the intention.

The two fixes that get sold hardest and hold up worst

llms.txt

Ahrefs looked at server logs from 137,000 domains in May 2026. Of the sites with a valid llms.txt file, 97% received no requests for it at all during the study period. Of the requests that did arrive, AI retrieval bots accounted for 1.1%. The rest was overwhelmingly SEO tools, validators, and technology profilers checking whether the file existed. Google has said publicly that it doesn’t use llms.txt for AI features.

Publishing one costs an hour and does no harm. It is not why you’re invisible, and the hour is better spent on the curl command above.

Schema markup

The correlation is real and it’s why this advice spread. Across 6 million URLs, Ahrefs found pages cited by AI were nearly three times more likely to carry JSON-LD than uncited pages. If you stop there, “add schema, get cited” looks obvious.

Then they ran it as an experiment. They tracked 1,885 pages that added JSON-LD between August 2025 and March 2026, matched them against 4,000 control pages, and ran a difference-in-differences analysis to strip out platform-wide trends. AI Overview citations moved −4.6%. AI Mode moved +2.4%. ChatGPT moved +2.2%. The two positives were statistically indistinguishable from zero, and the negative was small.

Google’s own documentation says the same thing more plainly: there is no special structured data required to appear in AI Overviews or AI Mode.

Two honest qualifications. Schema still earns rich results in classic search, and that’s reason enough to keep it. And the tested pages already had 100+ AI Overview citations before the change, meaning they were already being read and parsed successfully. Adding schema to a page that no system has ever understood is a different situation, and it wasn’t measured. What the data doesn’t support is treating markup as the lever that turns citations on.

Why AI Overviews don’t appear for some searches at all

This gets tangled up with the visibility question and it’s a separate thing. Google doesn’t generate an Overview for every query. Coverage is thin or absent on many health and finance queries, on navigational and brand searches, on very fresh news, and on low-volume long tail. Presence also moves with core updates, sometimes losing whole query categories overnight.

If you’re tracking a keyword where the box doesn’t render, there’s nothing to be cited in and no problem to fix. Confirm the surface exists for the query before you spend a sprint optimising for it.

What does seem to move it

I’m going to be careful here, because the platforms publish almost nothing about how sources get selected, and anyone claiming a confirmed mechanism is guessing with confidence. What follows is pattern-matching from published studies and from watching our own queries, not documented behaviour.

Be liftable. Answer-first passages, plainly named entities, structure a parser doesn’t have to interpret. This is the same discipline that gets a passage into an Overview, and it’s covered properly in how to show up in Google AI Overviews.

Be corroborated somewhere other than your own site. AI answers are assembled from what the web says about you, not just what you say about yourself. For any comparison or recommendation query, a large share of the sources will be roundups, forums, and third-party comparisons rather than vendor pages. If nothing off your domain says the thing you want the model to say, the model has no reason to say it.

Be worth quoting. If your page restates what the other candidates already state, there’s no reason to select it over them. That’s information gain, and it matters more in a retrieval system than it did in a ranked list, because a generated answer only needs one source per claim.

The order to check things in

1. Run the prompt eight to ten times per engine, in fresh sessions, and log which domains get cited.
2. Fetch your page with a bot user agent and confirm the copy is in the raw HTML.
3. Check robots.txt, WAF rules, and CDN bot settings for each named bot, training and retrieval separately.
4. Check server logs for whether the retrieval bots are getting 200s.
5. Confirm the AI surface even renders for the query you’re tracking.
6. Look at the sub-questions the topic decomposes into, not just the head term.
7. Find whether anything off your site corroborates the claim you want cited.

Housekeeping like llms.txt and markup goes after all of that, not before it.

Building the brief around the sub-questions rather than the head term is the part that’s easiest to get wrong and hardest to retrofit, which is most of what Ranklet does when it reads the live SERP and the question clusters around a keyword. You can try it on one of your own keywords if you want to see the decomposition before you write.

Is being cited even worth chasing?

Worth asking before you reorganise a quarter around it.

Seer Interactive tracked 53 brands across 5.47 million queries and 2.43 billion organic impressions from January 2025 through February 2026. On informational queries, averaged across 2025, organic click-through was 2.07% when the brand was cited in the Overview, 0.94% when it wasn’t, and 3.35% on the same kind of query with no Overview present. Seer are explicit that they can’t claim causation, since stronger brands are also more likely to be cited in the first place.

Read it as damage control rather than upside. Being cited roughly doubles your clicks compared to being ignored, and still leaves you well below what the query was worth before the box existed. That’s a good reason to fix a crawler block this afternoon. It’s a bad reason to spend three weeks on files that 97% of sites never get a single request for.