Why Your Content Isn't Showing Up in AI Search

Someone asks ChatGPT a question your article answers better than anything else on the internet. It cites three other pages, you’re position four in Google for the same words, and Perplexity has never heard of you.

There are four reasons this happens, and they’re worth separating because the fixes have nothing to do with each other. Your content may not be in the HTML the crawler receives, because it only appears after JavaScript runs. Your crawlers may be blocked, often by a CDN default nobody set on purpose. The engine may be answering sub-questions your page isn’t on. Or the page is read fine and simply isn’t worth quoting over the alternatives.

Search for the fix, though, and every result gives you the same three-item checklist instead: publish an llms.txt file, add schema markup, unblock the AI crawlers. One of those three is worth doing. Two have now been tested at scale, and the results don’t support the advice.

First, make sure you’re actually invisible

Most “I’m not showing up in AI search” reports are a sample size problem.

LLM answers are non-deterministic. Ask the same question twice in the same session and you can get different citations, different phrasing, sometimes a different conclusion. Add memory, location, and account-level personalisation, and one test tells you almost nothing. I’ve watched a page appear in four of ten runs of an identical prompt.

Before you change anything, run the prompt eight to ten times in fresh sessions, on each engine you care about, and log which domains get cited. Two patterns matter. If you appear a few times out of ten, you’re in the candidate pool and you have a strength problem. If you appear zero times out of thirty across three engines while a competitor appears in most runs, something structural is wrong and the rest of this post applies.

Also be precise about which surface you mean. Google AI Overviews, Google AI Mode, ChatGPT, Perplexity, and Copilot select sources differently and from different indexes. “AI search” is not one system, and a fix for one is often irrelevant to another.

The thing that changed: ranking stopped being the ticket

For most of 2025, the honest answer to “how do I get cited” was “rank in the top ten.” Ahrefs analysed 1.9 million citations from a million AI Overviews in July 2025 and found 76.1% of cited pages ranked in the top 10 for the query that triggered the Overview.

They re-ran it. In March 2026, across 4 million AI Overview URLs from 863,000 keyword SERPs, that number was 37.9%. Ahrefs attributes the drop to query fan-out: rather than summarising the results for the query you typed, the system decomposes it into sub-queries, runs those, and pulls sources from the SERPs it generated internally.

That single number reframes the whole problem. “I rank number one and I’m not cited” used to be a symptom of something broken on your page. Now it’s the normal case, because you’re being evaluated against a set of SERPs you never saw. The queries that determine your citation are the sub-questions the engine invented, not the head term in your rank tracker. Which is why the People Also Ask cluster is more useful now than it was two years ago. It’s the closest publicly visible proxy for how a system decomposes a topic.

The blockers that genuinely make you invisible

Your content only exists after JavaScript runs

This is the highest-yield check and almost nobody runs it.

Vercel measured real crawler traffic across its network for The Rise of the AI Crawler and found that none of the major AI crawlers render JavaScript. ClaudeBot fetched JavaScript files in 23.84% of its requests and GPTBot in 11.50%, but neither executed them. They read the raw HTML response and nothing else. Googlebot does render, which is why a client-rendered page can rank perfectly well in classic search and be a blank shell to ChatGPT.

The test takes thirty seconds:

curl -A "GPTBot" -s https://yoursite.com/your-page | grep "a distinctive sentence from your article"

If your body copy isn’t in that response, no AI crawler has ever read it. This bites hardest on React and Vue marketing sites, and on individual components that get injected client-side even when the rest of the page is server-rendered. Review widgets, FAQ accordions, tabbed specs, and comparison tables are the usual offenders, and those are exactly the passages an answer engine would want to lift.

One caveat I want to be straight about: Vercel’s measurement is from late 2024, and I couldn’t find a comparably rigorous public re-measurement since. No vendor documents JavaScript execution for these bots either, so it remains the best available evidence rather than a settled law.

You’re blocking the crawlers, possibly by accident

Two things happened that put blocks on sites whose owners never chose them.

The first was the 2023 scraping panic, when a lot of teams pasted Disallow rules for GPTBot into robots.txt and then forgot. The second is infrastructure. Cloudflare changed its default on 1 July 2025 so that new domains on its network block AI crawlers unless the owner opts in. If your site launched after that date behind Cloudflare and nobody explicitly allowed the bots, you’re blocked and your robots.txt looks clean.

The trap underneath both is that training and retrieval use different user agents. GPTBot is OpenAI’s training crawler. OAI-SearchBot and ChatGPT-User are what fetch pages to answer a live question. Blocking one and expecting the other to behave is a common and completely silent failure. Check each bot by name, in robots.txt, in your WAF rules, and in your CDN’s bot management settings, then confirm against server logs that the retrieval bots are actually getting 200s from you. The log is the truth; the config file is the intention.

The two fixes that get sold hardest and hold up worst

Before the studies, it’s worth reading what Google’s own documentation on AI features and your website says, because it disposes of both in one sentence: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.”

That’s the publisher of the largest AI search surface saying the checklist is not the mechanism. The independent data agrees.

llms.txt

Ahrefs looked at server logs from 137,000 domains in May 2026. Of the sites with a valid llms.txt file, 97% received no requests for it at all during the study period. Of the requests that did arrive, AI retrieval bots accounted for 1.1%. The rest was overwhelmingly SEO tools, validators, and technology profilers checking whether the file existed. The file is mostly being read by the industry that recommends it.

Publishing one costs an hour and does no harm. It is not why you’re invisible, and the hour is better spent on the curl command above.

Schema markup

The correlation is real and it’s why this advice spread. Across 6 million URLs, Ahrefs found pages cited by AI were nearly three times more likely to carry JSON-LD than uncited pages. If you stop there, “add schema, get cited” looks obvious.

Then they ran it as an experiment. They tracked 1,885 pages that added JSON-LD between August 2025 and March 2026, matched them against 4,000 control pages, and ran a difference-in-differences analysis to strip out platform-wide trends. AI Overview citations moved −4.6%. AI Mode moved +2.4%. ChatGPT moved +2.2%. The two positives were statistically indistinguishable from zero, and the negative was small.

Two honest qualifications. Product, review, and article markup still earn rich results in classic search, and that’s reason enough to keep maintaining them. And the tested pages already had 100+ AI Overview citations before the change, meaning they were already being read and parsed successfully. Adding schema to a page that no system has ever understood is a different situation, and it wasn’t measured. What the data doesn’t support is treating markup as the lever that turns citations on.

FAQPage is the sharpest example of why the markup-as-lever instinct ages badly. It was the most-recommended schema in SEO for years. Google restricted it to authoritative government and health sites in 2023, and its structured data documentation now carries a deprecation notice: the rich result stopped appearing in Search on 7 May 2026. If you add FAQPage markup to a SaaS blog today, there is no feature left for it to earn. Google says leaving existing markup in place does no harm, so this isn’t urgent cleanup. It’s a reason to stop treating “add the schema” as a strategy rather than as maintenance.

Why AI Overviews don’t appear for some searches at all

This gets tangled up with the visibility question and it’s a separate thing. Google doesn’t generate an Overview for every query. Coverage is thin or absent on many health and finance queries, on navigational and brand searches, on very fresh news, and on low-volume long tail. Presence also moves with core updates, sometimes losing whole query categories overnight.

If you’re tracking a keyword where the box doesn’t render, there’s nothing to be cited in and no problem to fix. Confirm the surface exists for the query before you spend a sprint optimising for it.

What does seem to move it

I’m going to be careful here, because the platforms publish almost nothing about how sources get selected, and anyone claiming a confirmed mechanism is guessing with confidence. What follows is pattern-matching from published studies and from watching our own queries, not documented behaviour.

Be liftable. Answer-first passages, plainly named entities, structure a parser doesn’t have to interpret. This is the same discipline that gets a passage into an Overview, and it’s covered properly in how to show up in Google AI Overviews.

Be corroborated somewhere other than your own site. AI answers are assembled from what the web says about you, not just what you say about yourself. For any comparison or recommendation query, a large share of the sources will be roundups, forums, and third-party comparisons rather than vendor pages. If nothing off your domain says the thing you want the model to say, the model has no reason to say it.

Be worth quoting. If your page restates what the other candidates already state, there’s no reason to select it over them. That’s information gain, and it matters more in a retrieval system than it did in a ranked list, because a generated answer only needs one source per claim.

The order to check things in

1. Run the prompt eight to ten times per engine, in fresh sessions, and log which domains get cited.
2. Fetch your page with a bot user agent and confirm the copy is in the raw HTML.
3. Check robots.txt, WAF rules, and CDN bot settings for each named bot, training and retrieval separately.
4. Check server logs for whether the retrieval bots are getting 200s.
5. Confirm the AI surface even renders for the query you’re tracking.
6. Look at the sub-questions the topic decomposes into, not just the head term.
7. Find whether anything off your site corroborates the claim you want cited.

Housekeeping like llms.txt and markup goes after all of that, not before it.

Building the brief around the sub-questions rather than the head term is the part that’s easiest to get wrong and hardest to retrofit, which is most of what Ranklet does when it reads the live SERP and the question clusters around a keyword. You can try it on one of your own keywords if you want to see the decomposition before you write.

Questions that keep coming up

Can I block AI crawlers and keep my Google rankings?

For most of them, yes. Google-Extended governs training and grounding for the Gemini apps and the Vertex AI API, and Google’s crawler documentation states that using it “doesn’t affect a site’s inclusion in Search, nor do we use Google-Extended as a ranking signal in Search.” OpenAI’s, Anthropic’s, and Perplexity’s crawlers are separate systems again, and blocking any of them costs you nothing in Google.

The exception is AI Overviews, and it catches people out. There’s no opt-out for the Overview specifically. Google treats AI features as part of Search, so the control is your robots.txt directive for Googlebot, which means blocking the Overview and blocking the ten blue links are the same action. If your worry is how much of the page gets summarised rather than whether you appear at all, nosnippet, data-nosnippet, and max-snippet are the granular controls, and they apply to ordinary snippets too.

How long does it take to show up after I fix a block?

Nobody publishes this, and anyone quoting you a number is guessing. What is observable: the retrieval bots fetch pages in response to live questions, so a newly unblocked page can be picked up within days. The training crawls and the indexes built from them move far more slowly, and some engines won’t reconsider a domain they previously got 403s from for a while.

Practically, watch your logs for the first successful fetch, then re-run your prompt set weekly rather than daily, or run-to-run noise will convince you of things that aren’t happening. If the bots are getting 200s and you’re still absent after a month, the problem has stopped being technical.

Do I need a separate GEO or AEO strategy?

Not as a separate content programme, on the evidence available so far. What genuinely differs is technical, whether a crawler can read the page at all, and structural, which sub-questions you appear for. Both are fixable inside the process you already run.

The discipline didn’t split in two. The failure modes just got quieter, because a page can now be invisible without any of your dashboards showing a drop.

Is being cited even worth chasing?

Worth asking before you reorganise a quarter around it.

Seer Interactive tracked 53 brands across 5.47 million queries and 2.43 billion organic impressions from January 2025 through February 2026. On informational queries, averaged across 2025, organic click-through was 2.07% when the brand was cited in the Overview, 0.94% when it wasn’t, and 3.35% on the same kind of query with no Overview present. Seer are explicit that they can’t claim causation, since stronger brands are also more likely to be cited in the first place.

Read it as damage control rather than upside. Being cited roughly doubles your clicks compared to being ignored, and still leaves you well below what the query was worth before the box existed. That’s a good reason to fix a crawler block this afternoon. It’s a bad reason to spend three weeks on files that 97% of sites never get a single request for.