SEO, AEO, GEO, llms.txt and robots.txt: how AI search actually reads your site

The one mistake that undoes everything else

Most people treat robots.txt as an on/off switch for appearing in Google. It is not. robots.txt controls crawling. Indexing is controlled by a noindex directive in a meta robots tag or an X-Robots-Tag header.

The two collide badly: if you block a page in robots.txt, a crawler can never reach it, so it can never read the noindex telling it not to index it. Google will keep the page, showing only a bare blue link with no description, and you will have no way to remove it through the page itself. To deindex a page reliably, let it be crawlable and serve noindex until it is out of the index.

A minimal, correct robots.txt for a site that wants to be found:

User-agent: *
Allow: /

Sitemap: https://www.taskveda.in/sitemap.xml

Note that this admits every AI crawler. Whether to allow them is a genuine editorial decision, not a technical one — blocking them removes your content from a growing discovery channel without improving your rankings anywhere.

llms.txt: a convention, not an engine feature

llms.txt is a proposed Markdown file, shaped like robots.txt, that gives a site owner a place to say “if you are a language model, read these pages first.”

Be clear-eyed about its status: no major engine requires it, and the large AI assistants do not depend on it. It is cheap to publish and occasionally picked up by crawlers and tools that choose to honour it, so it is worth having. But it is strictly additive — our own llms.txt does nothing for ranking. What actually gets content cited by AI systems is clean, linkable HTML and genuinely useful pages.

The shared foundation for SEO, AEO and GEO

Most of the work is shared, and it is unglamorous:

1. One H1 that names the page accurately. Not a slogan.
2. A question-shaped heading where the page actually answers a question. This is what earns the boxed answer in AEO.
3. Short factual paragraphs. Both featured snippets and AI answers need a passage they can lift whole. A 400-word block of prose cannot be quoted.
4. Real numbers, named sources. A generative engine will not attribute a vague claim. “3,400 students placed, cohort 2025” is citable; “we help students get placed” is not.
5. Internal links that describe the destination. This is the single highest-return SEO task most people skip. Every orphan page you own is authority you are throwing away.
6. Self-referencing canonicals and one clean absolute URL per page. A canonical pointing somewhere else quietly hands your rankings to that other page.

Where the three differ in practice

An order that actually works

1. Week 1 — crawlability. Fix robots.txt, self-canonicals, and any redirect loops. Nothing else can work while a crawler cannot fetch the page.
2. Week 1 — clean up orphans. Every page should have at least one real inbound internal link. This is free and often the biggest single win available.
3. Week 2 — publish llms.txt. One hour, given a clean URL list.
4. Weeks 3–4 — structure for extraction. Add question-shaped headings and tighten the top third of each page into quotable paragraphs.
5. Ongoing — earn links with something worth citing. Original data beats outreach. If your completion rates or placement outcomes are genuinely yours, that is the asset journalists and creators will cite.

What not to bother with

Keyword stuffing, near-duplicate pages with the keyword swapped, and buying directory links. The first two are the same doorway-page pattern that gets a young domain discounted or de-indexed wholesale; the third is ignored at best. If you publish two pages that differ only by a keyword, you have one page and two problems.