Reviewed July 2026. This article was substantially updated to reflect current web standards and practices.
Older SEO articles often describe 'getting spiders to notice' a site. Modern search discovery is better treated as a technical system: search engines find URLs, crawl accessible resources, render content, choose canonical versions, and decide whether pages are useful enough to index.
Make important URLs discoverable
Use crawlable HTML links from navigation and relevant content. Submit an accurate XML sitemap, keep canonical URLs consistent, and avoid orphaning priority pages behind scripts, forms, or internal search.
Control access carefully
A robots.txt rule controls crawling, not guaranteed indexing. Use an indexable response for pages you want found and a supported noindex directive for pages you want excluded. Do not block resources needed to render the main content.
Return meaningful HTTP responses
Real pages should return 200, moved pages should redirect to the best replacement, and missing pages should return 404 or 410. Avoid soft 404s, redirect chains, duplicate parameter pages, and contradictory canonicals.
Verify with search tools
Use Google Search Console and other webmaster tools to inspect representative URLs, sitemap processing, indexing reasons, mobile rendering, and structured data. Logs can show whether crawlers reach the pages you expect.
Discovery, crawling, and indexing are different
Discovery means a search engine learns that a URL exists through links, sitemaps, feeds, or previous crawls. Crawling means it requests the URL and associated resources. Rendering means it processes the response into a page representation. Indexing means it may store and organize the page for retrieval. A discovered URL is not guaranteed to be crawled or indexed.
Technical indexability checklist
- The preferred URL returns a successful response without a redirect chain.
- Robots rules allow required pages and resources to be crawled.
- No accidental
noindexdirective excludes the page. - The canonical points to the intended indexable URL.
- Primary content is present in rendered HTML and not hidden behind an interaction.
- Internal links use crawlable anchors and do not orphan the page.
- The XML sitemap includes only preferred indexable URLs with accurate modification dates.
When a page is not indexed
Inspect the exact URL in Search Console, compare the user-declared and selected canonical, and review the rendered page. Check for duplication, thin or obsolete content, soft 404 signals, access failures, redirect loops, and conflicting directives. Server logs can confirm whether crawlers request the URL and what response they receive.
Do not repeatedly resubmit unchanged low-value pages. Improve the content, internal context, and technical consistency, then allow search systems to recrawl. Indexation is not a substitute for a page that satisfies a real audience need.