robots.txt sits at example.com/robots.txt and instructs crawlers what they can and can't access. Mis-configuring it (Disallow: /) blocks your whole site from Google. It hides URLs from crawling, not from being indexed if linked.
/ Related terms
Need help with robots.txt?
We're a Colchester web studio. We build websites that get this stuff right by default.
Get a quote →