Robots.txt file on arb sites
- Thread starter sortran
- Start date
- Status
Yessir.
If you want to prevent spiders from crawling your entire site:
Or for specific pages:
You can use robots.txt to prevent an entire directory from being crawled too.
If you want to prevent spiders from crawling your entire site:
Code:
User-agent: *
Disallow: /Or for specific pages:
Code:
<meta name="robots" content="noindex,nofollow" />You can use robots.txt to prevent an entire directory from being crawled too.
- Status