# NDRI Dairy Microbiology Alumni Register, ndridmalumni.site # The pages are open to search engines: alumni should be able to find this directory. # The alumni photographs and the directory PDF are not offered for image indexing, # dataset collection or model training. This is the same policy the Foundation site # carries, and it matters more here, because these are 648 named living people. # # The photograph paths are already encoded in the payload so that a scraper cannot # harvest them. These lines say the same thing where a crawler can read it. # ---- default: crawl the pages, leave the photographs and the directory PDF alone ---- User-agent: * Allow: / Disallow: /photos/ Disallow: /archive/ Disallow: /NDRI_DM_ALUMNI_DIRECTORY_BOOK.pdf Disallow: /nhrp/NHRP2026_submission.pdf # ---- social link-preview fetchers need the images, or shared links look broken ---- User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Twitterbot Allow: / User-agent: WhatsApp Allow: / User-agent: Slackbot-LinkExpanding Allow: / User-agent: TelegramBot Allow: / # ---- image-specific crawlers ---- User-agent: Googlebot-Image Disallow: / User-agent: Bingbot-Image Disallow: / User-agent: msnbot-media Disallow: / User-agent: Yandex-Images Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: Pinterestbot Disallow: / User-agent: ImagesiftBot Disallow: / # ---- AI, dataset and scraping crawlers ---- User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Scrapy Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: / Sitemap: https://ndridmalumni.site/sitemap.xml