• Tutorials
  • Docs
  • FAQ
  •   Login
  •  Corporate site
  • English
  • Nederlands
  •   Login
  •   Corporate site
    English Nederlands
Getting started with Searchine
  • Step-by-step guide
  • Choosing the right widget
  • Embedded widgets
  • Advanced widgets
  • Search suggestions widget
Tutorials
  • Search & Indexing
    • Crawler setup
    • How does Searchine index your site
    • Exclude specific URLs
Developer documentation
  • Embedded widget
  • Advanced widget
  • Search suggestions widget
  • Search options
  • Examples
Frequently asked questions
Tutorials
Search & Indexing
Exclude specific URLs
  • Crawler setup
  • How does Searchine index your site

Exclude specific URLs

Searchines uses common practices to determine if a page will be indexed or not:

  1. If url is blocked by robots.txt.
  2. If meta robots tag blocks the page from indexing or following.
  3. A custom searchine meta element can be used.
How does Searchine index your site
© Searchine 2025  | 
 Contact
My Searchine  |  Corporate