Questions about our crawler and indexing technology

Searchine uses self made spider technology to crawl your site and analyse the documents structure and text.
We will take notice of hyperlinks to other pages / documents and follow those as well.
The crawler will take notice of noindex and nofollow requests as it will process robots / XML sitemaps and behave just like any other crawler.

It all starts with providing each content page with relevant and unique content and adding a page title and meta descriptions with the most important keywords.

Also make sure that duplicate pages have a canonical link tag so we can eliminate posslbe duplicate search results. 

At this points we are fully cloud-based. For enterprise customers, please contact us

The most effective way to prevent a page from being indexed is to have a noindex meta tag on the requested page.

<meta name="robots" contents="noindex, follow">

By using "noindex, follow" any link on the page will be followed. 

Another way is to have the page excluded in robots.txt

In the near future you can remove pages from the management portal. 
For now the most effective way is to have a noindex meta tag in your page.

That will depend on the subscription. 
We will take notice of the changefrequency property in sitemaps to determine content pages that change frequently and index those more often. 

Yes we will take notice of the robots.txt file and crawl any sitemap mentioned in the file.

Yes, we suggest using canonical URL's to prevent duplicate search results.

At this moment we don't support files. This is on our future roadmap.

Please contact us so we can inform you when we release this.

At this moment we don't have support for full ajax sites.