The BA Insight Website Crawler Connector for Search integrates content from any website—both public and secure—into leading enterprise search platforms including SharePoint, Azure AI Search, Elasticsearch, and others. It enables organizations to surface valuable website content, formerly unreachable via intranet or portal search, in a unified, consolidated index alongside internal repositories. This allows users to discover and access web documents through familiar enterprise search interfaces, regardless of the source.
The connector supports both full and incremental crawls for continuous, high-throughput indexing and strictly enforces source website security—so users only see documents they’re authorized for, even if the sites require authentication. Search results always point back to the original web pages, letting users click through for direct access to current content.
Problems Website Crawler Connector for Search Solves
Difficulty accessing and searching valuable data locked in public or secure company websites.
Fragmented search experiences because web content remains isolated from enterprise repositories.
Security concerns over surfacing and exposing protected web content in enterprise search.
Keeping search indexes continuously updated with changing web documents and assets.
Benefits Website Crawler Connector for Search Delivers
Connect Seamlessly Across Leading Platforms
Seamless integration with SharePoint, Azure AI Search, Elasticsearch, Amazon Kendra, Solr, and other major platforms.
Robust Security for Public and Private Sites
Honors security for both public and private sites; only users with access see protected website documents.
Full and Incremental Crawl Support
Supports both full and incremental crawls for ongoing, accurate indexing.
Preserved Source Document References
Maintains document pointers to original website source, improving user trust and experience.
Unified Access to Intranet, Portal, and Web Content
Drives productivity by enabling single-window access to content from intranet, portal, and the web.
Key Features of the Website Crawler Connector for Search
Key Features of the Website Crawler Connector for Search
Use the left and right arrow keys to move between tabs. Swipe or drag to scroll through more tabs. Press Enter or Space to activate a tab.
Comprehensive Website Indexing
Comprehensive Website Indexing
Crawls and securely indexes public and secure websites, supporting both full and incremental updates.
This connector empowers organizations to make website content instantly actionable within their enterprise search ecosystem. This unlocks external and internal information for productivity and informed decision-making.
FAQ
FAQ
-
Yes, it enforces site-specific authentication and access control so only authorized users see protected content.
-
It supports full and incremental crawls, keeping the search index aligned with frequent website updates.
-
The connector integrates with SharePoint Online and On-Premises, Azure AI Search, Elasticsearch, Solr, Amazon Kendra, and more.
-
No, the connector is agentless; it requires only public or authenticated access—no install required on the source site.
-
By aggregating web content and supporting classification/enrichment, the connector improves findability, filtering, and user navigation.