Skip to main content

Web Links and Website Crawler

Both of these features are designed to make it easier to work with content that is located on Internet resources, sites, web pages, or online documents.

This feature is only available to authorized users.

Links Crawler - where users can provide URLs of Google Docs and Google Sheets (each link from a new line, 10 links max). Also, straight website links are supported in a current library (https://ingestai.io/docs/howto). The Links Crawler will analyze all content on the page and after that in WebChat (https://ingestai.io/docs/webchat-embedding) or in some of these bots (https://ingestai.io/docs/discord) there is an opportunity to ask questions about this content. This feature is available for all account types.

Step 1​

Go to the Library tab and in the created library, need to switch from the [Files] tab to the [Links Crawler] tab.

Links Crawler Step 1

Step 2​

Fill in the "Library Datas" input a URL that needs analyzed by the Crawler and push on the [Save] button.

Links Crawler Step 2

All responses which were got save on the Prompts page (Pic 4)

Step 3​

After successfully creating the links in the library we can ask any questions about content which contained on this page. For example, we can ask in Web chat https://ingestai.io/docs/webchat-embedding

Web Crawler​

Web Crawler - where users can provide URLs of Google Docs and Google Sheets. Also, straight website links are supported in a current library (https://ingestai.io/docs/howto). The crawler will go to your site and crawl all the internal pages it finds on each page. The crawl limit is 100 links.

The Web Crawler will analyze all content on the page and after that in WebChat (https://ingestai.io/docs/webchat-embedding) or in some of these bots (https://ingestai.io/docs/discord) there is an opportunity to ask questions about this content. This feature is available only for PRO and BUSINESS account types.

Step 1​

Go to the Library tab and in the created library, need to switch from the [Files] tab to the [Web Crawler] tab.

Web Crawler Step 1

Step 2​

Fill in the “Library Datas” input a URL that needs analyzed by the Crawler and push on the [Crawl] button.

Web Crawler Step 2

Step 3​

After successfully creating the links in the library we can ask any questions about content which contained on this resource. For example, we can ask in Web chat https://ingestai.io/docs/webchat-embedding