> For the complete documentation index, see [llms.txt](https://docs.makerobos.com/frontman-enterprise-chatbot-platform/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.makerobos.com/frontman-enterprise-chatbot-platform/instinct-ai/train-ai/data-loaders/crawl-website.md).

# Crawl Website

Capture content from your website by simply entering the URL. This allows your AI to learn the specific language and terminology used on your site, fostering a natural and relevant conversational experience.

### The Frontman AI chatbot offers 3 website crawling options:

<div><figure><img src="/files/Eh1uK5z8wZuiM0HAY1pO" alt=""><figcaption></figcaption></figure> <figure><img src="/files/g1EU8NLoTGEQwnIfoIYN" alt=""><figcaption></figcaption></figure> <figure><img src="/files/acrUypTOLFLfAAF9qjix" alt=""><figcaption></figcaption></figure></div>

* **Single URL:** Allows importing content from a single specified webpage.
* **Multiple URLs:** Enables importing content from a list of provided URLs.
* **Site-wide (Sitemap.xml):** Facilitates importing content from an entire website by leveraging its sitemap file.

### Website Processing

<figure><img src="/files/ozSSkfrQhzjFPCMMFGjj" alt=""><figcaption></figcaption></figure>

Once you have crawled your website, the system will begin processing them. This may take a few minutes depending on the size and complexity of the files.

### Testing Your Chatbot

<figure><img src="/files/lEoFvoBzeX7YBV2UnUYJ" alt=""><figcaption></figcaption></figure>

Once the website processing is complete, you can test your chatbot's knowledge of the captured content.

1. **Access Crawled Data:** Navigate to the "Data Bank" section within the Frontman platform.
2. **View Crawled Website:** Select the "View" action for the desired crawled website. This will display a preview of the crawled content on the left side of the screen.
3. **Interact with Chatbot:** On the right side of the screen, you can interact with the chatbot by asking questions related to the crawled website. The chatbot will provide answers based on the captured information.