Create a new crawl request

POST

api

crawl

curl --request POST \
  --url https://api.trieve.ai/api/crawl \
  --header 'Authorization: <api-key>' \
  --header 'Content-Type: application/json' \
  --header 'TR-Dataset: <tr-dataset>' \
  --data '{
  "crawl_options": {
    "crawl_options": {
      "allow_external_links": false,
      "boost_titles": true,
      "exclude_tags": [
        "#ad",
        "#footer",
        "header",
        "head",
        "navbar",
        "footer",
        "aside",
        "nav",
        "form"
      ],
      "heading_remove_strings": [
        "Advertisement",
        "Sponsored"
      ],
      "ignore_sitemap": true,
      "include_tags": [],
      "interval": "daily",
      "limit": 50,
      "site_url": "nedzo.ai"
    }
  }
}'

{
  "attempt_number": 123,
  "crawl_options": {
    "crawl_options": {
      "allow_external_links": false,
      "boost_titles": true,
      "exclude_tags": [
        "#ad",
        "#footer",
        "header",
        "head",
        "navbar",
        "footer",
        "aside",
        "nav",
        "form"
      ],
      "heading_remove_strings": [
        "Advertisement",
        "Sponsored"
      ],
      "ignore_sitemap": true,
      "include_tags": [],
      "interval": "daily",
      "limit": 50,
      "site_url": "nedzo.ai"
    }
  },
  "crawl_type": "firecrawl",
  "created_at": "2023-11-07T05:31:56Z",
  "dataset_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "interval": "<string>",
  "next_crawl_at": "2023-11-07T05:31:56Z",
  "scrape_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "status": "Pending",
  "url": "<string>"
}

Authorizations

Authorization

string

header

required

Headers

TR-Dataset

string

required

The dataset id to use for the request

Body

application/json

JSON request payload to create a new crawl

crawl_options

object

required

Options for setting up the crawl which will populate the dataset.

crawl_options.add_chunks_to_dataset

boolean | null

Add chunks to the dataset that the crawl is created for, defaults to true

crawl_options.allow_external_links

boolean | null

Option for allowing the crawl to follow links to external websites.

crawl_options.body_remove_strings

string[] | null

Text strings to remove from body when creating chunks for each page

crawl_options.boost_titles

boolean | null

Boost titles such that keyword matches in titles are prioritized in search results. Strongly recommended to leave this on. Defaults to true.

crawl_options.exclude_paths

string[] | null

URL Patterns to exclude from the crawl

crawl_options.exclude_tags

string[] | null

Specify the HTML tags, classes and ids to exclude from the response.

crawl_options.heading_remove_strings

string[] | null

Text strings to remove from headings when creating chunks for each page

crawl_options.ignore_sitemap

boolean | null

Ignore the website sitemap when crawling, defaults to true.

crawl_options.include_paths

string[] | null

URL Patterns to include in the crawl

crawl_options.include_tags

string[] | null

Specify the HTML tags, classes and ids to include in the response.

crawl_options.interval

enum<string>

Interval at which specified site should be re-scraped

Available options:

daily,

weekly,

monthly

crawl_options.limit

integer | null

How many pages to crawl, defaults to 1000

crawl_options.scrape_options

object

Options for including an openapi spec or shopify settigns

crawl_options.site_url

string | null

The URL to crawl

crawl_options.webhook_metadata

any | null

Metadata to send back with the webhook call for each successful page scrape

crawl_options.webhook_urls

string[] | null

Host to call back on the webhook for each successful page scrape

Example:

{
  "crawl_options": {
    "allow_external_links": false,
    "boost_titles": true,
    "exclude_tags": [
      "#ad",
      "#footer",
      "header",
      "head",
      "navbar",
      "footer",
      "aside",
      "nav",
      "form"
    ],
    "heading_remove_strings": ["Advertisement", "Sponsored"],
    "ignore_sitemap": true,
    "include_tags": [],
    "interval": "daily",
    "limit": 50,
    "site_url": "nedzo.ai"
  }
}

Response

200

application/json

Crawl created successfully

attempt_number

integer

required

crawl_options

object

required

Options for setting up the crawl which will populate the dataset.

crawl_options.add_chunks_to_dataset

boolean | null

Add chunks to the dataset that the crawl is created for, defaults to true

crawl_options.allow_external_links

boolean | null

Option for allowing the crawl to follow links to external websites.

crawl_options.body_remove_strings

string[] | null

Text strings to remove from body when creating chunks for each page

crawl_options.boost_titles

boolean | null

Boost titles such that keyword matches in titles are prioritized in search results. Strongly recommended to leave this on. Defaults to true.

crawl_options.exclude_paths

string[] | null

URL Patterns to exclude from the crawl

crawl_options.exclude_tags

string[] | null

Specify the HTML tags, classes and ids to exclude from the response.

crawl_options.heading_remove_strings

string[] | null

Text strings to remove from headings when creating chunks for each page

crawl_options.ignore_sitemap

boolean | null

Ignore the website sitemap when crawling, defaults to true.

crawl_options.include_paths

string[] | null

URL Patterns to include in the crawl

crawl_options.include_tags

string[] | null

Specify the HTML tags, classes and ids to include in the response.

crawl_options.interval

enum<string>

Interval at which specified site should be re-scraped

Available options:

daily,

weekly,

monthly

crawl_options.limit

integer | null

How many pages to crawl, defaults to 1000

crawl_options.scrape_options

object

Options for including an openapi spec or shopify settigns

crawl_options.site_url

string | null

The URL to crawl

crawl_options.webhook_metadata

any | null

Metadata to send back with the webhook call for each successful page scrape

crawl_options.webhook_urls

string[] | null

Host to call back on the webhook for each successful page scrape

Example:

{
  "crawl_options": {
    "allow_external_links": false,
    "boost_titles": true,
    "exclude_tags": [
      "#ad",
      "#footer",
      "header",
      "head",
      "navbar",
      "footer",
      "aside",
      "nav",
      "form"
    ],
    "heading_remove_strings": ["Advertisement", "Sponsored"],
    "ignore_sitemap": true,
    "include_tags": [],
    "interval": "daily",
    "limit": 50,
    "site_url": "nedzo.ai"
  }
}

crawl_type

enum<string>

required

Available options:

firecrawl,

openapi,

shopify,

youtube

created_at

string

required

dataset_id

string

required

string

required

scrape_id

string

required

status

required

Available options:

Pending

url

string

required

interval

string | null

next_crawl_at

string | null

Was this page helpful?

Raise issue

Update a crawl request Delete a crawl request

curl --request POST \
  --url https://api.trieve.ai/api/crawl \
  --header 'Authorization: <api-key>' \
  --header 'Content-Type: application/json' \
  --header 'TR-Dataset: <tr-dataset>' \
  --data '{
  "crawl_options": {
    "crawl_options": {
      "allow_external_links": false,
      "boost_titles": true,
      "exclude_tags": [
        "#ad",
        "#footer",
        "header",
        "head",
        "navbar",
        "footer",
        "aside",
        "nav",
        "form"
      ],
      "heading_remove_strings": [
        "Advertisement",
        "Sponsored"
      ],
      "ignore_sitemap": true,
      "include_tags": [],
      "interval": "daily",
      "limit": 50,
      "site_url": "nedzo.ai"
    }
  }
}'

{
  "attempt_number": 123,
  "crawl_options": {
    "crawl_options": {
      "allow_external_links": false,
      "boost_titles": true,
      "exclude_tags": [
        "#ad",
        "#footer",
        "header",
        "head",
        "navbar",
        "footer",
        "aside",
        "nav",
        "form"
      ],
      "heading_remove_strings": [
        "Advertisement",
        "Sponsored"
      ],
      "ignore_sitemap": true,
      "include_tags": [],
      "interval": "daily",
      "limit": 50,
      "site_url": "nedzo.ai"
    }
  },
  "crawl_type": "firecrawl",
  "created_at": "2023-11-07T05:31:56Z",
  "dataset_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "interval": "<string>",
  "next_crawl_at": "2023-11-07T05:31:56Z",
  "scrape_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "status": "Pending",
  "url": "<string>"
}

Chunk

Chunk Group

Topic

Message

Crawl

File

Analytics

Dataset

Organization

User

Auth

Health

Invitation

Stripe

Metrics

Create a new crawl request

Authorizations

Headers

Body

Response