GET
/
api
/
crawl
Get all crawl requests for a dataset
curl --request GET \
  --url https://api.trieve.ai/api/crawl \
  --header 'Authorization: <api-key>' \
  --header 'TR-Dataset: <tr-dataset>'
[
  {
    "attempt_number": 123,
    "crawl_options": {
      "crawl_options": {
        "allow_external_links": false,
        "boost_titles": true,
        "exclude_tags": [
          "#ad",
          "#footer",
          "header",
          "head",
          "navbar",
          "footer",
          "aside",
          "nav",
          "form"
        ],
        "heading_remove_strings": [
          "Advertisement",
          "Sponsored"
        ],
        "ignore_sitemap": true,
        "include_tags": [],
        "interval": "daily",
        "limit": 50,
        "site_url": "nedzo.ai"
      }
    },
    "crawl_type": "firecrawl",
    "created_at": "2023-11-07T05:31:56Z",
    "dataset_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
    "id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
    "interval": "<string>",
    "next_crawl_at": "2023-11-07T05:31:56Z",
    "scrape_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
    "status": "Pending",
    "url": "<string>"
  }
]

Authorizations

​
Authorization
string
header
required

Headers

​
TR-Dataset
string<uuid>
required

The dataset id to use for the request

Query Parameters

​
page
integer | null

The page number to retrieve

​
limit
integer | null

The number of items to retrieve per page

Response

200
application/json

Crawl requests retrieved successfully

The response is of type object[].