Chunk
- POSTCreate or Upsert Chunk or Chunks
- POSTSearch
- POSTAutocomplete
- POSTGet Recommended Chunks
- POSTScroll Chunks
- POSTCount chunks above threshold
- POSTGenerate suggested queries
- POSTRAG on Specified Chunks
- PUTUpdate Chunk
- PUTUpdate Chunk By Tracking Id
- GETGet Chunk By Id
- GETGet Chunk By Tracking Id
- POSTGet Chunks By Tracking Ids
- POSTGet Chunks By Ids
- DELDelete Chunk
- DELDelete Chunk By Tracking Id
- DELBulk Delete Chunks
- POSTSplit HTML Content into Chunks
Chunk Group
- POSTCreate or Upsert Group or Groups
- POSTSearch Over Groups
- POSTSearch Within Group
- POSTGet Recommended Groups
- POSTAdd Chunk to Group
- POSTAdd Chunk to Group by Tracking ID
- POSTGet Groups for Chunks
- GETGet Chunks in Group by Tracking ID
- GETGet Group by Tracking ID
- PUTUpdate Group
- DELRemove Chunk from Group
- DELDelete Group by Tracking ID
- DELDelete Group
- GETGet Group
- GETGet Chunks in Group
- GETGet Groups for Dataset
Message
Crawl
File
Analytics
Dataset
- POSTCreate Dataset
- POSTBatch Create Datasets
- POSTGet All Tags
- POSTGet events for the dataset
- PUTUpdate Dataset by ID or Tracking ID
- PUTClear Dataset
- GETGet Dataset By ID
- GETGet Dataset by Tracking ID
- GETGet Datasets from Organization
- POSTCreate ETL Job
- PUTCreate Pagefind Index for Dataset
- GETGet Pagefind Index Url for Dataset
- GETGet Usage By Dataset ID
- GETGet dataset crawl options
- GETGet apipublic page
- DELDelete Dataset
- DELDelete Dataset by Tracking ID
Organization
Health
Stripe
Metrics
Get all crawl requests for a dataset
This endpoint is used to get all crawl requests for a dataset.
curl --request GET \
--url https://api.trieve.ai/api/crawl \
--header 'Authorization: <api-key>' \
--header 'TR-Dataset: <tr-dataset>'[
{
"attempt_number": 123,
"crawl_options": {
"body_remove_strings": [
"Edit on github"
],
"boost_titles": true,
"exclude_paths": [
"https://example.com/exclude*"
],
"exclude_tags": [
"#ad",
"#footer"
],
"heading_remove_strings": [
"Advertisement",
"Sponsored"
],
"include_paths": [
"https://example.com/include*"
],
"include_tags": [
"h1",
"p",
"a",
".main-content"
],
"interval": "daily",
"limit": 1000,
"site_url": "https://example.com"
},
"crawl_type": "firecrawl",
"created_at": "2023-11-07T05:31:56Z",
"dataset_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
"id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
"interval": "<string>",
"next_crawl_at": "2023-11-07T05:31:56Z",
"scrape_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
"status": "Pending",
"url": "<string>"
}
]Authorizations
Headers
The dataset id to use for the request
Query Parameters
The page number to retrieve
The number of items to retrieve per page
Response
Options for setting up the crawl which will populate the dataset.
Option for allowing the crawl to follow links to external websites.
Text strings to remove from body when creating chunks for each page
Boost titles such that keyword matches in titles are prioritized in search results. Strongly recommended to leave this on. Defaults to true.
URL Patterns to exclude from the crawl
Specify the HTML tags, classes and ids to exclude from the response.
Text strings to remove from headings when creating chunks for each page
Ignore the website sitemap when crawling, defaults to true.
URL Patterns to include in the crawl
Specify the HTML tags, classes and ids to include in the response.
Interval at which specified site should be re-scraped
daily, weekly, monthly How many pages to crawl, defaults to 1000
Options for including an openapi spec or shopify settigns
OpenAPI json schema to be processed alongside the site crawl
Tag to look for to determine if a page should create an openapi route chunk instead of chunks from heading-split of the HTML
openapi The URL to crawl
Metadata to send back with the webhook call for each successful page scrape
Host to call back on the webhook for each successful page scrape
firecrawl, openapi, shopify, youtube Pending Was this page helpful?
curl --request GET \
--url https://api.trieve.ai/api/crawl \
--header 'Authorization: <api-key>' \
--header 'TR-Dataset: <tr-dataset>'[
{
"attempt_number": 123,
"crawl_options": {
"body_remove_strings": [
"Edit on github"
],
"boost_titles": true,
"exclude_paths": [
"https://example.com/exclude*"
],
"exclude_tags": [
"#ad",
"#footer"
],
"heading_remove_strings": [
"Advertisement",
"Sponsored"
],
"include_paths": [
"https://example.com/include*"
],
"include_tags": [
"h1",
"p",
"a",
".main-content"
],
"interval": "daily",
"limit": 1000,
"site_url": "https://example.com"
},
"crawl_type": "firecrawl",
"created_at": "2023-11-07T05:31:56Z",
"dataset_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
"id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
"interval": "<string>",
"next_crawl_at": "2023-11-07T05:31:56Z",
"scrape_id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
"status": "Pending",
"url": "<string>"
}
]