Crawl
Update a crawl request
This endpoint is used to update an existing crawl request for a dataset. The request payload should contain the crawl id and the crawl options to update for the crawl.
PUT
Authorizations
Headers
The dataset id to use for the request
Body
application/json
JSON request payload to update a crawl
Crawl ID to update
Options for setting up the crawl which will populate the dataset.
Example:
{
"crawl_options": {
"allow_external_links": false,
"boost_titles": true,
"exclude_tags": [
"#ad",
"#footer",
"header",
"head",
"navbar",
"footer",
"aside",
"nav",
"form"
],
"heading_remove_strings": ["Advertisement", "Sponsored"],
"ignore_sitemap": true,
"include_tags": [],
"interval": "daily",
"limit": 50,
"site_url": "nedzo.ai"
}
}
Response
200
application/json
Crawl updated successfully
Options for setting up the crawl which will populate the dataset.
Example:
{
"crawl_options": {
"allow_external_links": false,
"boost_titles": true,
"exclude_tags": [
"#ad",
"#footer",
"header",
"head",
"navbar",
"footer",
"aside",
"nav",
"form"
],
"heading_remove_strings": ["Advertisement", "Sponsored"],
"ignore_sitemap": true,
"include_tags": [],
"interval": "daily",
"limit": 50,
"site_url": "nedzo.ai"
}
}
Available options:
firecrawl
, openapi
, shopify
, youtube
Available options:
Pending
Was this page helpful?