Crawl
Get all crawl requests for a dataset
This endpoint is used to get all crawl requests for a dataset.
GET
Authorizations
Headers
The dataset id to use for the request
Query Parameters
The page number to retrieve
The number of items to retrieve per page
Response
200
application/json
Crawl requests retrieved successfully
Options for setting up the crawl which will populate the dataset.
Example:
{
"crawl_options": {
"allow_external_links": false,
"boost_titles": true,
"exclude_tags": [
"#ad",
"#footer",
"header",
"head",
"navbar",
"footer",
"aside",
"nav",
"form"
],
"heading_remove_strings": ["Advertisement", "Sponsored"],
"ignore_sitemap": true,
"include_tags": [],
"interval": "daily",
"limit": 50,
"site_url": "nedzo.ai"
}
}
Available options:
firecrawl
, openapi
, shopify
, youtube
Available options:
Pending
Was this page helpful?