Learn how to how use the crawl functionality within Trieve
site_url
: The URL of the website you want to crawl.interval
: How often you want to crawl the website.webhook_url
: URL to call for each successful page scrape.shopify
: This will crawl a Shopify store and extract product information.youtube
: This will crawl a Youtube channel and extract video information.crawl_type
parameter inside of the scrape_options
field and specify the type of crawl you want to do.
interval
parameter and specifying how often to recrawl. We will by
default crawl every month.