# Get All User Events Source: https://docs.trieve.ai/api-reference/analytics/get-all-user-events post /api/analytics/events/all This route allows you to view all user events. # Get Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-analytics post /api/analytics This route allows you to view the analytics for a dataset using a structured query. # Get Cluster Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-cluster-analytics post /api/analytics/search/cluster This route allows you to view the cluster analytics for a dataset. # Get Component Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-component-analytics post /api/analytics/events/component This route allows you to view the component analytics for a dataset. # Get CTR Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-ctr-analytics post /api/analytics/events/ctr This route allows you to view the CTR analytics for a dataset. # Get RAG Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-rag-analytics post /api/analytics/rag This route allows you to view the RAG analytics for a dataset. # Get Recommendation Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-recommendation-analytics post /api/analytics/recommendations This route allows you to view the recommendation analytics for a dataset. # Get Search Analytics Source: https://docs.trieve.ai/api-reference/analytics/get-search-analytics post /api/analytics/search This route allows you to view the search analytics for a dataset. # Get Top Datasets Source: https://docs.trieve.ai/api-reference/analytics/get-top-datasets post /api/analytics/top This route allows you to view the top datasets for a given type. # Get User Event By ID Source: https://docs.trieve.ai/api-reference/analytics/get-user-event-by-id get /api/analytics/events/{event_id} This route allows you to view an user event by its ID. You can pass in any type of event and get the details for that event. # Rate RAG Source: https://docs.trieve.ai/api-reference/analytics/rate-rag put /api/analytics/rag This route allows you to Rate a RAG query. # Rate Search Source: https://docs.trieve.ai/api-reference/analytics/rate-search put /api/analytics/search This route allows you to Rate a search query. # Send CTR Data Source: https://docs.trieve.ai/api-reference/analytics/send-ctr-data put /api/analytics/ctr This route allows you to send clickstream data to the system. Clickstream data is used to fine-tune the re-ranking of search results and recommendations. # Send User Event Data Source: https://docs.trieve.ai/api-reference/analytics/send-user-event-data put /api/analytics/events This route allows you to send user event data to the system. # Get Me Source: https://docs.trieve.ai/api-reference/auth/get-me get /api/auth/me Get the user corresponding to your current auth credentials. # Login Source: https://docs.trieve.ai/api-reference/auth/login get /api/auth This will redirect you to the OAuth provider for authentication with email/pass, SSO, Google, Github, etc. # Logout Source: https://docs.trieve.ai/api-reference/auth/logout delete /api/auth Invalidate your current auth credential stored typically stored in a cookie. This does not invalidate your API key. # OpenID Connect callback Source: https://docs.trieve.ai/api-reference/auth/openid-connect-callback get /api/auth/callback This is the callback route for the OAuth provider, it should not be called directly. Redirects to browser with set-cookie header. # null Source: https://docs.trieve.ai/api-reference/auth/post-apiauthcreate_api_only_user post /api/auth/create_api_only_user # Add Chunk to Group Source: https://docs.trieve.ai/api-reference/chunk-group/add-chunk-to-group post /api/chunk_group/chunk/{group_id} Route to add a chunk to a group. One of chunk_id or chunk_tracking_id must be provided. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Add Chunk to Group by Tracking ID Source: https://docs.trieve.ai/api-reference/chunk-group/add-chunk-to-group-by-tracking-id post /api/chunk_group/tracking_id/{tracking_id} Route to add a chunk to a group by tracking id. One of chunk_id or chunk_tracking_id must be provided. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Autocomplete Search Over Groups Source: https://docs.trieve.ai/api-reference/chunk-group/autocomplete-search-over-groups post /api/chunk_group/group_oriented_autocomplete This route provides the primary autocomplete functionality for the API. This prioritize prefix matching with semantic or full-text search. # Create or Upsert Group or Groups Source: https://docs.trieve.ai/api-reference/chunk-group/create-or-upsert-group-or-groups post /api/chunk_group Create new chunk_group(s). This is a way to group chunks together. If you try to create a chunk_group with the same tracking_id as an existing chunk_group, this operation will fail. Only 1000 chunk groups can be created at a time. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Delete Group Source: https://docs.trieve.ai/api-reference/chunk-group/delete-group delete /api/chunk_group/{group_id} This will delete a chunk_group. If you set delete_chunks to true, it will also delete the chunks within the group. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Delete Group by Tracking ID Source: https://docs.trieve.ai/api-reference/chunk-group/delete-group-by-tracking-id delete /api/chunk_group/tracking_id/{tracking_id} Delete a chunk_group with the given tracking id. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get Chunks in Group Source: https://docs.trieve.ai/api-reference/chunk-group/get-chunks-in-group get /api/chunk_group/{group_id}/{page} Route to get all chunks for a group. The response is paginated, with each page containing 10 chunks. Page is 1-indexed. # Get Chunks in Group by Tracking ID Source: https://docs.trieve.ai/api-reference/chunk-group/get-chunks-in-group-by-tracking-id get /api/chunk_group/tracking_id/{group_tracking_id}/{page} Route to get all chunks for a group. The response is paginated, with each page containing 10 chunks. Support for custom page size is coming soon. Page is 1-indexed. # Get Group Source: https://docs.trieve.ai/api-reference/chunk-group/get-group get /api/chunk_group/{group_id} Fetch the group with the given id. # Get Group by Tracking ID Source: https://docs.trieve.ai/api-reference/chunk-group/get-group-by-tracking-id get /api/chunk_group/tracking_id/{tracking_id} Fetch the group with the given tracking id. get_group_by_tracking_id # Get Groups for Chunks Source: https://docs.trieve.ai/api-reference/chunk-group/get-groups-for-chunks post /api/chunk_group/chunks Route to get the groups that a chunk is in. # Get Groups for Dataset Source: https://docs.trieve.ai/api-reference/chunk-group/get-groups-for-dataset get /api/dataset/groups/{dataset_id}/{page} Fetch the groups which belong to a dataset specified by its id. # Get Recommended Groups Source: https://docs.trieve.ai/api-reference/chunk-group/get-recommended-groups post /api/chunk_group/recommend Route to get recommended groups. This route will return groups which are similar to the groups in the request body. You must provide at least one positive group id or group tracking id. # Remove Chunk from Group Source: https://docs.trieve.ai/api-reference/chunk-group/remove-chunk-from-group delete /api/chunk_group/chunk/{group_id} Route to remove a chunk from a group. Auth'ed user or api key must be an admin or owner of the dataset's organization to remove a chunk from a group. # Search Over Groups Source: https://docs.trieve.ai/api-reference/chunk-group/search-over-groups post /api/chunk_group/group_oriented_search This route allows you to get groups as results instead of chunks. Each group returned will have the matching chunks sorted by similarity within the group. This is useful for when you want to get groups of chunks which are similar to the search query. If choosing hybrid search, the top chunk of each group will be re-ranked using scores from a cross encoder model. Compatible with semantic, fulltext, or hybrid search modes. # Search Within Group Source: https://docs.trieve.ai/api-reference/chunk-group/search-within-group post /api/chunk_group/search This route allows you to search only within a group. This is useful for when you only want search results to contain chunks which are members of a specific group. If choosing hybrid search, the results will be re-ranked using scores from a cross encoder model. # Update Group Source: https://docs.trieve.ai/api-reference/chunk-group/update-group put /api/chunk_group Update a chunk_group. One of group_id or tracking_id must be provided. If you try to change the tracking_id to one that already exists, this operation will fail. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Autocomplete Source: https://docs.trieve.ai/api-reference/chunk/autocomplete post /api/chunk/autocomplete This route provides the primary autocomplete functionality for the API. This prioritize prefix matching with semantic or full-text search. # Bulk Delete Chunks Source: https://docs.trieve.ai/api-reference/chunk/bulk-delete-chunk delete /api/chunk Delete multiple chunks using a filter. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Count chunks above threshold Source: https://docs.trieve.ai/api-reference/chunk/count-chunks-above-threshold post /api/chunk/count This route can be used to determine the number of chunk results that match a search query including score threshold and filters. It may be high latency for large limits. There is a dataset configuration imposed restriction on the maximum limit value (default 10,000) which is used to prevent DDOS attacks. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Create or Upsert Chunk or Chunks Source: https://docs.trieve.ai/api-reference/chunk/create-or-upsert-chunk-or-chunks post /api/chunk Create new chunk(s). If the chunk has the same tracking_id as an existing chunk, the request will fail. Once a chunk is created, it can be searched for using the search endpoint. If uploading in bulk, the maximum amount of chunks that can be uploaded at once is 120 chunks. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Delete Chunk Source: https://docs.trieve.ai/api-reference/chunk/delete-chunk delete /api/chunk/{chunk_id} Delete a chunk by its id. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Delete Chunk By Tracking Id Source: https://docs.trieve.ai/api-reference/chunk/delete-chunk-by-tracking-id delete /api/chunk/tracking_id/{tracking_id} Delete a chunk by tracking_id. This is useful for when you are coordinating with an external system and want to use the tracking_id to identify the chunk. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Generate suggested queries Source: https://docs.trieve.ai/api-reference/chunk/generate-suggested-queries post /api/chunk/suggestions This endpoint will generate 3 suggested queries based off a hybrid search using RAG with the query provided in the request body and return them as a JSON object. # Get Chunk By Id Source: https://docs.trieve.ai/api-reference/chunk/get-chunk-by-id get /api/chunk/{chunk_id} Get a singular chunk by id. # Get Chunk By Tracking Id Source: https://docs.trieve.ai/api-reference/chunk/get-chunk-by-tracking-id get /api/chunk/tracking_id/{tracking_id} Get a singular chunk by tracking_id. This is useful for when you are coordinating with an external system and want to use your own id as the primary reference for a chunk. # Get Chunks By Ids Source: https://docs.trieve.ai/api-reference/chunk/get-chunks-by-ids post /api/chunks Get multiple chunks by multiple ids. # Get Chunks By Tracking Ids Source: https://docs.trieve.ai/api-reference/chunk/get-chunks-by-tracking-ids post /api/chunks/tracking Get multiple chunks by ids. # Get Recommended Chunks Source: https://docs.trieve.ai/api-reference/chunk/get-recommended-chunks post /api/chunk/recommend Get recommendations of chunks similar to the positive samples in the request and dissimilar to the negative. # RAG on Specified Chunks Source: https://docs.trieve.ai/api-reference/chunk/rag-on-specified-chunks post /api/chunk/generate This endpoint exists as an alternative to the topic+message resource pattern where our Trieve handles chat memory. With this endpoint, the user is responsible for providing the context window and the prompt and the conversation is ephemeral. # Scroll Chunks Source: https://docs.trieve.ai/api-reference/chunk/scroll-chunks post /api/chunks/scroll Get paginated chunks from your dataset with filters and custom sorting. If sort by is not specified, the results will sort by the id's of the chunks in ascending order. Sort by and offset_chunk_id cannot be used together; if you want to scroll with a sort by then you need to use a must_not filter with the ids you have already seen. There is a limit of 1000 id's in a must_not filter at a time. # Search Source: https://docs.trieve.ai/api-reference/chunk/search post /api/chunk/search This route provides the primary search functionality for the API. It can be used to search for chunks by semantic similarity, full-text similarity, or a combination of both. Results' `chunk_html` values will be modified with `` or custom specified tags for sub-sentence highlighting. # Split HTML Content into Chunks Source: https://docs.trieve.ai/api-reference/chunk/split-html-content-into-chunks post /api/chunk/split This endpoint receives a single html string and splits it into chunks based on the headings and body content. The headings are split based on heading html tags. chunk_html has a maximum size of 256Kb. # Update Chunk Source: https://docs.trieve.ai/api-reference/chunk/update-chunk put /api/chunk Update a chunk. If you try to change the tracking_id of the chunk to have the same tracking_id as an existing chunk, the request will fail. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Update Chunk By Tracking Id Source: https://docs.trieve.ai/api-reference/chunk/update-chunk-by-tracking-id put /api/chunk/tracking_id/update Update a chunk by tracking_id. This is useful for when you are coordinating with an external system and want to use the tracking_id to identify the chunk. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Create a new crawl request Source: https://docs.trieve.ai/api-reference/crawl/create-a-new-crawl-request post /api/crawl This endpoint is used to create a new crawl request for a dataset. The request payload should contain the crawl options to use for the crawl. # Delete a crawl request Source: https://docs.trieve.ai/api-reference/crawl/delete-a-crawl-request delete /api/crawl/{crawl_id} This endpoint is used to delete an existing crawl request for a dataset. The request payload should contain the crawl id to delete. # Get all crawl requests for a dataset Source: https://docs.trieve.ai/api-reference/crawl/get-all-crawl-requests-for-a-dataset get /api/crawl This endpoint is used to get all crawl requests for a dataset. # Update a crawl request Source: https://docs.trieve.ai/api-reference/crawl/update-a-crawl-request put /api/crawl This endpoint is used to update an existing crawl request for a dataset. The request payload should contain the crawl id and the crawl options to update for the crawl. # Batch Create Datasets Source: https://docs.trieve.ai/api-reference/dataset/batch-create-datasets post /api/dataset/batch_create_datasets Datasets will be created in the org specified via the TR-Organization header. Auth'ed user must be an owner of the organization to create datasets. If a tracking_id is ignored due to it already existing on the org, the response will not contain a dataset with that tracking_id and it can be assumed that a dataset with the missing tracking_id already exists. # Clear Dataset Source: https://docs.trieve.ai/api-reference/dataset/clear-dataset put /api/dataset/clear/{dataset_id} Removes all chunks, files, and groups from the dataset while retaining the analytics and dataset itself. The auth'ed user must be an owner of the organization to clear a dataset. # Clone Dataset Source: https://docs.trieve.ai/api-reference/dataset/clone-dataset post /api/dataset/clone Clones a dataset and creates a new dataset with the same configuration and chunks. The auth'ed user must be an owner of the organization to clone a dataset. # Create Dataset Source: https://docs.trieve.ai/api-reference/dataset/create-dataset post /api/dataset Dataset will be created in the org specified via the TR-Organization header. Auth'ed user must be an owner of the organization to create a dataset. # Create ETL Job Source: https://docs.trieve.ai/api-reference/dataset/create-etl-job post /api/etl/create_job This endpoint is used to create a new ETL job for a dataset. # Create Pagefind Index for Dataset Source: https://docs.trieve.ai/api-reference/dataset/create-pagefind-index-for-dataset put /api/dataset/pagefind Uses pagefind to index the dataset and store the result into a CDN for retrieval. The auth'ed user must be an admin of the organization to create a pagefind index for a dataset. # Delete Dataset Source: https://docs.trieve.ai/api-reference/dataset/delete-dataset delete /api/dataset/{dataset_id} Auth'ed user must be an owner of the organization to delete a dataset. # Delete Dataset by Tracking ID Source: https://docs.trieve.ai/api-reference/dataset/delete-dataset-by-tracking-id delete /api/dataset/tracking_id/{tracking_id} Auth'ed user must be an owner of the organization to delete a dataset. # Get All Tags Source: https://docs.trieve.ai/api-reference/dataset/get-all-tags post /api/dataset/get_all_tags Scroll through all tags in the dataset and get the number of chunks in the dataset with that tag plus the total number of unique tags for the whole datset. # Get Dataset By ID Source: https://docs.trieve.ai/api-reference/dataset/get-dataset-by-id get /api/dataset/{dataset_id} Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get Dataset by Tracking ID Source: https://docs.trieve.ai/api-reference/dataset/get-dataset-by-tracking-id get /api/dataset/tracking_id/{tracking_id} Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # null Source: https://docs.trieve.ai/api-reference/dataset/get-dataset-crawl-options get /api/dataset/crawl_options/{dataset_id} # Get Datasets from Organization Source: https://docs.trieve.ai/api-reference/dataset/get-datasets-from-organization get /api/dataset/organization/{organization_id} Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get events for the dataset Source: https://docs.trieve.ai/api-reference/dataset/get-events-for-the-dataset post /api/dataset/events Get events for the dataset specified by the TR-Dataset header. # Get file and chunk creation queue lengths Source: https://docs.trieve.ai/api-reference/dataset/get-file-and-chunk-creation-queue-lengths get /api/dataset/get_dataset_queue_lengths Get the queue lengths for file and chunk creation. # Get Pagefind Index Url for Dataset Source: https://docs.trieve.ai/api-reference/dataset/get-pagefind-index-url-for-dataset get /api/dataset/pagefind Returns the root URL for your pagefind index, will error if pagefind is not enabled # Get Usage By Dataset ID Source: https://docs.trieve.ai/api-reference/dataset/get-usage-by-dataset-id get /api/dataset/usage/{dataset_id} Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Update Dataset by ID or Tracking ID Source: https://docs.trieve.ai/api-reference/dataset/update-dataset-by-id-or-tracking-id put /api/dataset One of id or tracking_id must be provided. The auth'ed user must be an owner of the organization to update a dataset. # Ab Test Source: https://docs.trieve.ai/api-reference/experiment/ab-test post /api/experiment/ab-test Get a user's treatment for an experiment. Auth'ed user must be an owner of the organization to get a user's treatment. # Create Experiment Source: https://docs.trieve.ai/api-reference/experiment/create-experiment post /api/experiment Experiment will be created in the dataset specified via the TR-Dataset header. Auth'ed user must be an owner of the organization to create an experiment. # Delete Experiment Source: https://docs.trieve.ai/api-reference/experiment/delete-experiment delete /api/experiment/{experiment_id} Delete an experiment. Auth'ed user must be an owner of the organization to delete an experiment. # Update Experiment Source: https://docs.trieve.ai/api-reference/experiment/update-experiment put /api/experiment Update an experiment. Auth'ed user must be an owner of the organization to update an experiment. # Create Presigned CSV/JSONL S3 PUT URL Source: https://docs.trieve.ai/api-reference/file/create-presigned-url-for-csv-jsonl post /api/file/csv_or_jsonl This route is useful for uploading very large CSV or JSONL files. Once you have completed the upload, chunks will be automatically created from the file for each line in the CSV or JSONL file. The chunks will be indexed and searchable. Auth'ed user must be an admin or owner of the dataset's organization to upload a file. # Delete File Source: https://docs.trieve.ai/api-reference/file/delete-file delete /api/file/{file_id} Delete a file from S3 attached to the server based on its id. This will disassociate chunks from the file, but only delete them all together if you specify delete_chunks to be true. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get File with Signed URL Source: https://docs.trieve.ai/api-reference/file/get-file get /api/file/{file_id} Get all of the information for a file along with a signed s3 url corresponding to the file_id requested such that you can download the file. # Get File with Signed URL Source: https://docs.trieve.ai/api-reference/file/get-file-with-signed-url get /api/file/{file_id} Get all of the information for a file along with a signed s3 url corresponding to the file_id requested such that you can download the file. # Get Files and Group IDs for Dataset Source: https://docs.trieve.ai/api-reference/file/get-files-and-group-ids-for-dataset get /api/dataset/files/{dataset_id}/{page} Get all files and their group ids which belong to a given dataset specified by the dataset_id parameter. 10 files and group ids are returned per page. This route may return the same file multiple times if the file is associated with multiple groups. # Get Files and Group IDs for Dataset Source: https://docs.trieve.ai/api-reference/file/get-files-for-dataset get /api/dataset/files/{dataset_id}/{page} Get all files and their group ids which belong to a given dataset specified by the dataset_id parameter. 10 files and group ids are returned per page. This route may return the same file multiple times if the file is associated with multiple groups. # Scroll Files with Groups Source: https://docs.trieve.ai/api-reference/file/scroll-files get /api/dataset/scroll_files Scroll through the files along with their groups in a dataset. This is useful for paginating through files. The cursor is used to fetch the next page of files. The page size is used to specify how many files to fetch per page. The default page size is 10. # Scroll Files with Groups Source: https://docs.trieve.ai/api-reference/file/scroll-files-with-groups get /api/dataset/scroll_files Scroll through the files along with their groups in a dataset. This is useful for paginating through files. The cursor is used to fetch the next page of files. The page size is used to specify how many files to fetch per page. The default page size is 10. # Upload File Source: https://docs.trieve.ai/api-reference/file/upload-file post /api/file Upload a file to S3 bucket attached to your dataset. You can select between a naive chunking strategy where the text is extracted with Apache Tika and split into segments with a target number of segments per chunk OR you can use a vision LLM to convert the file to markdown and create chunks per page. You must specifically use a base64url encoding. Auth'ed user must be an admin or owner of the dataset's organization to upload a file. # Upload HTML Page Source: https://docs.trieve.ai/api-reference/file/upload-html-page post /api/file/html_page Chunk HTML by headings and queue for indexing into the specified dataset. # Health Check Source: https://docs.trieve.ai/api-reference/health/health-check get /api/health Confirmation that the service is healthy and can make embedding vectors # Delete Invitation Source: https://docs.trieve.ai/api-reference/invitation/delete-invitation delete /api/invitation/{invitation_id} Delete an invitation by id. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get Invitations Source: https://docs.trieve.ai/api-reference/invitation/get-invitations get /api/invitations/{organization_id} Get all invitations for the organization. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Send Invitation Source: https://docs.trieve.ai/api-reference/invitation/send-invitation post /api/invitation Invitations act as a way to invite users to join an organization. After a user is invited, they will automatically be added to the organization with the role specified in the invitation once they set their. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Create message Source: https://docs.trieve.ai/api-reference/message/create-message post /api/message Create message. Messages are attached to topics in order to coordinate memory of gen-AI chat sessions.Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Edit Image Source: https://docs.trieve.ai/api-reference/message/edit-image post /api/message/edit_image Uses `gpt-image-1` to edit an images based on a given prompt. Note that the images must be base64 encoded and all must have the same mime type. # Edit message Source: https://docs.trieve.ai/api-reference/message/edit-message put /api/message This will delete the specified message and replace it with a new message. All messages after the message being edited in the sort order will be deleted. The new message will be generated by the AI based on the new content provided in the request body. The response will include Chunks first on the stream if the topic is using RAG. The structure will look like `[chunks]||mesage`. See docs.trieve.ai for more information. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Generate Message Completions Source: https://docs.trieve.ai/api-reference/message/generate-message-completions post /api/message/generate_message_completions Uses `openai` to generate a message completion for a given prompt. # Get a message by its ID Source: https://docs.trieve.ai/api-reference/message/get-a-message-by-its-id get /api/message/{message_id} Quickly get the full object for a given message. From the message, you can get the topic and all messages which exist on that topic. # Get all messages for a given topic Source: https://docs.trieve.ai/api-reference/message/get-all-messages-for-a-given-topic get /api/messages/{messages_topic_id} If the topic is a RAG topic then the response will include Chunks first on each message. The structure will look like `[chunks]||mesage`. See docs.trieve.ai for more information. # Get tool function parameters Source: https://docs.trieve.ai/api-reference/message/get-tool-function-parameters post /api/message/get_tool_function_params This endpoint will generate the parameters for a tool function based on the user's message and image URL provided in the request body. The response will include the parameters for the tool function as a JSON object. # Regenerate message Source: https://docs.trieve.ai/api-reference/message/regenerate-message-1 patch /api/message Regenerate the assistant response to the last user message of a topic. This will delete the last message and replace it with a new message. The response will include Chunks first on the stream if the topic is using RAG. The structure will look like `[chunks]||mesage`. See docs.trieve.ai for more information. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Transcribe Audio Source: https://docs.trieve.ai/api-reference/message/transcribe-audio post /api/message/transcribe_audio Uses `whisper-1` to transcribe an audio file passed in as a base64 encoded string. # Get Prometheus Metrics Source: https://docs.trieve.ai/api-reference/metrics/get-prometheus-metrics post /metrics This route allows you to view the number of items in each queue in the Prometheus format. # Create Organization Source: https://docs.trieve.ai/api-reference/organization/create-organization post /api/organization Create a new organization. The auth'ed user who creates the organization will be the default owner of the organization. # Create Organization Api Key Source: https://docs.trieve.ai/api-reference/organization/create-organization-api-key post /api/organization/api_key Create a new api key for the organization. Successful response will contain the newly created api key. # Delete Organization Source: https://docs.trieve.ai/api-reference/organization/delete-organization delete /api/organization/{organization_id} Delete an organization by its id. The auth'ed user must be an owner of the organization to delete it. # Delete Organization Api Key Source: https://docs.trieve.ai/api-reference/organization/delete-organization-api-key delete /api/organization/api_key/{api_key_id} Delete an api key for the auth'ed organization. # Get Organization Source: https://docs.trieve.ai/api-reference/organization/get-organization get /api/organization/{organization_id} Fetch the details of an organization by its id. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get Organization Api Keys Source: https://docs.trieve.ai/api-reference/organization/get-organization-api-keys get /api/organization/api_key Get the api keys which belong to the organization. The actual api key values are not returned, only the ids, names, and creation dates. # null Source: https://docs.trieve.ai/api-reference/organization/get-organization-usage get /api/organization/usage/{organization_id} # Get Organization Users Source: https://docs.trieve.ai/api-reference/organization/get-organization-users get /api/organization/users/{organization_id} Fetch the users of an organization by its id. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Update All Dataset Configurations Source: https://docs.trieve.ai/api-reference/organization/update-all-dataset-configurations post /api/organization/update_dataset_configs Update the configurations for all datasets in an organization. Only the specified keys in the configuration object will be changed per dataset such that you can preserve dataset unique values. Auth'ed user or api key must have an owner role for the specified organization. # Update Organization Source: https://docs.trieve.ai/api-reference/organization/update-organization put /api/organization Update an organization. Only the owner of the organization can update it. # null Source: https://docs.trieve.ai/api-reference/public/get-apipublic_page get /api/public_page/{dataset_id} # Send a Shopify user event Source: https://docs.trieve.ai/api-reference/public/send-a-shopify-user-event post /api/shopify/user_event This endpoint is used to send a Shopify user event to all users in the organization. # Cancel Subscription Source: https://docs.trieve.ai/api-reference/stripe/cancel-subscription delete /api/stripe/subscription/{subscription_id} Cancel a subscription by its id # Checkout Source: https://docs.trieve.ai/api-reference/stripe/checkout get /api/stripe/payment_link/{plan_id}/{organization_id} Get a 303 SeeOther redirect link to the stripe checkout page for the plan and organization # Update Payment Method Source: https://docs.trieve.ai/api-reference/stripe/create-checkout-session-setup post /api/stripe/checkout/setup/{organization_id} Update a your payment method to a new one # Estimate Bill From Range Source: https://docs.trieve.ai/api-reference/stripe/estimate-bill-from-range get /api/stripe/estimate_bill/{plan_id} Return the amount you will be billed from a date range if you were on usage based pricing # Get All Invoices Source: https://docs.trieve.ai/api-reference/stripe/get-all-invoices get /api/stripe/invoices/{organization_id} Get a list of all invoices # Get All Plans Source: https://docs.trieve.ai/api-reference/stripe/get-all-plans get /api/stripe/plans Get a list of all plans # Get All Usage Plans Source: https://docs.trieve.ai/api-reference/stripe/get-all-usage-plans get /api/stripe/usage_plans Get a list of all usage_based plans # Update Payment Method Source: https://docs.trieve.ai/api-reference/stripe/update-payment-method post /api/stripe/checkout/setup/{organization_id} Update a your payment method to a new one # Update Subscription Plan Source: https://docs.trieve.ai/api-reference/stripe/update-subscription-plan patch /api/stripe/subscription_plan/{subscription_id}/{plan_id} Update a subscription to a new plan # Clone Topic Source: https://docs.trieve.ai/api-reference/topic/clone-topic post /api/topic/clone Create a new chat topic from a `topic_id`. The new topic will be attched to the owner_id and act as a coordinator for conversation message history of gen-AI chat sessions. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Create Topic Source: https://docs.trieve.ai/api-reference/topic/create-topic post /api/topic Create a new chat topic. Topics are attached to a owner_id's and act as a coordinator for conversation message history of gen-AI chat sessions. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Delete Topic Source: https://docs.trieve.ai/api-reference/topic/delete-topic delete /api/topic/{topic_id} Delete an existing chat topic. When a topic is deleted, all associated chat messages are also deleted. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Get All Topics for Owner ID Source: https://docs.trieve.ai/api-reference/topic/get-all-topics-for-owner-id get /api/topic/owner/{owner_id} Get all topics belonging to an arbitary owner_id. This is useful for managing message history and chat sessions. It is common to use a browser fingerprint or your user's id as the owner_id. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Update Topic Source: https://docs.trieve.ai/api-reference/topic/update-topic put /api/topic Update an existing chat topic. Currently, only the name of the topic can be updated. Auth'ed user or api key must have an admin or owner role for the specified dataset's organization. # Delete User Api Key Source: https://docs.trieve.ai/api-reference/user/delete-user-api-key delete /api/user/api_key/{api_key_id} Delete an api key for the auth'ed user. # Get User Api Keys Source: https://docs.trieve.ai/api-reference/user/get-user-api-keys get /api/user/api_key Get the api keys which belong to the auth'ed user. The actual api key values are not returned, only the ids, names, and creation dates. # Update User Org Role Source: https://docs.trieve.ai/api-reference/user/update-user-org-role put /api/user Update a user's information for the org specified via header. If the user_id is not provided, the auth'ed user will be updated. If the user_id is provided, the role of the auth'ed user or api key must be an admin (1) or owner (2) of the organization. # Build Search for Ecommerce Source: https://docs.trieve.ai/examples/ecommerce Learn how to build a job board with Trieve # Build a product inventory with Trieve In this guide we are going to use the Amazon Berkeley objects dataset ([https://amazon-berkeley-objects.s3.amazonaws.com/index.html](https://amazon-berkeley-objects.s3.amazonaws.com/index.html)) to build an inventory to search with Trieve ## Cloning the example repository ```sh theme={null} git clone https://github.com/devflowinc/examples cd examples/amazon-berkeley-dataset ``` ## Getting the data ### CSV Metadata ```sh theme={null} wget https://amazon-berkeley-objects.s3.amazonaws.com/archives/abo-listings.tar tar xvf abo-listings.tar gzip -d listings/metadata/listings_*.json.gz ``` This should extract multiple lines that have ### Image CSV Data ```sh theme={null} wget https://amazon-berkeley-objects.s3.amazonaws.com/images/metadata/images.csv.gz gzip -d images.csv.gz ``` The `images.csv` is formatted as the following. ```csv theme={null} image_id,height,width,path 010-mllS7JL,106,106,14/14fe8812.jpg 01dkn0Gyx0L,122,122,da/daab0cad.jpg 01sUPg0387L,111,111,d2/d2daaae9.jpg ``` ## Running the ingest script To run the ingest script you first need to add or modify a .env file with ```sh theme={null} TRIEVE_API_KEY="tr-********************************" TRIEVE_DATASET_ID="********-****-****-****-************" ``` Then run the ingest script ```sh theme={null} bun index.ts ``` A breakdown of the script is below ### Types The first step is to define all the types ```ts theme={null} interface Chunk { chunk_html: string; link: string; tracking_id: string; tag_set: string[]; metadata: { [key: string]: string; }; time_stamp: string; upsert_by_tracking_id: boolean; } interface LanguageTaggedValue { language_tag: string; value: string; } interface ItemDimension { unit: string; value: number; } interface ItemDimensions { height: ItemDimension; length: ItemDimension; width: ItemDimension; } interface Node { node_id: number; node_name: string; } interface Item { brand: LanguageTaggedValue[]; bullet_point: LanguageTaggedValue[]; color: LanguageTaggedValue[]; item_id: string; price?: number; image_url?: string; item_name: LanguageTaggedValue[]; model_name: LanguageTaggedValue[]; model_number: { value: string }[]; model_year: { value: number }[]; product_type: { value: string }[]; style: LanguageTaggedValue[]; main_image_id: string; other_image_id: string[]; item_keywords: LanguageTaggedValue[]; country: string; marketplace: string; domain_name: string; node: Node[]; item_dimensions?: ItemDimensions; } ``` ### Image Hashmap Since images and json bodies are in separate files, we parse the images into a hashmap ```js theme={null} // Function to parse CSV data and store it in a hashmap function parseCSV(csvData: string): Map { const lines = csvData.split("\n"); const hashmap = new Map(); for (let i = 1; i < lines.length; i++) { const values = lines[i].trim().split(","); let imageId: string | null = null; let path: string | null = null; if (values[0]) { imageId = values[0].trim(); } if (values[3]) { path = values[3].trim(); } if (imageId != null && path != null) { hashmap.set(imageId, path); } } return hashmap; } // Read CSV file const imageFilePath = "./images.csv"; const csvImageData = await Bun.file(imageFilePath).text(); let imageHashMap = parseCSV(csvImageData); if (imageHashMap == null) { console.log("Failed to load image csv data"); } ``` ### Processing a Singular Chunk For each row of json we construct a `Chunk` object, each of which can be pushed into Trieve separately or in bulk. We use the `imageHashMap` object constructed from before to get the `image_url` for each. ```ts theme={null} function processLine(line: string) { const item: Item = JSON.parse(line); let image_url; let imageId = item.main_image_id == "" ? null : item.main_image_id; let imagePath = imageHashMap.get(imageId); let price = [10, 25, 50, 100, 500, 1000][Math.floor(Math.random() * 6)]; if (imagePath != null) { image_url = `https://amazon-berkeley-objects.s3.amazonaws.com/images/small/${imagePath}`; } let searchableString = ""; // Safely adds a field to the searchable string if it exists const addField = ( field: string | undefined, prefix: string = "", postfix: string = "\n" ) => { if (field) { searchableString += `${prefix}${field}${postfix}`; } }; addField(price?.toString(), "Price: $"); // Process each field with a safe check and appropriate formatting addField(item.brand?.[0]?.value, "Brand: "); addField(item.item_name?.[0]?.value, "Product Name: "); item.bullet_point?.forEach((bp) => addField(bp.value, "", ";")); addField(item.color?.[0]?.value, "Color: "); addField(item.model_name?.[0]?.value, "Model Name: "); addField(item.model_number?.[0]?.value, "Model Number: "); // For numerical fields, ensure existence before converting to string if (item.model_year?.[0]?.value !== undefined) { addField(item.model_year[0].value.toString(), "Model Year: "); } addField(item.product_type?.[0]?.value, "Product Type: "); addField(item.style?.[0]?.value, "Style: "); item.item_keywords?.forEach((kw) => addField(kw.value, "", ";")); addField(item.country, "Country: "); addField(item.marketplace, "Marketplace: "); addField(item.domain_name, "Domain: "); const metadata: Partial = { ...item }; metadata.image_url = image_url; metadata.price = price; const chunkData: Chunk = { chunk_html: searchableString.trim(), link: `https://${item.domain_name}/dp/${item.item_id}`, tracking_id: item.item_id, tag_set: item.item_keywords?.map((kw) => kw.value), metadata, image_urls: [ image_url ?? "" ], upsert_by_tracking_id: true, }; return chunkData; } ``` ### Iterating and sending chunks to Trieve The last step is we iterate through all the json files, process each line into a Chunk object. We use the `/chunk` endpoint to create chunks, in this case we are sending chunks in a batch size of 120 which is the max for this route. ```ts theme={null} const directoryPath = "./listings/metadata/"; const files = await readdir(directoryPath) for (const file of files) { const fullPath = resolve(directoryPath + file); const fileStream = createReadStream(fullPath); const rl = readline.createInterface({ input: fileStream, crlfDelay: Infinity, }); const items: Chunk[] = []; for await (const line of rl) { try { const chunkData: Chunk = processLine(line); const options = { method: "POST", headers: { "TR-Dataset": trieveDatasetId, Authorization: trieveApiKey, "Content-Type": "application/json", }, body: JSON.stringify(chunk), }; await fetch("https://api.trieve.ai/api/chunk", options); } catch (error) { console.error("Error parsing JSON from line:", error); } } const batchSize = 120; const chunkedItems: Chunk[][] = []; for (let i = 0; i < items.length; i += batchSize) { const chunk = items.slice(i, i + batchSize); chunkedItems.push(chunk); } for (const chunk of chunkedItems) { try { console.log(`Creating chunk`); await chunkApi.createChunk(trieveDatasetId, chunk); } catch (error) { console.error(`Failed to create chunk`); console.error(error); } } } ``` Checkout the api reference for the [upload API](/api-reference/chunk/create-or-upsert-chunk-or-chunks) to view all of the parameters that can be passed in with the chunk. # Build Search for a Job Board Source: https://docs.trieve.ai/examples/job-board Learn how to build a job board with Trieve This guide will help you set up and run a demo using a sample job listing dataset on the Trieve platform. Follow the steps below to get everything working. ## Steps to Set Up the Demo 1. **Clone the repository:** ```bash theme={null} git clone https://github.com/devflowinc/examples ``` 2. **Install the required dependencies:** This demo uses Bun as a runtime. Install Bun by running: ```bash theme={null} curl -fsSL https://bun.sh/install | bash ``` Install the dependencies by running: ```bash theme={null} cd job-board-example bun install ``` 3. **Create an account on Trieve:** Go to [Trieve Dashboard](https://dashboard.trieve.ai) and create an account. 4. **Create a new dataset and copy the dataset ID.** You can either create a dataset on the Trieve Dashboard or use our [CLI](https://github.com/devflowinc/trieve-CLI) 5. **Create a new API key and copy the API key.** 6. **Create an `.env` file in the root of the project and add the following environment variables:** ```bash theme={null} TRIEVE_API_KEY=your-api-key TRIEVE_DATASET_ID=your-dataset-id ``` 7. **Download the dataset** ```bash theme={null} wget -O jobs.csv https://query.data.world/s/pgpakbonuiwfiltmu443p474knfork?dws=00000 ``` 8. **Run the demo:** ```bash theme={null} bun upload-indeed-listings.ts ``` 9. **View the results** Head over to [https://search.trieve.ai](https://search.trieve.ai) and select your dataset. Try out a few sample queries like `software engineer at google` ![title](https://cdn.trieve.ai/docs/indeed_google.png) or `show me some listings knowing I have experience as a mechanic` ![title](https://cdn.trieve.ai/docs/indeed_mechanic.png) ## Breakdown of the script ### Creating a data model to represent the data When we receive the CSV data, we need to turn it into a data model to be able to effectively pull the data out of it. The data model for this CSV looks like: ```javascript theme={null} interface JobData { "Job Title": string; "Job Description"?: string; Location: string; City: string; State: string; Country: string; "Zip Code": string; "Apply Url": string; "Company Name": string; "Employer Logo": string; Companydescription: string; "Employer Location": string; "Employer City": string; "Employer State": string; "Employer Country": string; "Employer Zip Code": string; "Uniq Id": string; "Crawl Timestamp": string; } ``` ### Creating the Searchable Chunk HTML The Chunk HTML should contain the data that we want to be searchable. In this case, we include the job title, description and other text fields that provide more context to the embedding model so that it can generate a better embedding to represent the job. It is typically best practice to put a `\n\n` between each field to separate them so that the model can differentiate between them. The following function transforms job data into the searchable chunk HTML: ```javascript theme={null} function jobToSearchableString(job) { let searchableString = ""; const addField = (field, prefix = "", postfix = "\n\n") => { if (field) { searchableString += `${prefix}${field}${postfix}`; } }; addField(job["Job Title"], "Job Title: "); addField(job["Job Description"], "Job Description: "); addField(job["Location"], "Location: "); addField(job["City"], "City: "); addField(job["State"], "State: "); addField(job["Country"], "Country: "); addField(job["Company Name"], "Company Name: "); addField(job["Companydescription"], "Company Description: "); addField(job["Employer Location"], "Employer Location: "); addField(job["Employer City"], "Employer City: "); addField(job["Employer State"], "Employer State: "); addField(job["Employer Country"], "Employer Country: "); return searchableString.trim(); } ``` ### Creating a Chunk When possible, use our bulk upload API to upload data in chunks of up to 120 to take advantage of the speed of the API. To efficiently upload data, we divide it into chunks. Here’s the code snippet for creating a chunk with the data: ```javascript theme={null} const createChunkData = items.map((item) => ({ // Searchable Data chunk_html: jobToSearchableString(item), // Link to job listing link: item["Apply Url"] ?? "", // ID to be able to differentiate this chunk and track it outside of Trieve // You can refer to the chunk using this ID so that you don't have to keep // track of the Trieve ID tracking_id: item["Uniq Id"] ?? "", // Tags to associate with chunk tag_set: [ item["Employer City"] ?? "", item["Employer State"] ?? "", item["Employer Country"] ?? "", ], // Other metadata of the chunk metadata: extractMetadata(item), // Timestamp to associate with chunk time_stamp: new Date(item["Crawl Timestamp"]).toISOString() ?? null, // With this param set to true, the API will not error if a chunk // with the same tracking ID is passed. // Instead it will just update the existing chunk. upsert_by_tracking_id: true, })); // Define the chunk structure interface Chunk { chunk_html: string; link: string; tracking_id: string; tag_set: string[]; metadata: { [key: string]: string; }; time_stamp: string; upsert_by_tracking_id: boolean; } const chunkSize = 50; const chunkedItems: Chunk[][] = []; // We combine chunks into groups of 50 for (let i = 0; i < createChunkData.length; i += chunkSize) { const chunk = createChunkData.slice(i, i + chunkSize); chunkedItems.push(chunk); } for (const chunk of chunkedItems) { try { console.log(`Creating chunk`); const options = { method: "POST", headers: { "TR-Dataset": trieveDatasetId, Authorization: trieveApiKey, "Content-Type": "application/json", }, body: JSON.stringify(chunk), }; await fetch("https://api.trieve.ai/api/chunk", options); } catch (error) { console.error(`Failed to create chunk`); console.error(error); } } ``` Checkout the api reference for the [upload API](/api-reference/chunk/create-or-upsert-chunk-or-chunks) to view all of the parameters that can be passed in with the chunk. By following this guide, you should be able to set up and run the demo successfully. If you encounter any issues, refer back to the steps and ensure all configurations are correctly set up. # Introduction Source: https://docs.trieve.ai/getting-started/introduction Trieve is an API for building search, and RAG experiences. ## Quick Start Guide Check out the API Reference to see all of the available endpoints and options for integrating Trieve into your application. Get started with Trieve quickly. Learn how to build a search experience for a job board using Trieve. Learn how to build a search experience for an ecommerce platform using Trieve. Search and RAG are powerful primitives, but can take months to implement and deploy in production. Trieve abstracts away the complexity of these tasks, allowing you to focus on building your product. With Trieve, you can build and deploy a ChatGPT like experience or a search engine on your data in a matter of hours. [Trieve Cloud](https://trieve.ai) has been sunset as of Nov 1st 2025. To use Trieve refer to the [Self Hosting](/self-hosting/docker-compose) guide. ### Demos of Trieve in Action ![](https://trieve.b-cdn.net/docs/hackernews-search.png) ![](https://trieve.b-cdn.net/docs/yc-search.png) * [YCombinator Companies Search](https://yc.trieve.ai) - [📝 Building Search For the YC Company Directory With Trieve, Bun, and SolidJS ](https://trieve.ai/building-search-for-yc-company-directory/) * [SteamDB Search](https://steamdb.trieve.ai) * [📝 Introducing Yet Another Steam Recommendation Engine!](https://trieve.ai/steam-recommendations) * **This documentation site itself!** (and all Mintlify documentation sites) * [📝 Success Story: Mintlify](https://trieve.ai/success-story-mintlify/) If you are interested in a managed instance and/or SLA, reach out to us by emailing [humans@trieve.ai](mailto:humans@trieve.ai). To engage with the Trieve community, join our our [Discord](https://discord.com/invite/E9sPRZqpDT). ### Client Libraries * [Trieve TypeScript/JavaScript Client](https://ts-sdk.trieve.ai) ### System Diagram Services in blue are horizontally scalable and can be run in a distributed manner. ![trieve-system-diagram](https://cdn.trieve.ai/docs/Trieve-System-Diagram.webp) # Quickstart Source: https://docs.trieve.ai/getting-started/quickstart Quickly start building search, recommendations, and RAG for your application with Trieve Head over to the [Trieve Dashboard](https://dashboard.trieve.ai) to create an account and dataset. With a free account, you can upload up to 1000 chunks and 512 MB of files. ![title](https://trieve.b-cdn.net/2024-08-07_14-19.png) You can interact with Trieve in two ways, either by using the UIs or by using the [API](/api-reference/). ### Using the UIs After creating your dataset on the dashboard, you will be brought to the dataset page. From there, you can upload your data by clicking on the "Upload File" button. ![title](https://trieve.b-cdn.net/2024-08-07_14-25.png) Our API will automatically chunk your data for you in the background, and upload the resulting chunks into your Trieve dataset. ### Using the API In order to interact with the API, you will need to grab your dataset ID and API key from the dashboard. You can find your dataset ID on the dataset page: ![title](https://trieve.b-cdn.net/2024-08-07_14-29.png) You can generate an API key on the API keys page: ![title](https://trieve.b-cdn.net/2024-08-07_14-30.png) You can then use the API to upload your data, by providing the dataset ID and API key in the headers of your request. ``` { "Authorization": "", "TR-Dataset": "" } ``` You can then use our [upload file route](/api-reference/file/upload-file) to upload your files and start interacting with your data. If you would like to manually pass chunks to Trieve, you can use the [create chunks route](/api-reference/chunk/create-or-upsert-chunk-or-chunks) You can now interact with your data in a variety of ways. You can search your data, get recommendations, or use RAG to generate responses to user queries using our UIs or API. ### Searching You can try searching on the sample data you added by visiting [search.trieve.ai](https://search.trieve.ai) and entering a query in the search bar. Make sure to select the dataset you uploaded the sample data to using the selector dropdown on the top left. ![example-search-ui-query](https://cdn.trieve.ai/docs/example-search-ui-query.png) ### RAG (Retrieval Augmented Generation) You can try RAG on the sample data you added by visiting [chat.trieve.ai](https://chat.trieve.ai) and entering a query in the chat box. ![trieve-rag-example](https://cdn.trieve.ai/docs/trieve-rag-example.png) # Screenshots Source: https://docs.trieve.ai/getting-started/screenshots Screenshots of Trieve in action across various use cases. ![title](https://trieve.b-cdn.net/2024-08-08_15-22.png) ![title](https://trieve.b-cdn.net/2024-08-08_15-22_1.png) ![title](https://trieve.b-cdn.net/2024-08-08_15-23.png) ![title](https://trieve.b-cdn.net/2024-08-08_15-24.png) ![title](https://trieve.b-cdn.net/2024-08-08_15-24_1.png) # Trieve Primitives Source: https://docs.trieve.ai/getting-started/trieve-primitives Basic terms and concepts used commonly within the Trieve ecosystem. * **Chunk**: A chunk is a piece of data that is uploaded to Trieve. It is the smallest unit of data that can be searched, recommended, or used in RAG. Chunks are typically created by chunking a larger piece of data into smaller pieces. For example, a document can be chunked into paragraphs, sentences, or even words depending on the use case. * **Tag Set**: A tag set is a collection of tags that can be associated with a chunk. Tags can be used to categorize or filter chunks based on specific criteria. For example, a chunk representing a job posting could be tagged with the job title, location, company name, etc. * **Metadata**: Metadata is a json object that can be associated with a chunk. Metadata can be used to store additional information about a chunk, such as the source of the data, creation date, author, etc. Although filtering by metadata is supported, it is not recommended if latency is a concern. * **Groups**: Groups are a way to associate related chunks together. Chunks that belong to the same group are considered to be related in some way. For example, chunks representing different sections of a document can be grouped together to indicate that they are part of the same document. You can perform queries within a group or across groups. * **Dataset**: A dataset is a collection of chunks. Datasets are created in the Trieve dashboard and are used to organize and manage chunks. Datasets can be used to search, recommend, or generate responses using RAG. * **Search**: Search is the process of finding relevant chunks in a dataset based on a query. Trieve provides a search API that allows you to search for chunks based on text similarity. * **Recommendations**: Recommendations are a list of chunks that are similar to a given chunk. Trieve provides a recommendation API that allows you to get recommendations for a chunk based on text similarity. * **RAG (Retrieval Augmented Generation)**: RAG is a technique that combines search and generation to generate responses to user queries. Trieve provides a RAG API that allows you to generate responses to user queries based on the content of your dataset. # RAG with Trieve Source: https://docs.trieve.ai/guides/RAG-with-trieve Learn how to chat with your data with Trieve ## Overview We provide functionality to do conversational search over your data using the Retrieve and Generate (RAG) model. This model allows you to chat with your data in a conversational manner, asking questions and getting responses in a natural language format. This is mainly done through our [RAG on specified chunks route](/api-reference/chunk/rag-on-specified-chunks) and [RAG route](/api-reference/message/create-message). ## RAG on specified chunks This strategy allows you to specify the chunks that you want to chat with. This is useful when you want to control which exact chunks are used in the conversation. [API Reference](/api-reference/chunk/rag-on-specified-chunks) ### Important Parameters * `prompt`: This is the user's question or response to the LLM * `chunk_ids`: This is the list of chunk IDs that you want to chat with * `prev_messages`: This is the list of previous messages in the conversation. This is useful when you want to continue a conversation from a previous point. The `prev_messages` field's format is a list of dictionaries, where each dictionary has the following fields: * `content`: The message that was sent * `role`: The sender of the message. This can be either `user` or `assistant` ## RAG over all chunks This strategy allows you to chat with all the chunks in your dataset. This is useful when you want to chat with all the chunks in your dataset, and have the model do the searching for you. 1. To use the `RAG` route, you need to instantiate a topic with Trieve to store all of the messages in the conversation. This is done through the [create topic route](/api-reference/topic/create-topic). ### Important Parameters * `first_user_message`: This is the first message that the user sends to the model. This is used to generate a name for the topic. If not provided, the model will use the name from the name field. * `name`: This is the name of the topic. * `owner_id`: This is the ID of the user that owns the topic. This is typically a browser fingerprint or your user's id. It is used to group topics together for a user. 2. You can then use the `topic_id` that is returned from the `create-topic` route to use the [create message route](/api-reference/message/create-message). ### Important Parameters * `topic_id`: This is the ID of the topic that you want to chat with * `new_message_content`: This is the new message that the user sends to the model * `filters`: This is the filters that you want to apply to the chunks that are searched. This is useful when you want to constrain the search to a specific subset of your data. View the explanation on how to filter in the [searching with Trieve guide](/guides/searching-with-trieve#filtering). # Analytics with Trieve Source: https://docs.trieve.ai/guides/analytics-quickstart Learn how to get started with Trieve Analytics ## Overview Trieve Analytics is a powerful tool that allows you to analyze your queries and gain insights from them. ## Setting Up Trieve Analytics Trieve automatically collects all of the searches and RAG chats that are made in your dataset. You don't have to worry about manually tracking them. ## Automatically Tracked Analytics ## Search Analytics We provide extensive analytics on your searches all of which can be found in the [search analytics route](api-reference/analytics/get-search-analytics). These analytics include: * **Head Queries**: Head queries is a collection of the most common queries. * **Low Confidence Queries**: The queries where all your results have scores, which can be useful for improving your search results. * **No Results Queries**: This table shows the queries that returned no results, which can be useful for improving your search results. * **Popular Filters**: This table shows the most popular filters used in searches. * **Latency / time**: This graph shows the average latency of searches over time. * **Searches / time**: This graph shows the number of searches made over time. You can see all of this data on the dashboard in the analytics section, as well as be able to export them to a CSV so that you can analyze them in your own tools. ![](https://trieve.b-cdn.net/Screenshot%202024-10-01%20at%201.46.51%E2%80%AFPM.png) ## RAG Chat Analytics We provide extensive analytics on your RAG chats all of which can be found in the [RAG analytics route](api-reference/analytics/get-rag-analytics). These analytics include: * **Usage over time**: This graph shows the number of RAG chats made over time. * **All RAG Chats**: This table shows all of the RAG chats made in your dataset. You can see all of this data on the dashboard in the analytics section, as well as be able to export them to a CSV so that you can analyze them in your own tools. ![](https://trieve.b-cdn.net/Screenshot%202024-10-01%20at%202.06.27%E2%80%AFPM.png) ## Custom Metrics ### 1. Getting the Request ID All searches, recommendations, and chats return a `requestID`. With this you can track: * Clicks * user ratings (on any scale you want) * Add to Cart's * User Views * Purchase's forwarding this `query_id` to any custom #### Request ID from Searches All Searches Return an `id`, this `id` is your Request ID ```json search_response.json {5} theme={null} { "chunks" : [ // ... Your search response ], "id": "28f37011-179e-4927-9138-771132d0b6c3" // Request ID } ``` #### Request ID from Recommendations Calls to `/api/chunk/recommend` return an `id` ```json recommend_response.json {5} theme={null} { "chunks" : [ // ... Your Chunk Data ], "id": "28f37011-179e-4927-9138-771132d0b6c3" // Request ID // ... other data } ``` #### Request ID from LLM Messages Calls to `/api/message` return a streaming response so the id is not located in the payload, the id is instead placed on the header `TR-QueryID`. ### 2. Enriching Events #### Track Click Through Rate data Send Click-Through Rate data to Trieve using the [send CTR data route](/api-reference/analytics/send-ctr-data). Referencing your search `id` from the previous step. `chunk_id` in this case is the chunk the user clicked on from the search ```json curl theme={null} curl -X POST https://api.trieve.ai/api/analytics/ctr \ -H "TR-Dataset: YOUR_DATASET_ID" \ -H "Authorization: " \ -d '{ "ctr_type": "search | rag", "clicked_chunk_id": "", "position": 1, "request_id": "" }' ``` ```ts TS SDK theme={null} import { TrieveSDK } from "trieve-ts-sdk"; let search_id = ""; // Set your search_id here let chunk_id = ""; // The chunk clicked let api_key = "tr-**************"; // Your api key here export const trieve = new TrieveSDK({ apiKey: "tr-********************************", datasetId: api_key, }); const data = await trieve.sendCTRData({ ctr_type: "search | rag", clicked_chunk_id: chunk_id, position: 1, request_id: search_id, }); ``` #### Allow the user to rate the search You can also send user feedback to the Trieve API using the [rate search query route](/api-reference/analytics/rate-search) or [rate RAG query route](/api-reference/analytics/rate-rag). The `rating` parameter can be any number of your choosing. Feel free to make your own rating scales ```json curl theme={null} curl -X POST https://api.trieve.ai/api/analytics/rate-search \ -H "TR-Dataset: YOUR_DATASET_ID" \ -H "Authorization: YOUR_API_KEY" \ -d '{ "query_id": "", "rating": 5, "note": "Great results!" }' ``` ```ts TS SDK theme={null} import { TrieveSDK } from "trieve-ts-sdk"; let search_id = ""; // Set your search_id here let api_key = "tr-**************"; // Your api key here export const trieve = new TrieveSDK({ apiKey: "tr-********************************", datasetId: "", }); const data = await trieve.rateSearchQuery({ query_id: search_id, rating: 5, note: "Great results!", }); ``` #### Custom event types You can send custom events to the Trieve API using the [send event route](/api-reference/analytics/send-user-event-data). Referencing your search `id` from the previous step ```json curl theme={null} curl -X POST https://api.trieve.ai/api/analytics/event \ -H "TR-Dataset: YOUR_DATASET_ID" \ -H "Authorization: YOUR_API_KEY" \ -d '{ "event_type": "add_to_cart", "event_name": "Add to Cart", "items": [ "Cheesesticks", "Pizza", ], "request_id": "" }' ``` ```ts TS SDK theme={null} import { TrieveSDK } from "trieve-ts-sdk"; let search_id = ""; // Set your search_id here let api_key = "tr-**************"; // Your api key here export const trieve = new TrieveSDK({ apiKey: "tr-********************************", datasetId: api_key, }); const data = await trieve.sendEventData({ event_type: "add_to_cart", event_name: "Add to Cart", items: ["Cheesesticks", "Pizza"], request_id: search_id, }); ``` ### Querying Custom Metrics You can query this custom event data using the [get all events route](/api-reference/analytics/get-all-events). This will return all of the custom events that you have sent to the Trieve API. ```bash curl theme={null} curl --request POST \ --url https://api.trieve.ai/api/analytics/events \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --data '{ "filter": { "date_range": { "gt": "2021-08-10T00:00:00Z", "lt": "2021-08-11T00:00:00Z" }, "event_type": "", "is_conversion": true, "metadata_filter": "path = \"value\"", "user_id": "user1" } }' ``` # Uploading Chunks to Trieve Source: https://docs.trieve.ai/guides/create-chunk Learn how to upload your chunks to Trieve ## Overview We provide an easy interface for users to upload their chunks individually so that users should have control over how they chunk their data ## Uploading Individual Chunks to Trieve The main route that we expose to handle this functionality is our [create chunk route](/api-reference/chunk/create-or-upsert-chunk-or-chunks). ### Important Parameters * **`chunk_html`**: This is the content that will be embedded and made searchable. You can pass HTML here, and our service will automatically clean it for the embeddings. * **`group_ids` and `group_tracking_ids`**: These fields let you specify a group to associate the chunks with, useful for linking multiple chunks from one document. * **`link`, `location`, `tag_set`, `num_value`, and `time_stamp`**: These fields are indexed to enable fast filtering of chunks based on these attributes. * **`metadata`**: This field allows you to include any arbitrary metadata in the form of a JSON object with the chunk. For the best filtering performance, we recommend using the `link`, `location`, `tag_set`, `num_value` and `time_stamp` rather than the metadata field as there are dedicated indexes for these. The metadata has an index built on it, but it is only optimized for match queries * **`tracking_id`**: This field allows you to assign an arbitrary ID to the chunk, aiding in coordination with your database system. You can search for chunks using this ID. * **`weight`**: This field allows you to assign a weight to the chunks, which can influence the chunk's ranking within search results. This is similar to merchandising features on other platforms. * **`semantic_boost` and `fulltext_boost`**: These fields allow you to boost the relevance of the chunk in the search results, by aligning the chunk closer to a specified phrase. This is useful for ensuring that for longer chunks, you can manually specify the most important part of the chunk, and improve relevance for the query patterns of your users. ### Formatting chunk\_html Creating a good `chunk_html` is crucial for the quality of your search results. Here are some tips to help you create a good `chunk_html`: * **Include all relevant information**: Make sure to include all the information that you want to be searchable in the `chunk_html`. * **Use newlines**: Put semantically distinct information on separate lines to help the model understand the structure of the text and search quality for each field better * **Label each field**: If you have multiple fields in your `chunk_html`, label each field with a header to help the model better embed the text and improve search quality. Example of a good `chunk_html`: ```json theme={null} Price: $50\n Brand: AmazonBasics\n Product Name: AmazonBasics Gaming Office Chair, Racing Design, PU leather, White + Philips Hue Play White double pack Bundle\n Description: High-back gaming chair provides ultimate comfort and control, whether at work or play; black with red accents;Made of premium PU leather upholstery on top, PVC material along the sides and bottom, and a nylon base to provide sporty racer look;Custom fit with height-adjustable armrest and tilt control for easily reclining; headset pillow and lumbar cushion offer added support;Compact and versatile, perfect TV backlight;\n Color: White / Black\n Product Type: CHAIR\n Style: Office Chair + Hue Play\n Furniture;chair;desk accessories;desk chair;game chair;gaming;gaming chair;gaming office chair;home furniture;office and computer chair;office chair;chair;chairs ergonomic;computer chair;desk chair; ergonomic office chair;essential gaming chair;gaming chair office;gaming desk;modern desk chair;office chair; Country: GB\n Marketplace: Amazon\n Domain: amazon.co.uk ``` ### Example Create Chunk Request Whenever you make a request to the Trieve API, you need to include the `TR-Dataset` header with your dataset ID and the `Authorization` header with your API key. For more performant uploads, we recommend you batch your chunk uploads. You can upload multiple chunks in a single request by passing an array of chunks in the body of the request up to 120 at a time. ```json theme={null} POST /api/chunk Headers: { "TR-Dataset": "", "Authorization": "" } Body: { "chunk_html": "EcoFusion Technologies provides innovative, eco-friendly technology solutions. We specialize in renewable ...", "link": "https://example.com", "tracking_id": "134", "image_urls": ["https://example.com"], "tag_set": ["324", "product", "sale"], "metadata": { "phone_number": "000-000-0000", "price_range": "$21-$25", "reviews_count": "15", "address": { "city": "Austin", "country": "United States", "state": "Texas", "zip_code": "78781" } } } ``` # Crawling Websites with Trieve Source: https://docs.trieve.ai/guides/dataset-crawls Learn how to how use the crawl functionality within Trieve ## Overview We provide a simple way to crawl websites, Shopify stores, and Youtube channels and extract data from them. This guide will walk you through the steps to crawl websites with Trieve. ### Crawling a Website To start a crawl job, you need to make a POST request to the [create crawl](/api-reference/crawl/crawl-a-new-crawl-request) route. ### Important Parameters * `site_url`: The URL of the website you want to crawl. * `interval`: How often you want to crawl the website. * `webhook_url`: URL to call for each successful page scrape. By default, we will use a stripped down version of Firecrawl that we built called [firecrawl-simple](https://github.com/devflowinc/firecrawl-simple) in order to extract data from the url you provided, however you can also specify what kind of website you are crawling and we will tailor the crawling strategy for that. We support 2 other kinds of crawls: * `shopify`: This will crawl a Shopify store and extract product information. * `youtube`: This will crawl a Youtube channel and extract video information. To specify the kind of crawl you want to do, you can use the `crawl_type` parameter inside of the `scrape_options` field and specify the type of crawl you want to do. ```json theme={null} { "site_url": "https://www.example.com", "interval": "monthly", "webhook_url": "https://www.example.com/webhook", "scrape_options": { "crawl_type": "shopify" | "youtube" } } ``` There are some other parameters that will allow you customize the chunks that are created from your crawl job. You can find more information about these parameters in the [API reference](/api-reference/crawl/crawl-a-new-crawl-request). We allow for the site to be periodically rescraped as well to keep your data up to date by using the `interval` parameter and specifying how often to recrawl. We will by default crawl every month. ### Viewing Crawl Data You can view all the crawls that you have created for your dataset and their realtime statuses by using the [get all crawls](/api-reference/crawl/get-all-crawl-requests-for-a-dataset) route. # Using Groups with Trieve Source: https://docs.trieve.ai/guides/group-with-trieve Learn how to create and use groups with Trieve ## Overview We provide the ability to cluster and structure related chunks of data. This allows for specific search and recommendation paradigms -- constraining results to specific groups. ## Groups vs Tags Tags provide users the ability, through the `tag_set` attribute in chunks and groups, to categorize and label specific chunks, allowing for quick filtering. This lacks many of the higher level functionality around groups, which allow you to organize and search related content at a broader level. ## Creating Groups To create a group, use the [create or upsert group API](/api-reference/chunk-group/create-or-upsert-group-or-groups) route. ### Important Parameters * `name`: The name of the group. This does not need to be unique. * `description`: A description of the group and its corresponding chunks. * `metadata`: A JSON object containing any additional information to associate with chunks in this chunk group. * `tracking_id`: An optional, unique identifier to assign to this chunk group. * `tag_set`: A list of strings to categorize chunks within this chunk group. ```json curl theme={null} curl --request POST \ --url https://api.trieve.ai/api/chunk_group \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Dataset: ' \ --data '{ "name": "Example Group", "description": "This is an example group.", "metadata": {}, "tag_set": ["example", "guide", "group"], "tracking_id": "EXAMPLEGROUPGUIDE", }' ``` ### Accessing Chunks in a Group Trieve allows you to assign unique, arbitrary IDs to groups, through the `group_tracking_id` field, to sync with external systems. This also means that you can access and search groups using this field. For example, to fetch all chunks in a group by the `group_tracking_id`, use the [get chunks in group by tracking ID](/api-reference/chunk-group/get-chunks-in-group-by-tracking-id) route. ```json curl theme={null} curl --request GET \ // Replace {group_tracking_id} with the actual tracking ID of the chunk group --url https://api.trieve.ai/api/chunk_group/tracking_id/{group_tracking_id}/{page} \ --header 'Authorization: ' \ --header 'TR-Dataset: ' ``` ## Adding Chunks to Groups To add an existing chunk to a group, use the [add chunk to group](/api-reference/chunk-group/add-chunk-to-group) route. ```json curl theme={null} curl --request POST \ // Replace {group_id} with the Trieve generated ids for the group --url https://api.trieve.ai/api/chunk_group/chunk/{group_id} \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Dataset: ' \ --data '{ // ID of the chunk to add to the group "chunk_id": "********-****-****-****-********", // Tracking ID of the chunk to add to the group "chunk_tracking_id": "" }' ``` `group_id` is the **Trieve generated** ids for groups whereas the `group_tracking_id` is a **user-assigned** tracking\_ids for groups. Groups with `group_ids` must be created first and cannot be arbitrarily created. To assign a chunk to a group on creation, use the [create or upsert chunk or chunks](/api-reference/chunk/create-or-upsert-chunk-or-chunks) route and specify the `group_ids` or `group_tracking_ids`. Specify one or more existing group IDs that the chunks should be placed into, or provide group tracking IDs. If any IDs within the list of `group_tracking_ids` do not exist, new groups corresponding to those tracking IDs will be created. ```json curl theme={null} curl --request POST \ --url https://api.trieve.ai/api/chunk \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Dataset: ' \ --data '{ "chunk_html": "

Some HTML content

", // Enter a list of group IDs to associate with the chunk "group_ids": [ "********-****-****-****-************" ], // Enter a list of group tracking IDs to associate with the chunk // If a group_tracking_id does not exist, a corresponding group will be created "group_tracking_ids": [ "example_group_tracking_id" ], "metadata": { "key1": "value1", "key2": "value2" }, "link": "https://example.com", "tracking_id": "example_chunk_tracking_id" }' ``` ## View Groups in Search Playground Once you create a group and assign chunks, you can easily view and manage these groups through the [search playground](https://search.trieve.ai/). To view groups: 1. Navigate and login to the search playground at [https://search.trieve.ai/](https://search.trieve.ai/). 2. Click on `Groups` in the Navbar. 3. View group details (such as name, description, and date created). 4. Click on any group name to access its corresponding chunks. ## Searching Groups In addition to searching over independent chunks, we allow users to [search within groups](/api-reference/chunk-group/search-within-group) and [search over groups](/api-reference/chunk-group/search-over-groups). Search over groups returns the groups of chunks similar to the search query while search within groups restricts the search to a single group. Search *over* groups can be performed similar to search over chunks (checkout guide [here](/guides/searching-with-trieve)), with the addition of the `group_size` field to specify the number of chunks to fetch for each group. Similarly, to search *within* a group, use the `group_id` or `group_tracking_id` fields to specify which group you want to search in. ## Recommendations Groups Trieve also provides the ability to get recommended groups and recommended chunks within a group through the [get recommended groups](/api-reference/chunk-group/get-recommended-groups) and the [get recommended chunks](/api-reference/chunk/get-recommended-chunks) routes. Fetching recommended groups through the get recommended groups route is similiar to using the get recommended chunks route (checkout guide [here](/guides/recommending-with-trieve)), with the addition of: * `positive_group_tracking_ids`: the tracking IDs of groups that serve as positive examples when curating recommendations. * `negative_group_tracking_ids`: the tracking IDs of groups that serve as negative examples when getting recommendations. * `group_size`: the number of chunks to be fetched for each group. * `limit`: the max number of groups (rather than chunks) that should be returned. To get recommended chunks within a group, you can use the `get_recommended_chunks` route with a filter on `group_ids` or `group_tracking_ids`. # Setup Image Search Source: https://docs.trieve.ai/guides/image-search-setup Learn how to setup image search with Trieve ## Overview Trieve provides a simple way to integrate image search into your application. This guide will walk you through the steps to set up image search with Trieve. For a full implementation example, take a look at the way we implement image search in our [search component](https://github.com/devflowinc/trieve/blob/e716ab0b276ec056c1d2e554acf46301d27ba487/clients/search-component/src/TrieveModal/Search/UploadImage.tsx). ## Uploading an Image ### 1. Allow Users to Select an Image To let users upload an image, use an `` element with a file picker. Here’s a simple React component for an image uploader: ```typescript theme={null} import React, { useState, useRef } from 'react'; import { uploadFile, getPresignedUrl } from 'trieve-sdk'; export const ImageUploader = ({ trieveSDK }) => { const fileInputRef = useRef(null); const handleFileSelect = (event: React.ChangeEvent) => { const selectedFile = event.target.files?.[0]; if (selectedFile) { handleFileUpload(selectedFile); } }; return (
); }; ``` ### 2. Upload the Image to Trieve After the user selects an image, you can either upload it to Trieve and get a URL for it or use your own CDN. Here’s how you can do it with Trieve: ```typescript theme={null} const handleFileUpload = async (selectedFile: File) => { try { // Convert the file to a base64 string const base64 = await convertToBase64(selectedFile); // Upload the file to Trieve and get a unique file ID const fileId = await uploadFile(trieveSDK, selectedFile.name, base64, { create_chunks: false // Set to false to ensure that your file isn't indexed }); // Get a public URL for the uploaded image const imageUrl = await getPresignedUrl(trieveSDK, fileId); return imageUrl; } catch (error) { console.error('Image upload failed:', error); return null; } }; ``` *** ## Running an Image Search Once you have the uploaded image’s URL, you can use it to search for visually similar images. ### 1. Perform an Image Search Pass the image URL into a search query using Trieve’s SDK: ```typescript theme={null} const performImageSearch = async (imageUrl: string) => { try { const searchResults = await trieveSDK.search({ query: { image_url: imageUrl }, search_type: 'semantic', page_size: 10, // Adjust as needed }); return searchResults.chunks; } catch (error) { console.error('Image search failed:', error); return []; } }; ``` ### 2. Display Search Results Once you get the results, you can display them in your UI as thumbnails, links, or however you’d like. # Multi-tenant applications with Trieve Source: https://docs.trieve.ai/guides/multi-tenency Learn how to structure your data for multi-tenant applications with Trieve ## Overview We provide the ability to create and independently manage datasets for multi-tenant use cases. ## Creating a Dataset You should have one dataset per tenant or unique knowledge base. To create a dataset, use the [create dataset route](/api-reference/dataset/create-dataset). These datasets are kept isolated from each other and can be configured independently, making them perfect to represent each tenant within your application. Each dataset can have its own configurations, tags, and crawl options. ### Important parameters * `tracking_id`: A unique, optional tracking ID for the dataset that reflects the id of the tenant within your system. You can use this tracking id in the `TR-Dataset` header to specify the dataset for the request rather than the dataset id. * `crawl_options`: Provides the options to setup crawling to populate your dataset (e.g., include/exclude paths, tags, and more). * `dataset_name`: The name of the dataset. This must be a unique within the organization. * `server_configuration`: Provide the server configuration for the dataset such as RAG and system prompt, stop tokens, embedding models, and more. Example of creating a dataset through the API: ```json curl theme={null} curl --request POST \ --url https://api.trieve.ai/api/dataset \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "dataset_name": "New Dataset", // Replace with a name for your dataset "organization_id": "********-****-****-****-************", // Update with the desired configurations for your dataset "server_configuration": { "BM25_ENABLED": true, "DISTANCE_METRIC": "cosine", "EMBEDDING_MODEL_NAME": "text-embedding-3-small", "LLM_DEFAULT_MODEL": "gpt-3.5-turbo-1106", "RAG_PROMPT": "Use the following retrieved documents...", "SEMANTIC_ENABLED": true, "SYSTEM_PROMPT": "You are a helpful assistant", } }' ``` ## Update configuration across datasets You can also manage all of your dataset configurations at once using the the [update all dataset configurations route](/api-reference/organization/update-all-dataset-configurations). Use the `server_configuration` parameter to pass in a new configuration for all datasets in the organization. Example of updating all dataset configurations in an organization: ```json theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/update_dataset_configs \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ // Specify the organization whose dataset configurations you want to update --data '{ "organization_id": "********-****-****-****-************", "server_configuration": { // Keys that you would like to change } }' ``` Only the specified keys in the `server_configuration` object will be updated for each dataset, keeping the unique values for other fields unchanged. ## Creating an Organization It is very rare that you would need to create an organization through the API, but it is possible and explained below. The main route we use to expose this functionality is the [create organization route](/api-reference/organization/create-organization). Use the `name` parameter to pass a arbitrary, unique name which will be used to identify the organization. We recommend that you create seperate organizations for your main application as well as a staging environment for testing. Example of creating an organization on demand through the API: ```json theme={null} curl --request PUT \ --url https://api.trieve.ai/api/organization \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "name": "My Organization" // Replace with a unique name for your organization }' ``` # Recommending with Trieve Source: https://docs.trieve.ai/guides/recommending-with-trieve Learn how to recommend content with Trieve ## Overview We provide the ability to get recommended chunks similar to your data through the [get recommended chunks route](/api-reference/chunk/get-recommended-chunks) or through the [get recommended groups route](/api-reference/chunk-group/get-recommended-groups). ## Different Recommendation Types We have two different recommendation types: 1. **Recommend Chunks**: Recommends chunks that are similar to the provided examples. This is useful when your chunks are independent and do not need to be grouped together. 2. **Recommend Groups**: Recommends groups that are similar to the provided examples. It returns the groups and the top chunks within each group that are most similar to your input, providing better search quality for datasets with highly related chunks within groups. ## Different Recommendation Strategies We offer different recommendation strategies for users to choose from: 1. **`average_vector`**: Averages the vectors of the positive examples and finds the most similar chunks. * This strategy averages all the positive and negative examples separately, then combines them into a single vector. It then searches for chunks that are most similar to this combined vector. 2. **`best_score`**: Finds chunks that are most similar to any of the positive examples provided. * This strategy measures each example against every other example, then selects the best positive and best negative scores. ## Important Parameters * **`positive_examples` and `positive_tracking_ids`**: Provide the IDs of chunks of which similar ones will be recommended. * **`negative_examples` and `negative_tracking_ids`**: Provide the IDs of chunks of which dissimilar ones will be avoided. * **`recommendation_type`**: Can be either `semantic` or `fulltext`. * **Semantic**: Uses cosine distance. * **Fulltext**: Uses a SPLADE model. * **`strategy`**: Can be either `average_vector` or `best_score`. * **`slim_chunks`**: Reduces the size of the returned chunks by excluding unnecessary fields. * **`filters`**: Apply filters to get exactly the results you want. View the explanation on how to filter in the [searching with Trieve guide](/guides/searching-with-trieve#filtering). # Searching with Trieve Source: https://docs.trieve.ai/guides/searching-with-trieve Learn how to search over your data with Trieve ## Overview We provide the ability for you to search your data in a fast and performant manner. We have multiple search paradigms, which are exposed through the [search over chunks route](/api-reference/chunk/search), the [search within groups route](/api-reference/chunk-group/search-within-group), and the [search over groups route](/api-reference/chunk-group/search-over-groups). * `query`: The user query that is embedded and searched against the dataset. * `search_type`: Can be semantic, fulltext, or hybrid. Semantic: Uses cosine distance to determine the most relevant results. Fulltext: Uses a SPLADE model to find the most relevant results. Hybrid: Uses a reranker model that pulls one page of results from both fulltext and semantic searches to find the most relevant results. * `page`: The page of chunks to fetch. Pages are 1-indexed. * `page_size`: This lets you tune the number of results that are returned. * `highlight_results`: Enables subsentence highlighting of relevant portions of the text. * `slim_chunks`: Excludes chunk\_html from the returned results to reduce network bandwidth. Useful for large chunks. * `recency_bias`: A value from 0-1 that tunes how much the recency of chunks (based on the timestamp field) affects the ranking. * `sort_options`: Options on how to sort. * `filters`: Apply filters to get exactly the results you want. ## Search Modes Trieve offers 4 different types of search. ### Semantic Search Semantic search uses an embeddnig model to generate a query vector. Defaults to using cosine similarity and `jina-base-en` Trieve uses only the embedding model to select and rerank the results. This search\_type is `semantic`. ### Full Text search FullText search uses a SPLADE model to find the most relevant results to your given `query`. This search\_type is `fulltext`. ### BM25 BM25 is the classical type of search index, it uses the BM25 ranking function to determine the results that are most similar to your given `query`. This search\_type is `bm25`. ### Hybrid Hybrid search, does both a full text search, and semantic search. From those results it then uses a *reranker model* ( defaults to bge-reranker-large). This search\_type is `hybrid`. ## Search Paradigms We offer three different search strategies for you to choose from: 1. [**Search over chunks**](/api-reference/chunk/search): This strategy allows you to search all of your chunks independently. This is useful when your chunks are independent and do not need to be grouped together. 2. [**Search within groups**](/api-reference/chunk-group/search-within-group): This strategy lets you constrain your results to within a selected group. This is useful for searching distinct groups within your dataset independently. 3. [**Search over groups**](/api-reference/chunk-group/search-over-groups): This strategy allows you to search over the groups of chunks within your dataset. This returns the groups and the top chunks within each group that matched your query, providing better search quality for datasets with highly related chunks within groups. ## Search over chunks ```json theme={null} POST /api/chunk/search Headers: { "TR-Dataset": "", "Authorization": "tr-*******************" } Body: { "query": "How to search with Trieve", "search_type": "fulltext", "page": 1, "page_size": 10, "score_threshold": 0.5 } ``` ## Search within group ```json theme={null} POST /api/chunk_group/search Headers: { "TR-Dataset": "", "Authorization": "tr-*******************" } Body: { "group_tracking_id": "my-group-tracking-id", "query": "How to search with Trieve", "search_type": "fulltext", "page": 1, "page_size": 10, "score_threshold": 0.5 } ``` ## Search over groups ```json theme={null} POST api/chunk_group/group_oriented_search Headers: { "TR-Dataset": "", "Authorization": "tr-*******************" } Body: { "group_tracking_id": "my-group-tracking-id", "group_size": 5, "query": "How to search with Trieve", "search_type": "fulltext", "page": 1, "page_size": 10, "score_threshold": 0.5 } ``` You can use the search UI at [search.trieve.ai](https://search.trieve.ai) to A/B test which search method works best for you. ## Filters Trieve filters are structured around three clauses: * `must`: All filters within this clause must be matched to return the chunks. * `must_not`: All filters in this clause must not be matched to return the chunks. * `should`: Any of these conditions can be matched to return a chunk. Each clause contains a `field_condition`. * `range`: Match a number between a range of `lt`, `gt`, `lte` or `gte` * `match_all`: A list, every field must have have a match. * `match_any`: A list, at least 1 field must be present. * `date_range`: Match a date between a range of `lt`, `gt`, `lte` or `gte` * `geo_radius`: Match a radius based on a `center` and a `radius` * `boolean`: Matches if the field is true or false > Get chunks with both "CO" and "321" in their `tag_set` ```json theme={null} "filters": { "must": [ { "field": "tag_set", "match_all": ["CO", "321"] } ] } ``` > Get chunks with either "CO" OR "321" in their tag\_set: ```json theme={null} "filters": { "must": [ { "field": "tag_set", "match_any": ["CO", "321"] } ] } ``` > Get chunks that are tagged within a GEO radius ```json theme={null} "filters": { "must": [ { "field": "geo_radius", "geo_radius": { "center": { "lat": 20, "long": -30 }, "radius": 20, } } ] } ``` > Get chunks with neither "CO" nor "321" in their tag\_set: ```json theme={null} "filters": { "must_not": [ { "field": "tag_set", "match_all": ["CO", "321"] } ] } ``` > Get chunks that either don't have "CO" in their tag\_set or don't have "321" in their tag\_set: ```json theme={null} "filters": { "must_not": [ { "field": "tag_set", "match_any": ["CO", "321"] }, ] } ``` > Get chunks that either have "CO" in their tag\_set or "[http://example.com](http://example.com)" in their link: ```json theme={null} "filters": { "should": [ { "field": "tag_set", "match": ["CO"] }, { "field": "link", "match": ["http://example.com"] } ] } ``` > Get Chunks that have num\_value between 20 and 30 ```json theme={null} "filters": { "must": [ { "field": "num_value", "range": { "gte": 20.0, "lte": 30.0, "gt": 30.0, "lt": 20.0 } } ] } ``` ## Rerank By `rerank_type` can be either * `fulltext`: This will use the `fulltext` index to rerank the results, if `search_type` is `fulltext` then nothing different will happen. * `cross_encoder`: This will use the Reranker model that you predefined. By default `hybrid` search will use the `cross_encoder`. * `bm25`: This will use the `bm25` matching algorithm rerank the results, if `search_type` is `bm25` then nothing different will happen. * `semantic`: This will use the `semantic` vectors to rerank the results, if `search_type` is `semantic` then nothing different will happen. ```json theme={null} { "sort_options": { "sort_by" { "rerank_type": "fulltext" } } } ``` ## Multi Query MultiQuery provides a way to give multiple `query` objects with a given weight bias. To use the `multiquery`, instead of a single string, the `query` parameter receives a list of tuples, value 1 being the query and value 2 being a value on how important it is. As an example, search > Searching, but the search term of "iphone" and a color. ```json theme={null} "query": [ [ "Flagship", 2 ], [ "Red", 2 ], [ "Iphone mini", 10 ] ] ``` ## Customizing your search models Trieve offers many ways to customize your embedding models and reranker models. Different embedding models and different reranker models are better suited for different tasks. ### Embedding Models Trieve supports multiple embedding models that can be used to search over your data. You can specify the embedding model to use in the `server_configuration` field when creating a dataset. After creating a dataset, you cannot change the embedding model. If you need to change the embedding model, you must create a new dataset. ### Reranker models Trieve supports multiple reranker models that can be used to rerank the search results. Currently, Trieve supports the BAAI `bge-reranker-large` model, AIMon's `aimon-rerank` model, and Cohere's `rerank-v3.5` model. ![](https://cdn.trieve.ai/docs/reranker-model-edit.png) #### BAAI/bge-reranker-large `bge-reranker-large` is a model by the Beijing Academy of Artificial Intelligence (BAAI) and is hosted by Trieve. This model does not require any additional configuration and will be used by default on all `hybrid` searches. To manually select the `bge-reranker-large` as your reranker model, make a request to the [update dataset route](/api-reference/dataset/update-dataset-by-id-or-tracking-id) with the following parameters: ```json theme={null} curl --request POST \ --url https://api.trieve.ai/api/dataset \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "dataset_name": "New Dataset", // Replace with a name for your dataset "organization_id": "********-****-****-****-************", // Update with the desired configurations for your dataset "server_configuration": { "RERANKER_MODEL_NAME":"bge-reranker-large" } }' ``` #### AIMon's aimon-rerank `aimon-rerank` is a model hosted by [AIMon](https://docs.aimon.ai/). To use this model, you must provide the `aimon_api_key` in the `server_configuration` field when creating a dataset. To switch your reranker model to AIMon's `aimon-rerank`, make a request to the [update dataset route](/api-reference/dataset/update-dataset-by-id-or-tracking-id) with the following parameters: ```json theme={null} curl --request POST \ --url https://api.trieve.ai/api/dataset \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "dataset_name": "New Dataset", // Replace with a name for your dataset "organization_id": "********-****-****-****-************", // Update with the desired configurations for your dataset "server_configuration": { "RERANKER_BASE_URL":"https://pbe-api.aimon.ai/v1/rerank-icl", "RERANKER_MODEL_NAME":"aimon-rerank", "RERANKER_API_KEY":"", "AIMON_RERANKER_TASK_DEFINITION": "" // A task definition can be used to specify the domain of the context documents for AIMon reranker. // example of a task definition: "Your task is to grade the relevance of context document(s) in the domain of music and arts." } }' ``` #### Cohere's rerank-v3.5 `rerank-v3.5` is a model hosted by [Cohere](https://docs.cohere.com/cohere-documentation). To use this model, you must provide the `cohere_api_key` in the `server_configuration` field when creating a dataset. To switch your reranker model to Cohere's `rerank-v3.5`, make a request to the [update dataset route](/api-reference/dataset/update-dataset-by-id-or-tracking-id) with the following parameters: ```json theme={null} curl --request POST \ --url https://api.trieve.ai/api/dataset \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "dataset_name": "New Dataset", // Replace with a name for your dataset "organization_id": "********-****-****-****-************", // Update with the desired configurations for your dataset "server_configuration": { "RERANKER_BASE_URL":"https://api.cohere.com/v2", "RERANKER_MODEL_NAME":"rerank-v3.5" "RERANKER_API_KEY":"" } }' ``` # Uploading CSV and JSONL Files to Trieve Source: https://docs.trieve.ai/guides/uploading-csv-and-jsonl-files Learn how to upload bulk structured data to Trieve ## Overview Trieve allows you to upload structured data in CSV and JSONL formats. We automatically create chunks for each row in the file, allowing you to search and filter your data easily. ## Uploading a CSV or JSONL File to Trieve Since CSV and JSONL files can be large, our API allows you to provision a signed PUT URL to upload the file directly to our storage. Once the file is uploaded, Trieve will automatically process the file and create chunks for each row. ### Step 1: Request a Signed PUT URL Use the [/api/file/csv\_or\_jsonl](/api-reference/file/create-presigned-url-for-csv-jsonl) to acquire a signed PUT URL for your CSV or JSONL file from the Trieve API. This URL is valid for 24 hours and allows you to upload the file directly to our storage. You can leverage the `mappings` field to control how the columns in the CSV or fields in the JSONL file are mapped to the chunks created by Trieve. This is optional and can be used to ensure that the data is structured correctly. ```sh theme={null} curl -X POST "https://api.trieve.ai/api/file/csv_or_jsonl" \ -H "Content-Type: application/json" \ -H "TR-Dataset: " \ -H "Authorization: " \ -d '{ "description": "This is an example file containing information about titanic passengers.", "file_name": "titantic.csv", "mappings": [ { "csv_jsonl_field": "PassengerId", "chunk_req_payload_field": "tracking_id" }, { "csv_jsonl_field": "Survived", "chunk_req_payload_field": "tag_set" }, { "csv_jsonl_field": "Fare", "chunk_req_payload_field": "num_value" } ], "link": "https://raw.githubusercontent.com/datasciencedojo/datasets/refs/heads/master/titanic.csv" }' ``` Trieve's API will respond with an object containing the signed PUT URL and the file's properties as shown in the below example. ```json theme={null} { "file_metadata": { "id": "9ab52e58-0b38-4e4c-b114-139337f0548e", "file_name": "titantic.csv", "created_at": "2024-12-07T06:00:13.984143747", "updated_at": "2024-12-07T06:00:13.984144067", "size": 0, "metadata": null, "link": "https://raw.githubusercontent.com/datasciencedojo/datasets/refs/heads/master/titanic.csv", "time_stamp": null, "dataset_id": "", "tag_set": null }, "presigned_put_url": "https://trieve-s3bucket.s3.amazonaws.com/trieve-s3bucket/" } ``` ### Step 2: Upload the File to the Signed PUT URL Use the signed PUT URL provided by Trieve to upload the CSV or JSONL file to our storage. You can use tools like `curl`, `wget`, or any other HTTP client to upload the file. ```sh theme={null} curl -o ./titanic.csv https://raw.githubusercontent.com/datasciencedojo/datasets/refs/heads/master/titanic.csv curl -X PUT -T ./titanic.csv "" ``` You are now done with the file upload process. Trieve will automatically process the file and create chunks for each row. You can check the progress by migrating your dataset's chunk count in the [dashboard](https://dashboard.trieve.ai) and test via the [search playground](https://search.trieve.ai) or [chat playground](https://chat.trieve.ai). ## Advanced Options Additional options are available to customize the csv or jsonl file upload process. Reference the documentation for our [/api/file/csv\_or\_jsonl](/api-reference/file/create-presigned-url-for-csv-jsonl) route for more information. # Uploading Files to Trieve Source: https://docs.trieve.ai/guides/uploading-files Learn how to upload your files to Trieve ## Overview You must specifically use the `base64url` encoding for the `base64_file` field. We provide the ability for users to upload their files to Trieve and use our automatic large language vision model or Apache Tika chunking. When uploading a file to Trieve, we automatically group the chunks together to link them. This is done through our [upload file route](/api-reference/file/upload-file). ## Uploading a File to Trieve Trieve supports various file types (e.g., HTML, DOCX, PDF). The file is uploaded to a S3 bucket associated with your dataset. As a user, you can use Apache Tika to convert these files to HTML or use a vision LLM to convert the files to markdown. To understand the difference between the two, refer to the section on [Apache Tika vs vision model chunking](#apache-tika-vs-gpt-4o-chunking). ### Important Parameters * **`base64_file`**: To allow users to pass metadata with their file uploads, we require you to specifically use the `base64url` encoding. Convert `+` to `-`, `/` to `_`, and remove the ending `=` if present. * **`file_name`**: The name of the file being uploaded, including the extension. This will become the name of the resulting group. * **`group_tracking_id`**: This field allows you to assign an arbitrary ID to the group, aiding in coordination with your database system. You can search for this group using this ID. * **`link`, `tag_set`, and `time_stamp`**: These fields are indexed to enable fast filtering of groups based on these attributes. * **`target_splits_per_chunk`**: This is an optional field to specify number of splits you want per chunk. If not specified, the default 20 is used. * **`metadata`**: This field allows you to include any arbitrary metadata in the form of a JSON object with the group. * **`pdf2md_options`**: This allows you to use vision LLM to convert the files to markdown. * **`use_pdf2md_ocr`**: If true, the file will be converted to markdown using vision LLM. You can test `pdf2md` performance at [pdf2md.trieve.ai](https://pdf2md.trieve.ai). For the best filtering performance, we recommend using the `link`, `tag_set`, and `time_stamp` fields, as there are dedicated indexes for these. The metadata field has an index built for match queries but is not optimized for range queries. ### Example Upload File Request Whenever you make a request to the Trieve API, you need to include the `TR-Dataset` header with your dataset ID and the `Authorization` header with your API key. ```json cURL theme={null} curl --request POST \ --url https://api.trieve.ai/api/file \ --header 'Authorization: ' \ // Replace with your API key --header 'Content-Type: application/json' \ --header 'TR-Dataset: ' \ // Replace with your dataset ID --data '{ "base64_file": "", "file_name": "example.pdf", "link": "https://example.com", "tag_set": [ "tag1", "tag2" ], "time_stamp": "2025-02-09T22:15:51", "target_splits_per_chunk": 20, "metadata": { "key1": "value1", "key2": "value2" }, "pdf2md_options": { "use_pdf2md_ocr": false // Set to true if you want to use a vision LLM to convert file to markdown } }' ``` ```python Python SDK theme={null} import trieve_py_client from trieve_py_client.models.upload_file_req_payload import UploadFileReqPayload from trieve_py_client.models.upload_file_response_body import UploadFileResponseBody from trieve_py_client.rest import ApiException from pprint import pprint configuration = trieve_py_client.Configuration( host = "https://api.trieve.ai" ) configuration.api_key['ApiKey'] = "" # Replace with your API key configuration.api_key_prefix['ApiKey'] = 'Bearer' with trieve_py_client.ApiClient(configuration) as api_client: api_instance = trieve_py_client.FileApi(api_client) tr_dataset = # Replace with your dataset ID upload_file_req_payload = trieve_py_client.UploadFileReqPayload( base64_file=, # Upload base64 encoded file file_name="example.pdf", link="https://example.com", tag_set=["tag1", "tag2"], time_stamp="2025-02-09T22:15:51", target_splits_per_chunk=20, metadata={ "key1": "value1", "key2": "value2" }, pdf2md_options={ "use_pdf2md_ocr": False # Set to True if you want Vision LLM to convert the file to Markdown } ) try: api_response = api_instance.upload_file_handler(tr_dataset, upload_file_req_payload) print("Uploading file response: \n") pprint(api_response) except Exception as e: print(f"Exception when uploading file: {e}\n") ``` ```python Python Requests theme={null} import requests url = "https://api.trieve.ai/api/file" headers = { "TR-Dataset": "", # Replace with your dataset ID "Authorization": "", # Replace with your API key "Content-Type": "application/json" } payload = { "base64_file": "", # Upload base64 encoded file "file_name": "example.pdf", "link": "https://example.com", "tag_set": ["tag1", "tag2"], "time_stamp": "2025-02-09T22:15:51", "target_splits_per_chunk": 20, "metadata": { "key1": "value1", "key2": "value2" }, "pdf2md_options": { "use_pdf2md_ocr": False # Set to True if you want vision LLM to convert the file to Markdown } } response = requests.request("POST", url, json=payload, headers=headers) print(response.text) ``` ### Example Response ```json 200 theme={null} { "file_metadata": { "created_at": "2025-02-09 22:30:00.000", "dataset_id": "********-****-****-****-************", "file_name": "example.pdf", "id": "********-****-****-****-************", "link": "https://example.com", "metadata": { "key1": "value1", "key2": "value2" }, "size": 1000, "tag_set": "tag1,tag2", "time_stamp": "2025-02-09 22:30:00.000", "updated_at": "2025-02-09 22:30:00.000" } } ``` ```json 400 theme={null} { "message": "Bad Request" } ``` ### File chunking with vision LLM When uploading a file to Trieve, you can use vision LLMs through our [pdf2md service](https://pdf2md.trieve.ai) for intelligent document parsing to easily convert PDF content to LLM-ready, structured Markdown. You can test `pdf2md` performance at [pdf2md.trieve.ai](https://pdf2md.trieve.ai). This allows for better preserving document context, readability, and structure and is especially useful when working with documents containing complex layouts (tables, lists, code blocks, etc). After the uploaded file is converted into structured Markdown, chunks are created based on the semantic structure of the document allowing for better semantic coherence. ### Apache Tika vs Vision LLM chunking **Apache Tika** is a more traditional approach that extracts raw text from a document and converts it into HTML. The chunks are subsequently created by splitting at fixed lengths or pre-defined delimeters. ***Recommended for:*** * Simple, unstructured documents (e.g. plain text files) **Vision LLM chunking** creates chunks based on the semantic structure of the document, allowing for sections (e.g. headers, lists, tables, etc) to stay grouped. ***Recommended for:*** * Complex document structures (e.g. technical documentation and reports) * Documents where context must be preserved ### Heading Based Chunking Heading based chunking allows you to use split contents based on the headings and body content of the document. * Each chunk will contain a heading and its corresponding content. * All related content (paragraphs, lists, tables) under the same heading are grouped together. For example, if you upload the following HTML: ```html theme={null}

Introduction

Hey, this is Trieve!

Background

Trieve brings AI search to the modern world.

``` The response will be: ```json theme={null} { "chunks": [ { "headings": ["Introduction"], "body": "

Introduction

Hey, this is Trieve!

" }, { "headings": ["Background"], "body": "

Background

Trieve brings AI search to the modern world.

" } ] } ``` ### Grouping with File Upload When uploading a file to Trieve, all chunks created from the file will be grouped together. The `file_name` field is used to specify the name of the resulting groups. Once uploaded, documents can be queried using the `file_name` field, allowing you to retrieve and perform operations on all chunks created from the file. # Using Exposed API keys with Trieve Source: https://docs.trieve.ai/guides/using-exposed-api-keys Learn how to create and use safe, access controlled api keys with Trieve ## Overview Proxying search and recommendations through your server is usually a bad idea. It adds latency, complexity, and can be a security risk. Instead, Trieve allows you to create and manage API keys that can be used directly in the client. To create an API key, use the [create Organization API key](/api-reference/organization/create-organization-api-key) route. * TS-SDK method: [createApiKey](https://ts-sdk.trieve.ai/functions/Organization_Methods.createOrganizationApiKey.html) * Python SDK method: [create\_api\_key](https://github.com/devflowinc/trieve/blob/main/clients/python-sdk/trieve_py_client/api/org_api.py) Roughly, you can visualize the flow we are encouraging with the following diagram. Information on how to scope down the access a given API key has is provided are in the next several sections. ![scoped api key pattern diagram](https://trieve.b-cdn.net/docs/client-api-key-req-pattern.png) All of the below examples can be used in combination with each other. For example, you can create an API key that can only access specific organizations, datasets, and tags within those datasets. ## Limit to read-only access Specify the `role` field with the value `0` in your request. ```json curl {8} theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/api-key \ --header 'Authorization --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "name": "Read only API key", "role": 0 }' ``` ## Set a time expiry Specify the `expires_at` field with the time in the future that the API key should expire in your request. This field accepts only ISO 8601 formatted dates. ```json curl {7} theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/api-key \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "name": "expires at '2024-11-14 00:00:00' UTC", "expires_at": "2024-11-14 00:00:00", "role": 0, }' ``` ## Scope access to specific organizations Specify the `organization_ids` field with the organization IDs that the API key should have access to in your request. ```json curl {8-10} theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/api-key \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "name": "Can only make requests to the 3c90c3cc... organization", "organization_ids": [ "3c90c3cc-0d44-4b50-8888-8dd25736052a" ], "role": 1, }' ``` ## Scope access to specific datasets Specify the `dataset_ids` field with the dataset IDs that the API key should have access to in your request. ```json curl {8-10} theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/api-key \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "name": "Can only make requests to the 9u90c3cc... dataset", "dataset_ids": [ "9u90c3cc-0d44-4b50-8888-8dd25736052a" ], "role": 1, }' ``` ## Scope access to specific routes ```json curl {8-22} theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/api-key \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --header 'TR-Organization: ' \ --data '{ "name": "Can only make requests to the 9u90c3cc... dataset", "scopes": [ "POST /api/chunk/search", "POST /api/chunk_group/search", "POST /api/chunk/autocomplete", "POST /api/chunk_group/group_oriented_search", "POST /api/chunk/suggestions", "POST /api/chunk/count", "PUT /api/analytics/ctr", "PUT /api/analytics/search", "PUT /api/analytics/events", "PUT /api/analytics/rag", "POST /api/topic", "POST /api/message", "PUT /api/message" ], "role": 1, }' ``` ## Scope access to chunks matching a filter or other search request payload fields Specify a filter in the `default_params` field in your request. This filter will be applied to all requests made with this API key. You can use this to restrict access to data with more granularity than just dataset or organization. ```json curl {8-19} theme={null} curl --request POST \ --url https://api.trieve.ai/api/organization/api-key \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --data '{ "name": "Can only access chunks with the tag bucket-id-with-protected-access", "default_params": { "filters": { "must": [ { "field": "tag_set", "match_all": [ "bucket-id-with-protected-access", ] }, ] }, }, "role": 0, }' ``` # Setup Voice Search Source: https://docs.trieve.ai/guides/voice-search-setup Learn how to setup voice search with Trieve ## Overview Trieve provides a simple way to integrate voice search into your application. This guide will walk you through the steps to set up voice search with Trieve. For a full implementation example, take a look at the way we implement voice search in our [search component](https://github.com/devflowinc/trieve/blob/e716ab0b276ec056c1d2e554acf46301d27ba487/clients/search-component/src/TrieveModal/Search/UploadAudio.tsx). ## Capturing Audio from the Microphone To enable voice search, you’ll need to capture audio from the user’s microphone using the browser’s `MediaRecorder` API. ### 1. Create a Voice Search Button The following React component lets users start and stop voice recording: ```tsx theme={null} import React, { useState } from "react"; export const VoiceSearchButton = () => { const [recording, setRecording] = useState(false); const [mediaRecorder, setMediaRecorder] = useState(null); const startRecording = async () => { try { const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); const mimeType = navigator.userAgent.includes('Firefox') ? 'audio/webm' : 'audio/mp4'; const recorder = new MediaRecorder(stream, { mimeType }); let audioChunks: Blob[] = []; recorder.ondataavailable = (e) => audioChunks.push(e.data); recorder.onstop = async () => { const audioBlob = new Blob(audioChunks); const base64Audio = await convertBlobToBase64(audioBlob); handleSearch(base64Audio); }; setMediaRecorder(recorder); recorder.start(); setRecording(true); } catch (error) { console.error("Microphone access error:", error); } }; const stopRecording = () => { mediaRecorder?.stop(); setRecording(false); }; return ( ); }; ``` *** ## Converting Audio to Base64 To send the recorded audio to Trieve, convert the audio blob into a base64 string. ```typescript theme={null} const convertBlobToBase64 = (blob: Blob): Promise => { return new Promise((resolve) => { const reader = new FileReader(); reader.onloadend = () => { const base64String = (reader.result as string).split(',')[1]; resolve(base64String); }; reader.readAsDataURL(blob); }); }; ``` *** ## Sending Audio to Trieve for Search Once the audio is recorded and converted to base64, send it to Trieve. The platform will transcribe the speech using OpenAI Whisper and return the search results based on the transcription. We return the transcribed text in the response header as `x-tr-query`, which can be used to update the UI with the search query. ### 1. Handle Voice Search ```typescript theme={null} const handleSearch = async (audioBase64?: string) => { try { const response = await trieveClient.search({ audio_base64: audioBase64, search_type: "hybrid", score_threshold: 0.5, page_size: 10 }); // Retrieve transcribed text from the response header const queryText = response.headers.get('x-tr-query'); // Update UI with search results setSearchResults(response.data.chunks); setQuery(queryText || ""); } catch (error) { console.error("Voice search error:", error); } }; ``` *** ## Best Practices ### Optimize Performance * Keep recordings **under 30 seconds** to improve speed and accuracy. * Use high-quality microphones for clearer transcription results. ### Ensure Browser Compatibility Different browsers support different audio formats. Handle this by selecting the correct MIME type: ```typescript theme={null} const isFirefox = navigator.userAgent.includes('Firefox'); const mimeType = isFirefox ? 'audio/webm' : 'audio/mp4'; ``` # AWS Self Hosting Source: https://docs.trieve.ai/self-hosting/aws Learn how to self-host Trieve on AWS # Installing Trieve on AWS (EKS) Things you need * Domain name * An allowance for at least 8vCPU for G and VT instances * helm cli * aws cli * kubectl * k9s (optional) ## Clone the Trieve repository ```sh theme={null} git clone https://github.com/devflowinc/trieve.git cd trieve ``` ## Login to AWS ```sh theme={null} aws configure ``` ## Provision Terraform aws should be configured with your IAM credentails chosen. Run the following commands to create the EKS cluster ```sh theme={null} cd terraform/aws terraform init terraform apply ``` ### Login to the cluster Set up your kubeconfig to point to the new cluster ```sh theme={null} aws eks update-kubeconfig --region us-east-2 --name trieve-aws-cluster ``` ### Install Ingress nginx + Cert Manager Ingress-nginx + Cert manager is how we will expose the trieve services to the internet. Feel free to use whatever ingress controller you are comfortable with. ```sh theme={null} # To install ingress-nginx kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml # To install cert-manager kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml # To install ClusterIssuer kubectl apply -f https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/test-production/cluster-issuer.yaml ``` ## Setup StorageClass ```sh theme={null} kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` ## Install Trieve ### Download values.yaml file ```sh theme={null} wget https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/values.yaml ``` For full configuration options, the values.yaml contains comments and examples. More information can be found in the [Trieve Helm Chart](https://github.com/devflowinc/trieve-helm/tree/main/charts/trieve) repository. ### Modify domain names for ingresses ### Download values.yaml file ```sh theme={null} wget https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/values.yaml ``` For full configuration options, the values.yaml contains comments and examples. More information can be found in the [Trieve Helm Chart](https://github.com/devflowinc/trieve-helm/tree/main/charts/trieve) repository. ### Modify domain names for ingresses ```yaml values.yaml {4} {15} {20} {29} {34} {48} {57} {66-69} {73-74} theme={null} domains: dashboard: disabled: false host: dashboard.yourdomain.com class: nginx # Annotations for the ingress # Annotations for the service that the ingress points to annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - dashboard.yourdomain.com secretName: dashboard-domain serviceAnnotations: {} server: disabled: false host: api.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - api.yourdomain.com secretName: api-domain serviceAnnotations: {} search: disabled: false host: search.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - search.yourdomain.com secretName: search-domain serviceAnnotations: {} chat: disabled: false host: chat.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - chat.yourdomain.com secretName: chat-domain serviceAnnotations: {} ## ... config: ## For the frontend to communicate vite: apiHost: https://api.yourdomain.com/api searchUiUrl: https://search.yourdomain.com chatUiUrl: https://chat.yourdomain.com dashboardUrl: https://dashboard.yourdomain.com ## For the backend oidc server oidc: issuerUrl: "https://auth.yourdomain.com/realms/trieve" authRedirectUrl: "https://auth.yourdomain.com/realms/trieve/protocol/openid-connect/auth" ``` Install the helm chart. Ensure it is installed twice ```sh theme={null} helm repo add trieve https://devflowinc.github.io/trieve-helm/ helm repo update helm upgrade -i -f helm/values.yaml trieve-local trieve/trieve helm upgrade -i -f helm/values.yaml trieve-local trieve/trieve ``` You will need to install the helm chart twice to ensure the crds are installed and ConfigMap's are updated properly. ### Verify the installation After installing, `kubectl get deployments` should look like this. ``` NAME READY UP-TO-DATE AVAILABLE AGE bktree-worker 1/1 1 1 14m chat 1/1 1 1 14m crawl-worker 1/1 1 1 14m dashboard 1/1 1 1 14m delete-worker 1/1 1 1 14m group-worker 1/1 1 1 14m ingest 10/10 10 10 14m keycloak-operator 1/1 1 1 46m redis 1/1 1 1 46m search 1/1 1 1 14m server 3/3 1 0 14m sync-qdrant 0/0 0 0 14m word-worker 1/1 1 1 14m ``` ### Set Ingress #### Get Ingress ip address ```sh theme={null} kubectl get ingress ``` You will get an output like this: ```sh theme={null} ingress-chat nginx chat.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-dashboard nginx dashboard.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-keycloak nginx auth.yourdomain.com 4.157.193.10 80, 443 20m ingress-search nginx search.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-server nginx api.yourdomain.com 4.157.193.10 80, 443 7m43s ``` #### Add A records Add A records to your domain registrar with the IP address of the ingress. ``` chat.yourdomain.com -> 4.157.193.10 # Example IP dashboard.yourdomain.com -> 4.157.193.10 # Example IP search.yourdomain.com -> 4.157.193.10 # Example IP api.yourdomain.com -> 4.157.193.10 # Example IP ``` Ensure you run `helm upgrade -i -f values.yaml trieve-local trieve/trieve` after making changes. to apply them. The Trieve server will not start if the keycloak domain is not set correctly. Redirect urls will need to be set for login to work. To configure the redirect urls, see [Setup Keycloak redirect urls](#setup-keycloak-redirect-urls) ## Post Installation ### Setup Keycloak redirect urls * Navigate to auth.yourdomain.com * Log in using the default credentials: * Username: admin * Password: password * Switch the Realm from "master" to "trieve" * Navigate to Clients → trieve → Settings * Configure the Valid Redirect URIs and Valid Post Logout Redirect URIs (ensure it is https) ![Keycloak Config](https://cdn.trieve.ai/blog/self-hosting-guide/keycloak-config-preview.webp) ### Verify Your Installation Follow these steps to confirm your Trieve instance is working correctly: Navigate to `dashboard.yourdomain.com` in your browser. You'll be prompted to register a new user account since this is a fresh installation. ![Empty Dashboard](https://cdn.trieve.ai/docs/empty-dashboard.png) Once logged in, create your first dataset and populate it with some example data. This will help you test the core functionality of your Trieve instance. Check the admin event log to ensure that chunks are being created successfully from your uploaded data. This confirms that the ingestion pipeline is working properly. ![Admin Event Log](https://cdn.trieve.ai/docs/admin-event-log.png) Navigate to `search.yourdomain.com` and perform a test search query using the data you just uploaded. This verifies that the search engine is functioning correctly. ![Search Page](https://cdn.trieve.ai/docs/search-page.png) Return to `dashboard.yourdomain.com` and examine your search analytics to confirm that queries are being tracked and analyzed properly. ![Search Analytics](https://cdn.trieve.ai/docs/search-analytics.png) ### 🚀 Congratulations! You're all set! You now have a fully functional Trieve instance running on AWS. Your self-hosted search infrastructure is ready to handle production workloads. ### 📚 Next Steps * [Upload Files](https://trieve.ai/guides/uploading-files) (ensure s3 is configured) * [Follow Quickstart Guide](https://trieve.ai/guides/quickstart) * [API Reference](https://trieve.ai/api-reference) * [Multi-tenency guide](https://trieve.ai/guides/multi-tenency) # Azure Self Hosting Source: https://docs.trieve.ai/self-hosting/azure Learn how to self-host Trieve on Azure # Installing Trieve on Azure (AKS) Things you need * Domain name * An allowance for at least 8vCPU for G and VT instances * helm cli * azure cli * kubectl * k9s (optional) ## Clone the Trieve repository ```sh theme={null} git clone https://github.com/devflowinc/trieve.git cd trieve ``` ## Login to Azure ```sh theme={null} az login ``` ## Provision Terraform Azure should be configured with your IAM credentials chosen. Run the following commands to create the AKS cluster ```sh theme={null} cd terraform/azure terraform init terraform apply ``` ### Login to the cluster Set up your kubeconfig to point to the new cluster ```sh theme={null} az aks get-credentials --resource-group trieve --name trieve ``` ### Install Ingress nginx + Cert Manager Ingress-nginx + Cert manager is how we will expose the trieve services to the internet. Feel free to use whatever ingress controller you are comfortable with. ```sh theme={null} # To install ingress-nginx kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml # To install cert-manager kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml # To install ClusterIssuer kubectl apply -f https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/test-production/cluster-issuer.yaml ``` ## Install Trieve ### Download values.yaml file ```sh theme={null} wget https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/values.yaml ``` For full configuration options, the values.yaml contains comments and examples. More information can be found in the [Trieve Helm Chart](https://github.com/devflowinc/trieve-helm/tree/main/charts/trieve) repository. ### Modify domain names for ingresses ```yaml values.yaml {4} {15} {20} {29} {34} {48} {57} {66-69} {73-74} theme={null} domains: dashboard: disabled: false host: dashboard.yourdomain.com class: nginx # Annotations for the ingress # Annotations for the service that the ingress points to annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - dashboard.yourdomain.com secretName: dashboard-domain serviceAnnotations: {} server: disabled: false host: api.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - api.yourdomain.com secretName: api-domain serviceAnnotations: {} search: disabled: false host: search.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - search.yourdomain.com secretName: search-domain serviceAnnotations: {} chat: disabled: false host: chat.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - chat.yourdomain.com secretName: chat-domain serviceAnnotations: {} ## ... config: ## For the frontend to communicate vite: apiHost: https://api.yourdomain.com/api searchUiUrl: https://search.yourdomain.com chatUiUrl: https://chat.yourdomain.com dashboardUrl: https://dashboard.yourdomain.com ## For the backend oidc server oidc: issuerUrl: "https://auth.yourdomain.com/realms/trieve" authRedirectUrl: "https://auth.yourdomain.com/realms/trieve/protocol/openid-connect/auth" ``` Install the helm chart. Ensure it is installed twice ```sh theme={null} helm repo add trieve https://devflowinc.github.io/trieve-helm/ helm repo update helm upgrade -i -f values.yaml trieve-local trieve/trieve helm upgrade -i -f values.yaml trieve-local trieve/trieve ``` You will need to install the helm chart twice to ensure the crds are installed and ConfigMap's are updated properly. ### Verify the installation After installing, `kubectl get deployments` should look like this. ``` NAME READY UP-TO-DATE AVAILABLE AGE bktree-worker 1/1 1 1 14m chat 1/1 1 1 14m crawl-worker 1/1 1 1 14m dashboard 1/1 1 1 14m delete-worker 1/1 1 1 14m group-worker 1/1 1 1 14m ingest 10/10 10 10 14m keycloak-operator 1/1 1 1 46m redis 1/1 1 1 46m search 1/1 1 1 14m server 3/3 1 0 14m sync-qdrant 0/0 0 0 14m word-worker 1/1 1 1 14m ``` ### Set Ingress #### Get Ingress ip address ```sh theme={null} kubectl get ingress ``` You will get an output like this: ```sh theme={null} ingress-chat nginx chat.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-dashboard nginx dashboard.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-keycloak nginx auth.yourdomain.com 4.157.193.10 80, 443 20m ingress-search nginx search.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-server nginx api.yourdomain.com 4.157.193.10 80, 443 7m43s ``` #### Add A records Add A records to your domain registrar with the IP address of the ingress. ``` chat.yourdomain.com -> 4.157.193.10 # Example IP dashboard.yourdomain.com -> 4.157.193.10 # Example IP search.yourdomain.com -> 4.157.193.10 # Example IP api.yourdomain.com -> 4.157.193.10 # Example IP ``` Ensure you run `helm upgrade -i -f values.yaml trieve-local trieve/trieve` after making changes. to apply them. The Trieve server will not start if the keycloak domain is not set correctly. Redirect urls will need to be set for login to work. To configure the redirect urls, see [Setup Keycloak redirect urls](#setup-keycloak-redirect-urls) ## Post Installation ### Setup Keycloak redirect urls * Navigate to auth.yourdomain.com * Log in using the default credentials: * Username: admin * Password: password * Switch the Realm from "master" to "trieve" * Navigate to Clients → trieve → Settings * Configure the Valid Redirect URIs and Valid Post Logout Redirect URIs (ensure it is https) ![Keycloak Config](https://cdn.trieve.ai/blog/self-hosting-guide/keycloak-config-preview.webp) ### Verify Your Installation Follow these steps to confirm your Trieve instance is working correctly: Navigate to `dashboard.yourdomain.com` in your browser. You'll be prompted to register a new user account since this is a fresh installation. ![Empty Dashboard](https://cdn.trieve.ai/docs/empty-dashboard.png) Once logged in, create your first dataset and populate it with some example data. This will help you test the core functionality of your Trieve instance. Check the admin event log to ensure that chunks are being created successfully from your uploaded data. This confirms that the ingestion pipeline is working properly. ![Admin Event Log](https://cdn.trieve.ai/docs/admin-event-log.png) Navigate to `search.yourdomain.com` and perform a test search query using the data you just uploaded. This verifies that the search engine is functioning correctly. ![Search Page](https://cdn.trieve.ai/docs/search-page.png) Return to `dashboard.yourdomain.com` and examine your search analytics to confirm that queries are being tracked and analyzed properly. ![Search Analytics](https://cdn.trieve.ai/docs/search-analytics.png) ## 🚀 Congratulations! You're all set! Whats Next? You now have a fully functional Trieve instance running on Azure. Your self-hosted search infrastructure is ready to handle production workloads. Want to find out what's next? * [Upload Files](https://trieve.ai/guides/uploading-files) (ensure s3 is configured) * [Follow Quickstart Guide](https://trieve.ai/guides/quickstart) * [API Reference](https://trieve.ai/api-reference) * [Multi-tenency guide](https://trieve.ai/guides/multi-tenency) # Docker Compose Setup Source: https://docs.trieve.ai/self-hosting/docker-compose Learn how to self-host Trieve with Docker Compose ## Docker Compose The Docker Compose self hosted option is the easiest way to get started self hosting Trieve. Things you need * Domain name * System with at least 4 CPU cores and 8GB of RAM (excluding the cpu embedding servers) * System with at least 4 CPU cores and >25GB of RAM (including the cpu embedding servers) ### Install Docker ```bash theme={null} curl https://get.docker.com | sh ``` ### Clone Trieve repository ```sh theme={null} git clone https://github.com/devflowinc/trieve cd trieve ``` ### Create `.env` file ```sh theme={null} cp .env.example .env ``` ### Start Trieve Services ```sh theme={null} docker compose up -d ``` ### Start Embedding servers We offer 2 docker-compose files for embedding servers. One for GPU and one for CPU. ```sh theme={null} docker compose -f docker-compose-cpu-embeddings.yml up -d ``` or ```sh theme={null} docker compose -f docker-compose-gpu-embeddings.yml up -d ``` \* Note on embedding servers. If you want to use a separate GPU enabled device for embedding servers you will need to update the following parameters ```sh theme={null} SPARSE_SERVER_QUERY_ORIGIN SPARSE_SERVER_DOC_ORIGIN EMBEDDING_SERVER_ORIGIN SPARSE_SERVER_QUERY_ORIGIN ``` ### Setup Caddy reverse proxy Install Caddy Edit the Caddyfile ```sh theme={null} nano /etc/caddy/Caddyfile ``` Add the following configuration ```Caddyfile theme={null} dashboard.yourdomain.com { reverse_proxy localhost:5173 } chat.yourdomain.com { reverse_proxy localhost:5175 } search.yourdomain.com { reverse_proxy localhost:5174 } api.yourdomain.com { reverse_proxy localhost:8090 } auth.yourdomain.com { reverse_proxy localhost:8080 } ``` Start Caddy, you may also need to reload the service ```sh theme={null} sudo systemctl reload caddy.service ``` ### Set the following A records for your domain to point to the server IP address. ``` A dashboard.yourdomain.com your-server-ip A chat.yourdomain.com your-server-ip A search.yourdomain.com your-server-ip A auth.yourdomain.com your-server-ip A api.yourdomain.com your-server-ip ``` ### Edit .env Most values can be left as default, the ones you do need to edit are ``` KC_HOSTNAME="auth.yourdomain.com" KC_PROXY=edge VITE_API_HOST=https://api.yourdomain.com/api VITE_SEARCH_UI_URL=https://search.yourdomain.com VITE_CHAT_UI_URL=https://chat.yourdomain.com VITE_ANALYTICS_UI_URL=https://analytics.yourdomain.com VITE_DASHBOARD_URL=https://dashboard.yourdomain.com OIDC_AUTH_REDIRECT_URL="https://auth.yourdomain.com/realms/trieve/protocol/openid-connect/auth" OIDC_ISSUER_URL="https://auth.yourdomain.com/realms/trieve" BASE_SERVER_URL="https://api.yourdomain.com" ``` ### Authorize keycloak redirect URLs Go to auth.yourdomain.com and login with the default credentials (user: admin password: aintsecure) 1. Change the Realm from master to trieve 2. Go to Clients -> vault -> Settings 3. Add the following to the Valid Redirect URIs and Valid Post Logout Redirect URIs ``` https://api.yourdomain.com/* https://dashboard.yourdomain.com/* https://chat.yourdomain.com/* https://search.yourdomain.com/* ``` ### Testing The fastest way to test is using the Trieve CLI ``` trieve login # Make sure to set the api url to http://localhost:8090 trieve dataset example ``` And there you have it. Your very own Trieve stack. Happy hacking 🚀 # GCP Self Hosting Source: https://docs.trieve.ai/self-hosting/gcp Learn how to self-host Trieve on Google Cloud Platform # Installing Trieve on GCP (GKE) Things you need * Domain name * An allowance for at least 8vCPU for G and VT instances * helm cli * google cloud cli * kubectl * k9s (optional) ## Clone the Trieve repository ```sh theme={null} git clone https://github.com/devflowinc/trieve.git cd trieve ``` ## Login to GCP ```sh theme={null} gcloud auth login ``` ## Provision Terraform gcloud should be configured with your credentials chosen. Run the following commands to create the GKE cluster: ```sh theme={null} cp terraform/gcloud/vars.tfvars.example terraform/gcloud/vars.tfvars # Edit terraform/gcloud/vars.tfvars with your project id, cluster name, region, and zone cd terraform/gcloud terraform init terraform apply -var vars.tfvars ``` ### Login to the cluster Set up your kubeconfig to point to the new cluster ```sh theme={null} gcloud container clusters get-credentials --region us-west2-a trievegcp ``` ### Install Ingress nginx + Cert Manager Ingress-nginx + Cert manager is how we will expose the trieve services to the internet. Feel free to use whatever ingress controller you are comfortable with. ```sh theme={null} # To install ingress-nginx kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml # To install cert-manager kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml # To install ClusterIssuer kubectl apply -f https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/test-production/cluster-issuer.yaml ``` ## Install Trieve ### Download values.yaml file ```sh theme={null} wget https://raw.githubusercontent.com/devflowinc/trieve-helm/refs/heads/main/charts/trieve/values.yaml ``` For full configuration options, the values.yaml contains comments and examples. More information can be found in the [Trieve Helm Chart](https://github.com/devflowinc/trieve-helm/tree/main/charts/trieve) repository. ### Modify domain names for ingresses ```yaml values.yaml {4} {15} {20} {29} {34} {48} {57} {66-69} {73-74} theme={null} domains: dashboard: disabled: false host: dashboard.yourdomain.com class: nginx # Annotations for the ingress # Annotations for the service that the ingress points to annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - dashboard.yourdomain.com secretName: dashboard-domain serviceAnnotations: {} server: disabled: false host: api.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - api.yourdomain.com secretName: api-domain serviceAnnotations: {} search: disabled: false host: search.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - search.yourdomain.com secretName: search-domain serviceAnnotations: {} chat: disabled: false host: chat.yourdomain.com class: nginx annotations: kubernetes.io/ingress.class: nginx acme.cert-manager.io/http01-edit-in-place: "true" cert-manager.io/cluster-issuer: "letsencrypt" nginx.ingress.kubernetes.io/ssl-redirect: "true" tls: - hosts: - chat.yourdomain.com secretName: chat-domain serviceAnnotations: {} ## ... config: ## For the frontend to communicate vite: apiHost: https://api.yourdomain.com/api searchUiUrl: https://search.yourdomain.com chatUiUrl: https://chat.yourdomain.com dashboardUrl: https://dashboard.yourdomain.com ## For the backend oidc server oidc: issuerUrl: "https://auth.yourdomain.com/realms/trieve" authRedirectUrl: "https://auth.yourdomain.com/realms/trieve/protocol/openid-connect/auth" ``` Install the helm chart. Ensure it is installed twice ```sh theme={null} helm repo add trieve https://devflowinc.github.io/trieve-helm/ helm repo update helm upgrade -i -f helm/values.yaml trieve-local trieve/trieve helm upgrade -i -f helm/values.yaml trieve-local trieve/trieve ``` You will need to install the helm chart twice to ensure the crds are installed and ConfigMap's are updated properly. ### Verify the installation After installing, `kubectl get deployments` should look like this. ``` NAME READY UP-TO-DATE AVAILABLE AGE bktree-worker 1/1 1 1 14m chat 1/1 1 1 14m crawl-worker 1/1 1 1 14m dashboard 1/1 1 1 14m delete-worker 1/1 1 1 14m group-worker 1/1 1 1 14m ingest 10/10 10 10 14m keycloak-operator 1/1 1 1 46m redis 1/1 1 1 46m search 1/1 1 1 14m server 3/3 1 0 14m sync-qdrant 0/0 0 0 14m word-worker 1/1 1 1 14m ``` ### Set Ingress #### Get Ingress ip address ```sh theme={null} kubectl get ingress ``` You will get an output like this: ```sh theme={null} ingress-chat gce chat.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-dashboard gce dashboard.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-keycloak gce auth.yourdomain.com 4.157.193.10 80, 443 20m ingress-search gce search.yourdomain.com 4.157.193.10 80, 443 7m43s ingress-server gce api.yourdomain.com 4.157.193.10 80, 443 7m43s ``` #### Add A records Add A records to your domain registrar with the IP address of the ingress. ``` chat.yourdomain.com -> 4.157.193.10 # Example IP dashboard.yourdomain.com -> 4.157.193.10 # Example IP search.yourdomain.com -> 4.157.193.10 # Example IP api.yourdomain.com -> 4.157.193.10 # Example IP ``` Ensure you run `helm upgrade -i -f helm/values.yaml trieve-local trieve/trieve` after making changes. to apply them. The Trieve server will not start if the keycloak domain is not set correctly. Redirect urls will need to be set for login to work. To configure the redirect urls, see [Setup Keycloak redirect urls](#setup-keycloak-redirect-urls) ## Post Installation ### Setup Keycloak redirect urls * Navigate to auth.yourdomain.com * Log in using the default credentials: * Username: admin * Password: password * Switch the Realm from "master" to "trieve" * Navigate to Clients → trieve → Settings * Configure the Valid Redirect URIs and Valid Post Logout Redirect URIs (ensure it is https) ![Keycloak Config](https://cdn.trieve.ai/blog/self-hosting-guide/keycloak-config-preview.webp) ### Verify Your Installation Follow these steps to confirm your Trieve instance is working correctly: Navigate to `dashboard.yourdomain.com` in your browser. You'll be prompted to register a new user account since this is a fresh installation. ![Empty Dashboard](https://cdn.trieve.ai/docs/empty-dashboard.png) Once logged in, create your first dataset and populate it with some example data. This will help you test the core functionality of your Trieve instance. Check the admin event log to ensure that chunks are being created successfully from your uploaded data. This confirms that the ingestion pipeline is working properly. ![Admin Event Log](https://cdn.trieve.ai/docs/admin-event-log.png) Navigate to `search.yourdomain.com` and perform a test search query using the data you just uploaded. This verifies that the search engine is functioning correctly. ![Search Page](https://cdn.trieve.ai/docs/search-page.png) Return to `dashboard.yourdomain.com` and examine your search analytics to confirm that queries are being tracked and analyzed properly. ![Search Analytics](https://cdn.trieve.ai/docs/search-analytics.png) ### 🚀 Congratulations! You're all set! You now have a fully functional Trieve instance running on GCP. Your self-hosted search infrastructure is ready to handle production workloads. ### 📚 Next Steps * [Upload Files](https://trieve.ai/guides/uploading-files) (ensure gcs is configured) * [Follow Quickstart Guide](https://trieve.ai/guides/quickstart) * [API Reference](https://trieve.ai/api-reference) * [Multi-tenency guide](https://trieve.ai/guides/multi-tenency) # SiteSearch Content Sites Guide Source: https://docs.trieve.ai/site-search/content-sites Step by step guide on setting up a scrape of any help center, blog, or content site, configuring the web component, and adding the script tag to your site. ## Video Guide