curl --request POST \
--url https://api.trieve.ai/api/dataset \
--header 'Authorization: <api-key>' \
--header 'Content-Type: application/json' \
--header 'TR-Organization: <tr-organization>' \
--data '{
"dataset_name": "My Dataset",
"organization_id": "00000000-0000-0000-0000-000000000000",
"server_configuration": {
"AIMON_RERANKER_TASK_DEFINITION": "Your task is to grade the relevance of context document(s) against the specified user query.",
"BM25_AVG_LEN": 256,
"BM25_B": 0.75,
"BM25_ENABLED": true,
"BM25_K": 0.75,
"DISTANCE_METRIC": "cosine",
"EMBEDDING_BASE_URL": "https://api.openai.com/v1",
"EMBEDDING_MODEL_NAME": "text-embedding-3-small",
"EMBEDDING_QUERY_PREFIX": "",
"EMBEDDING_SIZE": 1536,
"FREQUENCY_PENALTY": 0,
"FULLTEXT_ENABLED": true,
"INDEXED_ONLY": false,
"LLM_BASE_URL": "https://api.openai.com/v1",
"LLM_DEFAULT_MODEL": "gpt-3.5-turbo-1106",
"LOCKED": false,
"MAX_LIMIT": 10000,
"MESSAGE_TO_QUERY_PROMPT": "Write a 1-2 sentence semantic search query along the lines of a hypothetical response to: \n\n",
"N_RETRIEVALS_TO_INCLUDE": 8,
"PRESENCE_PENALTY": 0,
"QDRANT_ONLY": false,
"RAG_PROMPT": "Use the following retrieved documents to respond briefly and accurately:",
"SEMANTIC_ENABLED": true,
"STOP_TOKENS": [
"\n\n",
"\n"
],
"SYSTEM_PROMPT": "You are a helpful assistant",
"TEMPERATURE": 0.5,
"USE_MESSAGE_TO_QUERY_PROMPT": false
}
}'{
"created_at": "2021-01-01 00:00:00.000",
"id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
"name": "Trieve",
"organization_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
"server_configuration": {
"AIMON_RERANKER_TASK_DEFINITION": "Your task is to grade the relevance of context document(s) against the specified user query.",
"BM25_AVG_LEN": 256,
"BM25_B": 0.75,
"BM25_ENABLED": true,
"BM25_K": 0.75,
"DISTANCE_METRIC": "cosine",
"EMBEDDING_BASE_URL": "https://embedding.trieve.ai",
"EMBEDDING_MODEL_NAME": "jina-base-en",
"EMBEDDING_QUERY_PREFIX": "",
"EMBEDDING_SIZE": 768,
"FREQUENCY_PENALTY": 0,
"FULLTEXT_ENABLED": true,
"INDEXED_ONLY": false,
"LLM_BASE_URL": "https://api.openai.com/v1",
"LLM_DEFAULT_MODEL": "gpt-4o",
"LOCKED": false,
"MAX_LIMIT": 10000,
"MESSAGE_TO_QUERY_PROMPT": "Write a 1-2 sentence semantic search query along the lines of a hypothetical response to: \n\n",
"N_RETRIEVALS_TO_INCLUDE": 8,
"PRESENCE_PENALTY": 0,
"QDRANT_ONLY": false,
"RAG_PROMPT": "Use the following retrieved documents to respond briefly and accurately:",
"SEMANTIC_ENABLED": true,
"STOP_TOKENS": [
"\n\n",
"\n"
],
"SYSTEM_PROMPT": "You are a helpful assistant",
"TEMPERATURE": 0.5,
"USE_MESSAGE_TO_QUERY_PROMPT": false
},
"tracking_id": "foobar-dataset",
"updated_at": "2021-01-01 00:00:00.000"
}Dataset will be created in the org specified via the TR-Organization header. Auth’ed user must be an owner of the organization to create a dataset.
curl --request POST \
--url https://api.trieve.ai/api/dataset \
--header 'Authorization: <api-key>' \
--header 'Content-Type: application/json' \
--header 'TR-Organization: <tr-organization>' \
--data '{
"dataset_name": "My Dataset",
"organization_id": "00000000-0000-0000-0000-000000000000",
"server_configuration": {
"AIMON_RERANKER_TASK_DEFINITION": "Your task is to grade the relevance of context document(s) against the specified user query.",
"BM25_AVG_LEN": 256,
"BM25_B": 0.75,
"BM25_ENABLED": true,
"BM25_K": 0.75,
"DISTANCE_METRIC": "cosine",
"EMBEDDING_BASE_URL": "https://api.openai.com/v1",
"EMBEDDING_MODEL_NAME": "text-embedding-3-small",
"EMBEDDING_QUERY_PREFIX": "",
"EMBEDDING_SIZE": 1536,
"FREQUENCY_PENALTY": 0,
"FULLTEXT_ENABLED": true,
"INDEXED_ONLY": false,
"LLM_BASE_URL": "https://api.openai.com/v1",
"LLM_DEFAULT_MODEL": "gpt-3.5-turbo-1106",
"LOCKED": false,
"MAX_LIMIT": 10000,
"MESSAGE_TO_QUERY_PROMPT": "Write a 1-2 sentence semantic search query along the lines of a hypothetical response to: \n\n",
"N_RETRIEVALS_TO_INCLUDE": 8,
"PRESENCE_PENALTY": 0,
"QDRANT_ONLY": false,
"RAG_PROMPT": "Use the following retrieved documents to respond briefly and accurately:",
"SEMANTIC_ENABLED": true,
"STOP_TOKENS": [
"\n\n",
"\n"
],
"SYSTEM_PROMPT": "You are a helpful assistant",
"TEMPERATURE": 0.5,
"USE_MESSAGE_TO_QUERY_PROMPT": false
}
}'{
"created_at": "2021-01-01 00:00:00.000",
"id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
"name": "Trieve",
"organization_id": "e3e3e3e3-e3e3-e3e3-e3e3-e3e3e3e3e3e3",
"server_configuration": {
"AIMON_RERANKER_TASK_DEFINITION": "Your task is to grade the relevance of context document(s) against the specified user query.",
"BM25_AVG_LEN": 256,
"BM25_B": 0.75,
"BM25_ENABLED": true,
"BM25_K": 0.75,
"DISTANCE_METRIC": "cosine",
"EMBEDDING_BASE_URL": "https://embedding.trieve.ai",
"EMBEDDING_MODEL_NAME": "jina-base-en",
"EMBEDDING_QUERY_PREFIX": "",
"EMBEDDING_SIZE": 768,
"FREQUENCY_PENALTY": 0,
"FULLTEXT_ENABLED": true,
"INDEXED_ONLY": false,
"LLM_BASE_URL": "https://api.openai.com/v1",
"LLM_DEFAULT_MODEL": "gpt-4o",
"LOCKED": false,
"MAX_LIMIT": 10000,
"MESSAGE_TO_QUERY_PROMPT": "Write a 1-2 sentence semantic search query along the lines of a hypothetical response to: \n\n",
"N_RETRIEVALS_TO_INCLUDE": 8,
"PRESENCE_PENALTY": 0,
"QDRANT_ONLY": false,
"RAG_PROMPT": "Use the following retrieved documents to respond briefly and accurately:",
"SEMANTIC_ENABLED": true,
"STOP_TOKENS": [
"\n\n",
"\n"
],
"SYSTEM_PROMPT": "You are a helpful assistant",
"TEMPERATURE": 0.5,
"USE_MESSAGE_TO_QUERY_PROMPT": false
},
"tracking_id": "foobar-dataset",
"updated_at": "2021-01-01 00:00:00.000"
}The organization id to use for the request
JSON request payload to create a new dataset
Name of the dataset.
Lets you specify the configuration for a dataset
Show child attributes
The average length of the chunks in the index for BM25
The BM25 B parameter
Whether to use BM25
The BM25 K parameter
Whether to disable analytics
euclidean, cosine, manhattan, dot The base URL for the embedding API
The name of the embedding model to use
The prefix to use for the embedding query
The size of the embeddings
x >= 0The frequency penalty to use
Whether to use fulltext search
Whether to only use indexed chunks
The API version for the LLM API
The base URL for the LLM API
The default model to use for the LLM
Whether the dataset is locked to prevent changes or deletion
The maximum limit for the number of chunks for counting
x >= 0The maximum number of tokens to use in LLM Response
x >= 0The prompt to use for converting a message to a query
The number of retrievals to include with the RAG model
x >= 0Whether to enable pagefind indexing
The presence penalty to use
Show child attributes
Show child attributes
Show child attributes
x >= 0Show child attributes
Show child attributes
Show child attributes
Set content_only to true to only returning the chunk_html of the chunks. This is useful for when you want to reduce amount of data over the wire for latency improvement (typically 10-50ms). Default is false.
ChunkFilter is a JSON object which can be used to filter chunks. This is useful for when you want to filter chunks by arbitrary metadata. Unlike with tag filtering, there is a performance hit for filtering on metadata.
Show child attributes
All of these field conditions have to match for the chunk to be included in the result set.
Filters can be constructed using either fields on the chunk objects, ids or tracking ids of chunks, and finally ids or tracking ids of groups.
Show child attributes
Field is the name of the field to filter on. Commonly used fields are timestamp, link, tag_set, location, num_value, group_ids, and group_tracking_ids. The field value will be used to check for an exact substring match on the metadata values for each existing chunk. This is useful for when you want to filter chunks by arbitrary metadata. To access fields inside of the metadata that you provide with the card, prefix the field name with metadata..
Boolean is a true false value for a field. This only works for boolean fields. You can specify this if you want values to be true or false.
DateRange is a JSON object which can be used to filter chunks by a range of dates. This leverages the time_stamp field on chunks in your dataset. You can specify this if you want values in a certain range. You must provide ISO 8601 combined date and time without timezone.
{
"gt": "2021-01-01 00:00:00.000",
"gte": "2021-01-01 00:00:00.000",
"lt": "2021-01-01 00:00:00.000",
"lte": "2021-01-01 00:00:00.000"
}Show child attributes
Show child attributes
Match all lets you pass in an array of values that will return results if all of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.
Match any lets you pass in an array of values that will return results if any of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.
{
"field": "metadata.key1",
"match": ["value1", "value2"],
"range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}None of these field conditions can match for the chunk to be included in the result set.
Filters can be constructed using either fields on the chunk objects, ids or tracking ids of chunks, and finally ids or tracking ids of groups.
Show child attributes
Field is the name of the field to filter on. Commonly used fields are timestamp, link, tag_set, location, num_value, group_ids, and group_tracking_ids. The field value will be used to check for an exact substring match on the metadata values for each existing chunk. This is useful for when you want to filter chunks by arbitrary metadata. To access fields inside of the metadata that you provide with the card, prefix the field name with metadata..
Boolean is a true false value for a field. This only works for boolean fields. You can specify this if you want values to be true or false.
DateRange is a JSON object which can be used to filter chunks by a range of dates. This leverages the time_stamp field on chunks in your dataset. You can specify this if you want values in a certain range. You must provide ISO 8601 combined date and time without timezone.
{
"gt": "2021-01-01 00:00:00.000",
"gte": "2021-01-01 00:00:00.000",
"lt": "2021-01-01 00:00:00.000",
"lte": "2021-01-01 00:00:00.000"
}Show child attributes
Show child attributes
Match all lets you pass in an array of values that will return results if all of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.
Match any lets you pass in an array of values that will return results if any of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.
{
"field": "metadata.key1",
"match": ["value1", "value2"],
"range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}Only one of these field conditions has to match for the chunk to be included in the result set.
Filters can be constructed using either fields on the chunk objects, ids or tracking ids of chunks, and finally ids or tracking ids of groups.
Show child attributes
Field is the name of the field to filter on. Commonly used fields are timestamp, link, tag_set, location, num_value, group_ids, and group_tracking_ids. The field value will be used to check for an exact substring match on the metadata values for each existing chunk. This is useful for when you want to filter chunks by arbitrary metadata. To access fields inside of the metadata that you provide with the card, prefix the field name with metadata..
Boolean is a true false value for a field. This only works for boolean fields. You can specify this if you want values to be true or false.
DateRange is a JSON object which can be used to filter chunks by a range of dates. This leverages the time_stamp field on chunks in your dataset. You can specify this if you want values in a certain range. You must provide ISO 8601 combined date and time without timezone.
{
"gt": "2021-01-01 00:00:00.000",
"gte": "2021-01-01 00:00:00.000",
"lt": "2021-01-01 00:00:00.000",
"lte": "2021-01-01 00:00:00.000"
}Show child attributes
Show child attributes
Match all lets you pass in an array of values that will return results if all of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.
Match any lets you pass in an array of values that will return results if any of the items match. The match value will be used to check for an exact substring match on the metadata values for each existing chunk. If both match_all and match_any are provided, the match_any condition will be used.
{
"field": "metadata.key1",
"match": ["value1", "value2"],
"range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}{
"must": [
{
"field": "tag_set",
"match_all": ["A", "B"]
},
{
"field": "num_value",
"range": { "gte": 10, "lte": 25 }
}
]
}Get total page count for the query accounting for the applied filters. Defaults to false, but can be set to true when the latency penalty is acceptable (typically 50-200ms).
Page of chunks to fetch. Page is 1-indexed.
x >= 0Page size is the number of chunks to fetch. This can be used to fetch more than 10 chunks at a time.
x >= 0If true, stop words (specified in server/src/stop-words.txt in the git repo) will be removed. Queries that are entirely stop words will be preserved.
Set score_threshold to a float to filter out chunks with a score below the threshold for cosine distance metric. For Manhattan Distance, Euclidean Distance, and Dot Product, it will filter out scores above the threshold distance. This threshold applies before weight and bias modifications. If not specified, this defaults to no threshold. A threshold of 0 will default to no threshold.
Scoring options provides ways to modify the sparse or dense vector created for the query in order to change how potential matches are scored. If not specified, this defaults to no modifications.
Show child attributes
Boost the presence of certain tokens for fulltext (SPLADE) and keyword (BM25) search. I.e. boosting title phrases to priortize title matches or making sure that the listing for AirBNB itself ranks higher than companies who make software for AirBNB hosts by boosting the in-document-frequency of the AirBNB token (AKA word) for its official listing. Conceptually it multiples the in-document-importance second value in the tuples of the SPLADE or BM25 sparse vector of the chunk_html innerText for all tokens present in the boost phrase by the boost factor like so: (token, in-document-importance) -> (token, in-document-importance*boost_factor).
Show child attributes
Amount to multiplicatevly increase the frequency of the tokens in the phrase by
The phrase to boost in the fulltext document frequency index
Semantic boosting moves the dense vector of the chunk in the direction of the distance phrase for semantic search. I.e. you can force a cluster by moving every chunk for a PDF closer to its title or push a chunk with a chunk_html of "iphone" 25% closer to the term "flagship" by using the distance phrase "flagship" and a distance factor of 0.25. Conceptually it's drawing a line (euclidean/L2 distance) between the vector for the innerText of the chunk_html and distance_phrase then moving the vector of the chunk_html distance_factor*L2Distance closer to or away from the distance_phrase point along the line between the two points.
Show child attributes
Arbitrary float (positive or negative) specifying the multiplicate factor to apply before summing the phrase vector with the chunk_html embedding vector
Terms to embed in order to create the vector which is weighted summed with the chunk_html embedding vector
fulltext, semantic, hybrid, bm25 Set slim_chunks to true to avoid returning the content and chunk_html of the chunks. This is useful for when you want to reduce amount of data over the wire for latency improvement (typically 10-50ms). Default is false.
Sort Options lets you specify different methods to rerank the chunks in the result set. If not specified, this defaults to the score of the chunks.
Show child attributes
Location bias lets you rank your results by distance from a location. If not specified, this has no effect. Bias allows you to determine how much of an effect the location of chunks will have on the search results. If not specified, this defaults to 0.0. We recommend setting this to 1.0 for a gentle reranking of the results, >3.0 for a strong reranking of the results.
Show child attributes
Bias lets you specify how much of an effect the location of chunks will have on the search results. If not specified, this defaults to 0.0. We recommend setting this to 1.0 for a gentle reranking of the results, >3.0 for a strong reranking of the results.
Location that you want to use as the center of the search.
MMR Options lets you specify different methods to rerank the chunks in the result set using Maximal Marginal Relevance. If not specified, this defaults to the score of the chunks.
Show child attributes
Set use_mmr to true to use the Maximal Marginal Relevance algorithm to rerank the results.
Set mmr_lambda to a value between 0.0 and 1.0 to control the tradeoff between relevance and diversity. Closer to 1.0 will give more diverse results, closer to 0.0 will give more relevant results. If not specified, this defaults to 0.5.
Recency Bias lets you determine how much of an effect the recency of chunks will have on the search results. If not specified, this defaults to 0.0. We recommend setting this to 1.0 for a gentle reranking of the results, >3.0 for a strong reranking of the results.
Sort by lets you specify a method to sort the results by. If not specified, this defaults to the score of the chunks. If specified, this can be any key in the chunk metadata. This key must be a numeric value within the payload.
Show child attributes
Field to sort by. This has to be a numeric field with a Qdrant Range index on it. i.e. num_value and timestamp
desc, asc How many results to pull in before the sort
x >= 0Tag weights is a JSON object which can be used to boost the ranking of chunks with certain tags. This is useful for when you want to be able to bias towards chunks with a certain tag on the fly. The keys are the tag names and the values are the weights.
Show child attributes
Set use_weights to true to use the weights of the chunks in the result set in order to sort them. If not specified, this defaults to true.
Typo Options lets you specify different methods to correct typos in the query. If not specified, typos will not be corrected.
Show child attributes
Set correct_typos to true to correct typos in the query. If not specified, this defaults to false.
Words that should not be corrected. If not specified, this defaults to an empty list.
The TypoRange struct is used to specify the range of which the query will be corrected if it has a typo.
Show child attributes
The minimum number of characters that the query will be corrected if it has a typo. If not specified, this defaults to 5.
x >= 0The maximum number of characters that the query will be corrected if it has a typo. If not specified, this defaults to 8.
x >= 0Auto-require non-english words present in the dataset to exist in each results chunk_html text. If not specified, this defaults to true.
The TypoRange struct is used to specify the range of which the query will be corrected if it has a typo.
Show child attributes
The minimum number of characters that the query will be corrected if it has a typo. If not specified, this defaults to 5.
x >= 0The maximum number of characters that the query will be corrected if it has a typo. If not specified, this defaults to 8.
x >= 0Enables autocomplete on the search modal.
If true, quoted and - prefixed words will be parsed from the queries and used as required and negated words respectively. Default is false.
User ID is the id of the user who is making the request. This is used to track user interactions with the search results.
{
"filters": {
"must": [
{
"field": "num_value",
"range": { "gt": 0, "gte": 0, "lt": 1, "lte": 1 }
}
],
"must_not": [
{
"field": "metadata.key3",
"match": ["value5", "value6"]
}
],
"should": [
{
"field": "metadata.key1",
"match": ["value1", "value2"]
}
]
},
"score_threshold": 0.5,
"search_type": "semantic"
}Show child attributes
Show child attributes
Show child attributes
Show child attributes
Show child attributes
light, dark Whether or not to insert chunks into Postgres
The prompt to use for the RAG model
The base URL for the reranker API
The model name for the Reranker API
Whether to use semantic search
The stop tokens to use
The system prompt to use for the LLM
The temperature to use
Show child attributes
Show child attributes
Whether to use the message to query prompt
{
"AIMON_RERANKER_TASK_DEFINITION": "Your task is to grade the relevance of context document(s) against the specified user query.",
"BM25_AVG_LEN": 256,
"BM25_B": 0.75,
"BM25_ENABLED": true,
"BM25_K": 0.75,
"DISTANCE_METRIC": "cosine",
"EMBEDDING_BASE_URL": "https://embedding.trieve.ai",
"EMBEDDING_MODEL_NAME": "jina-base-en",
"EMBEDDING_QUERY_PREFIX": "",
"EMBEDDING_SIZE": 768,
"FREQUENCY_PENALTY": 0,
"FULLTEXT_ENABLED": true,
"INDEXED_ONLY": false,
"LLM_BASE_URL": "https://api.openai.com/v1",
"LLM_DEFAULT_MODEL": "gpt-4o",
"LOCKED": false,
"MAX_LIMIT": 10000,
"MESSAGE_TO_QUERY_PROMPT": "Write a 1-2 sentence semantic search query along the lines of a hypothetical response to: \n\n",
"N_RETRIEVALS_TO_INCLUDE": 8,
"PRESENCE_PENALTY": 0,
"QDRANT_ONLY": false,
"RAG_PROMPT": "Use the following retrieved documents to respond briefly and accurately:",
"SEMANTIC_ENABLED": true,
"STOP_TOKENS": ["\n\n", "\n"],
"SYSTEM_PROMPT": "You are a helpful assistant",
"TEMPERATURE": 0.5,
"USE_MESSAGE_TO_QUERY_PROMPT": false
}Optional tracking ID for the dataset. Can be used to track the dataset in external systems. Must be unique within the organization. Strongly recommended to not use a valid uuid value as that will not work with the TR-Dataset header.
Dataset created successfully
Timestamp of the creation of the dataset
Flag to indicate if the dataset has been deleted. Deletes are handled async after the flag is set so as to avoid expensive search index compaction.
Unique identifier of the dataset, auto-generated uuid created by Trieve
Name of the dataset
Unique identifier of the organization that owns the dataset
Configuration of the dataset for RAG, embeddings, BM25, etc.
Timestamp of the last update of the dataset
Tracking ID of the dataset, can be any string, determined by the user. Tracking ID's are unique identifiers for datasets within an organization. They are designed to match the unique identifier of the dataset in the user's system.
Was this page helpful?