From 680ce2a175616034f4f0b592f7b013de79c52300 Mon Sep 17 00:00:00 2001 From: AllenWriter Date: Sun, 27 Apr 2025 15:17:58 +0800 Subject: [PATCH 1/2] Docs: recovery dataset good description --- en/features/datasets/README.md | 134 +++++++++++++++ .../datasets/maintain-dataset-via-api.md | 152 ++++++++++++++++++ en/features/datasets/sync-from-notion.md | 74 +++++++++ 3 files changed, 360 insertions(+) create mode 100644 en/features/datasets/README.md create mode 100644 en/features/datasets/maintain-dataset-via-api.md create mode 100644 en/features/datasets/sync-from-notion.md diff --git a/en/features/datasets/README.md b/en/features/datasets/README.md new file mode 100644 index 000000000..c8de3c873 --- /dev/null +++ b/en/features/datasets/README.md @@ -0,0 +1,134 @@ +# Knowledge Import + +Most language models use outdated training data and have length limitations for the context of each request. For example, GPT-3.5 is trained on corpora from 2021 and has a limit of approximately 4k tokens per request. This means that developers who want their AI applications to be based on the latest and private context conversations must use techniques like embedding. + +Dify' knowledge feature allows developers (and even non-technical users) to easily manage knowledge and automatically integrate them into AI applications. All you need to do is prepare text content, such as: + +* Long text content (TXT, Markdown, DOCX, HTML, JSONL, or even PDF files) +* Structured data (CSV, Excel, etc.) + +Additionally, we are gradually supporting syncing data from various data sources to knowledge, including: + +* GitHub +* Databases +* Webpages +* ... + +{% hint style="info" %} +**Practice**: If your company wants to build an AI customer service assistant based on existing knowledge bases and product documentation, you can upload the documents to a knowledge base in Dify and create a conversational application. This might have taken you several weeks in the past and been difficult to maintain continuously. +{% endhint %} + +### Knowledge and Documents + +In Dify, knowledge bases are collections of documents. A knowledge base can be integrated as a whole into an application to be used as context. Documents can be uploaded by developers or operations staff, or synced from other data sources (typically corresponding to a file unit in the data source). + +**Steps to upload a document:** + +1. Upload your file, usually a long text file or a spreadsheet +2. Segment, clean, and preview +3. Dify submits it to the LLM provider for embedding as vector data and storage +4. Set metadata for the document +5. Ready to use in the application! + +#### How to write a good knowledge description + +When multiple knowledge bases are referenced in an application, AI uses the description of the knowledge and the user's question to determine which knowledge base to use to answer the user's question. Therefore, a well-written knowledge description can improve the accuracy of AI in selecting knowledge. + +The key to writing a good knowledge description is to clearly describe the content and characteristics of the knowledge. **It is recommended that the knowledge description begin with this: `Useful only when the question you want to answer is about the following: specific description`**. Here is an example of a real estate knowledge description: + +> Useful only when the question you want to answer is about the following: global real estate market data from 2010 to 2020. This data includes information such as the average housing price, property sales volume, and housing types for each city. In addition, this knowledge base also includes some economic indicators such as GDP and unemployment rate, as well as some social indicators such as population and education level. These indicators can help analyze the trends and influencing factors of the real estate market. With this data, we can understand the development trends of the global real estate market, analyze the changes in housing prices in various cities, and understand the impact of economic and social factors on the real estate market. + +### Create a knowledge + +1. Click on knowledge in the main navigation bar of Dify. On this page, you can see the existing knowledge bases. Click on "Create Knowledge" to enter the creation wizard. +2. If you have already prepared your files, you can start by uploading the files. +3. If you haven't prepared your documents yet, you can create an empty knowledge base first. + +### Uploading Documents By upload file + +1. Select the file you want to upload.We support batch uploads +2. Preview the full text +3. Perform segmentation and cleaning +4. Wait for Dify to process the data for you; this step usually consumes tokens in the LLM provider + +### Text Preprocessing and Cleaning + +Text Preprocessing and cleaning refers to Dify automatically segmenting and vectorizing your data documents so that user's questions (input) can match relevant paragraphs (Q to P), and generate results. + +When uploading a knowledge base, you need to select a **indexing mode** to specify how data is matched. This affects the accuracy of AI replies. + +In **High Quality mode**, OpenAI's embedding API is used for higher accuracy in user queries. + +In **Economic mode**, offline vector engines, keyword indexing etc. are used to reduce costs at the expense of lower accuracy. + +In **Segmenting in Question & Answer format**, instead of normal "Q to P" (question matches paragraphs), it uses "Q to Q" (question matches question) matching. After segmentation, Q\&A pairs are generated for each passage. When users ask questions, the system finds the most similar question and returns the corresponding passage as the answer. This is more precise because it directly matches the user's question and retrieves the information they need. + +> Questions have complete syntax while keywords lack semantics and context. So Q to Q improves clarity and handles similar high-frequency questions better. + +

In Segmenting in Question & Answer format, the text is summarized into multiple QA pairs

+ +

The difference between Q to P and Q to Q indexing modes

+ +### Modify Documents + +Modify Documents For technical reasons, if developers make the following changes to documents, Dify will create a new document for you, and the old document will be archived and deactivated: + +1. Adjust segmentation and cleaning settings +2. Re-upload the file + +Dify support customizing the segmented and cleaned text by adding, deleting, and editing paragraphs. You can dynamically adjust your segmentation to make your knowledge more accurate. Click **Document --> paragraph --> Edit** in the knowledge to modify paragraphs content and custom keywords. Click **Document --> paragraph --> Add segment --> Add a segment** to manually add new paragraph. Or click **Document --> paragraph --> Add segment --> Batch add** to batch add new paragraph. + +

Edit

+ +

add

+ +### Disabling and Archiving of Documents + +* **Disable, cancel disable**: The knowledge supports disabling documents or chunks that you temporarily do not want indexed. In the knowledge's document list, click the Disable button and the document will be disabled. You can also click the Disable button in the document details to disable the entire document or a segment. Disabled documents will not be indexed. To cancel the disable, click Enable on a disabled document. +* **Archive, Unarchive:** Some unused old document data can be archived if you don't want to delete it. After archiving, the data can only be viewed or deleted, not edited. In the document list of the knowledge, click the Archive button to archive the document. Documents can also be archived in the document details page. Archived documents will not be indexed. Archived documents can also be unarchived by clicking the Unarchive button. + +### Maintain Knowledge via API + +Head to [maintain-dataset-via-api.md](maintain-dataset-via-api.md "mention"). + +### Knowledge Settings + +Click **Settings** in the left navigation of the knowledge. You can change the following settings for the knowledge: + +* Knowledge **name** for identifying a knowledge base +* Knowledge **description** to allow AI to better use the knowledge appropriately. If the description is empty, Dify's automatic indexing strategy will be used. +* **Permissions** can be set to Only Me or All Team Members. Those without permissions cannot view and edit the knowledge. +* **Indexing mode**: In High Quality mode, OpenAI's embedding interface will be called to process and provide higher accuracy when users query. In Economic mode, offline vector engines, keyword indexing, etc. will be used to reduce accuracy without consuming tokens. + +Note: Upgrading the indexing mode from Economic to High Quality will incur additional token consumption. Downgrading from High Quality to Economic will not consume tokens. + +### Integrate into Applications + +Once the knowledge base is ready, it needs to be integrated into the application. When the AI application processes will automatically use the associated knowledge content as a reference context. + +1. Go to the application - Prompt Arrangement page +2. In the context options, select the knowledge you want to integrate +3. Save the settings to complete the integration + +### Q\&A + +**Q: What should I do if the PDF upload is garbled?** + +A: If your PDF parsing appears garbled under certain formatted contents, you could consider converting the PDF to Markdown format, which currently offers higher accuracy, or you could reduce the use of images, tables, and other formatted content in the PDF. We are researching ways to optimize the experience of using PDFs. + +**Q: How does the consumption mechanism of context work?** A: With a knowledge base added, each query will consume segmented content (currently embedding two chunks) + question + prompt + chat history combined. However, it will not exceed model limitations, such as 4096. + +**Q: Where does the embedded knowledge appear when asking questions?** A: It will be embedded as context before the question. + +**Q: Is there any priority between the added knowledge and OpenAI's answers?** A: The knowledge serves as context and is used together with questions for LLM to understand and answer; there is no priority relationship. + +**Q: Why can I hit in test but not in application?** A: You can troubleshoot issues by following these steps: + +1. Make sure you have added text on the prompt page and clicked on the save button in the top right corner. +2. Test whether it responds normally in the prompt debugging interface. +3. Try again in a new WebApp session window. +4. Optimize your data format and quality. For practice reference, visit [https://github.com/langgenius/dify/issues/90](https://github.com/langgenius/dify/issues/90) If none of these steps solve your problem, please join our community for help. + +**Q: Will APIs related to hit testing be opened up so that dify can access knowledge bases and implement dialogue generation using custom models?** A: We plan to open up Webhooks later on; however, there are no current plans for this feature. You can achieve your requirements by connecting to any vector database. + +**Q: How do I add multiple knowledge bases?** A: Due to short-term performance considerations, we currently only support one knowledge base. If you have multiple sets of data, you can upload them within the same knowledge base for use. diff --git a/en/features/datasets/maintain-dataset-via-api.md b/en/features/datasets/maintain-dataset-via-api.md new file mode 100644 index 000000000..b291b6f75 --- /dev/null +++ b/en/features/datasets/maintain-dataset-via-api.md @@ -0,0 +1,152 @@ +# Maintain Knowledge via API + +> Authentication, invocation method and application Service API remain consistent. The difference is that a knowledge API token can operate on all knowledge bases. + +### Benefits of Using the Knowledge API +* Sync your data systems to Dify knowledge to create powerful workflows. +* Provide knowledge list and document list APIs as well as detail query interfaces, to facilitate building your own data management page. +* Support both plain text and file uploads/updates documents, as well as batch additions and modifications, to simplify your sync process. +* Reduce manual document handling and syncing time, improving visibility of Dify's software and services. + +### How to use + +Please go to the knowledge page, you can switch tap to the API page in the navigation on the left side. On this page, you can view the API documentation provided by Dify and manage credentials for accessing the Knowledge API. + +

Knowledge API Document

+ +## **Create Empty Knowledge** + +**`POST /datasets`** + +{% hint style="warning" %} +Used only to create an empty dataset +{% endhint %} + +``` +curl --location --request POST 'https://api.dify.ai/v1/datasets' \ +--header 'Authorization: Bearer {api_key}' \ +--header 'Content-Type: application/json' \ +--data-raw '{"name": "name"}' + +``` + +#### **List of Knowledge** + + +``` +curl --location --request GET 'https://api.dify.ai/v1/datasets?page=1&limit=20' \ +--header 'Authorization: Bearer {api_key}' + +``` + +#### **Create A Document From Text** + +``` +curl --location --request POST '/document/create_by_text>' \\ +--header 'Authorization: Bearer {api_key}' \\ +--header 'Content-Type: application/json' \\ +--data-raw '{ + "name": "Dify", + "text": "Dify means Do it for you...", + "indexing_technique": "high_quality", + "process_rule": { + "rules": { + "pre_processing_rules": [{ + "id": "remove_extra_spaces", + "enabled": true + }, { + "id": "remove_urls_emails", + "enabled": true + }], + "segmentation": { + "separator": "###", + "max_tokens": 500 + } + }, + "mode": "custom" + } +}' + +``` + +#### **Create A Document From File** + +``` +curl --location POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create_by_file' \ +--header 'Authorization: Bearer {api_key}' \ +--form 'data="{ + "name": "Dify", + "indexing_technique": "high_quality", + "process_rule": { + "rules": { + "pre_processing_rules": [{ + "id": "remove_extra_spaces", + "enabled": true + }, { + "id": "remove_urls_emails", + "enabled": true + }], + "segmentation": { + "separator": "###", + "max_tokens": 500 + } + }, + "mode": "custom" + } + }"; + type=text/plain' \ +--form 'file=@"/path/to/file"' + +``` + +#### **Get Document Embedding Status** + +``` +curl --location --request GET 'https://api.dify.ai/v1/datasets/{dataset_id}/documents/{batch}/indexing-status' \ +--header 'Authorization: Bearer {api_key}' +``` + + +#### **Delete Document** + +``` +curl --location --request DELETE 'https://api.dify.ai/v1/datasets/{dataset_id}/documents/{document_id}' \ +--header 'Authorization: Bearer {api_key}' +``` + +#### **Get Document List** + +``` +curl --location --request GET 'https://api.dify.ai/v1/datasets/{dataset_id}/documents' \ +--header 'Authorization: Bearer {api_key}' + +``` + +#### **Add New Segment** + +``` +curl 'https://api.dify.ai/v1/datasets/aac47674-31a8-4f12-aab2-9603964c4789/documents/2034e0c1-1b75-4532-849e-24e72666595b/segment' \ + --header 'Authorization: Bearer {api_key}' \ + --header 'Content-Type: application/json' \ + --data-raw $'"segments":[ + {"content":"Dify means Do it for you", + "keywords":["Dify","Do"] + } + ]' + --compressed + +``` + + +### Error Message + +- `document_indexing`，document is in indexing status +- `provider_not_initialize`， Embedding model is not configured +- `not_found`，document not exist +- `dataset_name_duplicate` ，have existing knowledge name +- `provider_quota_exceeded`，The model quota has exceeded the limit +- `dataset_not_initialized`，The knowledge has not been initialized +- `unsupported_file_type`，Unsupported file type + - support file type：txt, markdown, md, pdf, html, htm, xlsx, docx, csv +- `too_many_files`，The number of files is too large, and only single file upload is temporarily supported +- `file_too_large`，The file is too large, supporting files under 15M \ No newline at end of file diff --git a/en/features/datasets/sync-from-notion.md b/en/features/datasets/sync-from-notion.md new file mode 100644 index 000000000..3d2c4d2a9 --- /dev/null +++ b/en/features/datasets/sync-from-notion.md @@ -0,0 +1,74 @@ +# Sync from Notion + +Dify knowledge supports importing from Notion and setting up **Sync** so that data is automatically synced to Dify after updates in Notion. + +### Authorization verification + +1. When creating a knowledge base, select the data source, click **Sync from Notion--Go to connect**, and complete the authorization verification according to the prompt. +2. You can also: click **Settings--Data Sources--Add a Data Source**, click Notion Source **Connect** to complete authorization verification. + +

Connect Notion

+ +### Import Notion data + +After completing authorization verification, go to the knowledge creation page, click **Sync from Notion**, and select the required authorization page to import. + +### Segmentation and cleaning + +Next, select your **segmentation settings** and **indexing method**, **save and process**. Wait for Dify to process this data, usually this step requires token consumption in LLM providers. Dify not only supports importing ordinary page types but also summarizes and saves the page attributes under the database type. + +_**Note: Images and files are not currently supported for import. Table data will be converted to text.**_ + +### Sync Notion data + +If your Notion content has been modified, you can click Sync directly on the Dify knowledge document list page to sync the data with one click(Please note that each time you click, the current content will be synchronized). This step requires token consumption. + +

Sync Notion data

+ +### (Community Edition) Notion Integration Configuration Guide + +Notion integration is divided into two ways: **internal integration** and **public integration** . It can be configured in Dify on demand. + +For the specific differences between the two integration methods, please refer to the [official doc of Notion](https://developers.notion.com/docs/authorization). + +#### 1. **Use internal integration** + +Create an integration in your [integration's settings](https://www.notion.so/my-integrations) page. By default, all integrations start with an internal integration; internal integrations will be associated with a workspace of your choice, so you need to be the workspace owner to create an integration. + +**Specific operation steps:** + +Click the " **New integration** " button, the type is Internal by default (cannot be modified), select the associated space, enter the name and upload the logo, and click "**Submit**" to create the integration successfully. + +
+ +Once the integration is created, you can update its settings as needed under the **Capabilities** tab and click the "**Show**" button under **Secrets** and then copy the Secrets. + +
+ +Copy it and back to the Dify source code , in the **.env** file configuration related environment variables, environment variables as follows: + +**NOTION\_INTEGRATION\_TYPE** = internal or **NOTION\_INTEGRATION\_TYPE** = public + +**NOTION\_INTERNAL\_SECRET**=you-internal-secret + +#### 2. **Use public integration** + +**You need to upgrade the internal integration to public integration** , navigate to the integrated Distribution page, and toggle the switch to expose the integration. + +To toggle the switch to public settings, you need to **fill in additional information in the Organization Information** form below, including your company name, website, and Retargeting URL, and click the "Submit" button. + +
+ +After your integration has been successfully made public in your [integration’s settings page](https://www.notion.so/my-integrations), you will be able to access the integration’s secrets in the Secrets tab. + +
+ +Back to the Dify source code , in the **.env** file configuration related environment variables , environment variables as follows: + +**NOTION\_INTEGRATION\_TYPE**=public + +**NOTION\_CLIENT\_SECRET**=you-client-secret + +**NOTION\_CLIENT\_ID**=you-client-id + +Once configured, you will be able to utilize Notion data import and sync functions in the knowledge section. From 4613e64b843d1bdd65b08685c94a4c9afb21f1ca Mon Sep 17 00:00:00 2001 From: AllenWriter Date: Sun, 27 Apr 2025 15:25:24 +0800 Subject: [PATCH 2/2] Docs: modify default value of INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH --- en/getting-started/install-self-hosted/environments.md | 2 +- jp/getting-started/install-self-hosted/environments.md | 2 +- zh_CN/getting-started/install-self-hosted/environments.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/en/getting-started/install-self-hosted/environments.md b/en/getting-started/install-self-hosted/environments.md index d8613474c..1fc8224d0 100644 --- a/en/getting-started/install-self-hosted/environments.md +++ b/en/getting-started/install-self-hosted/environments.md @@ -723,7 +723,7 @@ Used to set the browser policy for session cookies used for identity verificatio #### INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH -Configuration for document chunk length. It is used to control the size of text segments when processing long documents. Default: 500. Maximum: 4000. +Configuration for document chunk length. It is used to control the size of text segments when processing long documents. Default: 4000. **Larger Chunks** - Retain more context within each chunk, ideal for tasks requiring a broader understanding of the text. diff --git a/jp/getting-started/install-self-hosted/environments.md b/jp/getting-started/install-self-hosted/environments.md index 6e97f9922..6f02c0c62 100644 --- a/jp/getting-started/install-self-hosted/environments.md +++ b/jp/getting-started/install-self-hosted/environments.md @@ -712,7 +712,7 @@ API サービスによってインターフェース ID 検証にのみ使用さ #### INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH -文書チャンク長の設定。長文処理時のテキストセグメントサイズを制御するために使用。デフォルト値：500。最大値：4000。 +文書チャンク長の設定。長文処理時のテキストセグメントサイズを制御するために使用。デフォルト値：4000。 **大きなチャンク** - 単一のチャンク内により多くの文脈を保持でき、複雑または文脈依存のタスクに適しています。 diff --git a/zh_CN/getting-started/install-self-hosted/environments.md b/zh_CN/getting-started/install-self-hosted/environments.md index 119b12751..61920148f 100644 --- a/zh_CN/getting-started/install-self-hosted/environments.md +++ b/zh_CN/getting-started/install-self-hosted/environments.md @@ -692,7 +692,7 @@ WebApp Url，用于显示文件预览或下载 URL 到前端作为多模型输 #### INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH -文档分段长度配置，用于控制处理长文本时的分段大小。默认值：500。最大值：4000。 +文档分段长度配置，用于控制处理长文本时的分段大小。默认值：4000。 **较大分段** - 可在单个分段内保留更多上下文，适合需要处理复杂或上下文相关任务的场景。