knovuslab · ba5liel · Jul 29, 2023 · Jul 29, 2023 · Jul 29, 2023 · Jul 30, 2023
diff --git a/.github/workflows/deploy-to-node.yaml b/.github/workflows/deploy-to-node.yaml
@@ -81,6 +81,7 @@ jobs:
       BACKEND_CORS_ORIGINS: ${{ vars.BACKEND_CORS_ORIGINS }}
       WEB_INFERENCE_SERVER_HOST: ${{ vars.WEB_INFERENCE_SERVER_HOST }}
       WEB_ENABLE_CHAT: ${{ vars.WEB_ENABLE_CHAT }}
+      WEB_BYE: ${{ vars.WEB_BYE }}
       WEB_ENABLE_DRAFTS_WITH_PLUGINS: ${{ vars.WEB_ENABLE_DRAFTS_WITH_PLUGINS }}
       WEB_NUM_GENERATED_DRAFTS: ${{ vars.WEB_NUM_GENERATED_DRAFTS }}
       WEB_CURRENT_ANNOUNCEMENT: ${{ vars.WEB_CURRENT_ANNOUNCEMENT }}

diff --git a/README.md b/README.md
@@ -3,6 +3,10 @@
   <img width="auto" height="50px" src="https://github.com/LAION-AI/Open-Assistant/blob/main/assets/logo_crop.png"/>
 </h1>
 
+<blockquote>
+<p>:memo: <strong>NOTE</strong>: OpenAssistant is completed, and the project is now finished. Thank you to everyone who contributed! Check out our <a href="https://projects.laion.ai/Open-Assistant/blog/2023/10/25/open-assistant-is-completed">blog post</a> for more information. The final published oasst2 dataset can be found on HuggingFace at <a href="https://huggingface.co/datasets/OpenAssistant/oasst2">OpenAssistant/oasst2</a></p>
+</blockquote>
+
 <div align="center">
 
 <a href="https://github.com/LAION-AI/Open-Assistant/stargazers">![GitHub Repo stars](https://img.shields.io/github/stars/LAION-AI/Open-Assistant?style=social)</a>

diff --git a/ansible/deploy-to-node.yaml b/ansible/deploy-to-node.yaml
@@ -284,6 +284,7 @@
           INFERENCE_SERVER_API_KEY:
             "{{ lookup('ansible.builtin.env', 'WEB_INFERENCE_SERVER_API_KEY') }}"
           ENABLE_CHAT: "{{ lookup('ansible.builtin.env', 'WEB_ENABLE_CHAT') }}"
+          BYE: "{{ lookup('ansible.builtin.env', 'WEB_BYE') }}"
           ENABLE_DRAFTS_WITH_PLUGINS:
             "{{ lookup('ansible.builtin.env',
             'WEB_ENABLE_DRAFTS_WITH_PLUGINS')}}"

diff --git a/backend/requirements.txt b/backend/requirements.txt
@@ -10,8 +10,8 @@ loguru==0.6.0
 numpy>=1.23.2
 prometheus-fastapi-instrumentator==5.9.1
 psycopg2-binary==2.9.5
-pydantic==1.10.4
-pydantic[email]==1.10.4
+pydantic==1.10.7
+pydantic[email]==1.10.7
 python-dotenv==0.21.0
 python-jose[cryptography]==3.3.0
 redis==4.5.5

diff --git a/data/datasets/__init__.py b/data/datasets/__init__.py
@@ -4,6 +4,8 @@
     "tv_dialogue": "sedthh/tv_dialogue",  # TV and Movie dialogues and transcripts
     "fd_dialogue": "sedthh/fd_dialogue",  # TV and Movie dialogues and transcripts from ForeverDreaming
     "tlcv2.0_oa": "pythainlp/tlcv2.0_oa",  # Thai classical literature texts
+    "fa-isna-news": "pourmand1376/isna-news",  # Isna Persian News
+    "fa-wikipedia": "pourmand1376/fa-wikipedia",  # Farsi Wikipedia texts
 }
 
 INSTRUCTION_DATASETS = {

diff --git a/data/datasets/fa-isna-news/README.md b/data/datasets/fa-isna-news/README.md
@@ -0,0 +1,2 @@
+This text-only dataset is crawled from [Isna news](https://isna.ir/). This is
+biggest farsi news agency and thus the text is pretty clean.
diff --git a/data/datasets/fa-wikipedia/README.md b/data/datasets/fa-wikipedia/README.md
@@ -0,0 +1,6 @@
+This dataset is crawled from
+[farsi wikipedia](https://fa.wikipedia.org/wiki/%D8%B5%D9%81%D8%AD%D9%87%D9%94_%D8%A7%D8%B5%D9%84%DB%8C).
+This is valuable clean text data in persian (Farsi). It contains information
+about all subjects.
+
+It has 2.53M Articles.
diff --git a/docs/blog/2023-02-11-architecture.md b/docs/blog/2023-02-11-architecture.md
@@ -10,7 +10,7 @@ I thought it would make sense to document our architecture as a blog post, since
 it would be easily visible for everyone, and it would also be nice to look back
 a couple months from now and see how stuff has changed :)
 
-A lot of work was done in the past couple of week to establish the needed
+A lot of work was done in the past couple of weeks to establish the needed
 infrastructure both for data collection and inference purposes. Thanks to
 everyone, and especially community members who supported us so far!
 

diff --git a/docs/blog/2023-10-25-open-assistant-is-completed/index.mdx b/docs/blog/2023-10-25-open-assistant-is-completed/index.mdx
@@ -0,0 +1,20 @@
+---
+title: OpenAssistant is Completed!
+description: OpenAssistant is Completed!
+authors: [yk]
+tags: [open-assistant, youtube]
+image: https://img.youtube.com/vi/gqtmUHhaplo/0.jpg
+---
+
+import ReactPlayer from "react-player";
+
+<ReactPlayer
+  controls
+  width="100%"
+  url="https://www.youtube.com/embed/gqtmUHhaplo"
+/>
+
+The final published oasst2 dataset can be found on HuggingFace at
+[OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2).
+
+<!--truncate-->
diff --git a/docs/docs/faq.md b/docs/docs/faq.md
@@ -15,12 +15,12 @@ In this page, there are some of the most frequently asked questions.
 
 </summary>
 
-We have released candidate supervised finetuning (SFT) models using both Pythia
-and LLaMa, as well as candidate reward models for reinforcement learning from
-human feedback training using Pythia, which you can try, and are beginning the
-process of applying (RLHF). We have also released the first version of the
-OpenAssistant Conversations dataset
-[here](https://huggingface.co/datasets/OpenAssistant/oasst1).
+This project has concluded. We have released supervised finetuning (SFT) models
+using Llama 2, LLaMa, Falcon, Pythia, and StabeLM as well as reinforcement
+learning from human feedback trained models and reward models, all of which are
+available at [here](https://huggingface.co/OpenAssistant). In addition to our
+models, we have released three datasets from OpenAssistant conversations, and a
+[research paper](https://arxiv.org/abs/2304.07327).
 
 </details>
 
@@ -31,9 +31,8 @@ OpenAssistant Conversations dataset
 
 </summary>
 
-You can play with our best candidate model
-[here](https://open-assistant.io/chat) and provide thumbs up/down responses to
-help us improve the model in future!
+Our online demonstration is no longer available, but the models remain available
+to download [here](https://huggingface.co/OpenAssistant).
 
 </details>
 
@@ -44,37 +43,18 @@ help us improve the model in future!
 
 </summary>
 
-The candidate Pythia SFT models are
+All of our models are
 [available on HuggingFace](https://huggingface.co/OpenAssistant) and can be
-loaded via the HuggingFace Transformers library. As such you may be able to use
-them with sufficient hardware. There are also spaces on HF which can be used to
-chat with the OA candidate without your own hardware. However, these models are
-not final and can produce poor or undesirable outputs.
+loaded via the HuggingFace Transformers library or other runners if converted.
+As such you may be able to use them with sufficient hardware. There are also
+spaces on HF which can be used to chat with the OA candidate without your own
+hardware. However, some of these models are not final and can produce poor or
+undesirable outputs.
 
-LLaMa SFT models cannot be released directly due to Meta's license but XOR
+LLaMa (v1) SFT models cannot be released directly due to Meta's license but XOR
 weights are released on the HuggingFace org. Follow the process in the README
-there to obtain a full model from these XOR weights.
-
-</details>
-
-<details>
-<summary>
-
-### Is there an API available?
-
-</summary>
-
-There is no API currently available for Open Assistant. Any mention of an API in
-documentation is referencing the website's internal API. We understand that an
-API is a highly requested feature, but unfortunately, we can't provide one at
-this time due to a couple of reasons. Firstly, the inference system is already
-under high load and running off of compute from our sponsors. Secondly, the
-project's primary goal is currently data collection and model training, not
-providing a product.
-
-However, if you're looking to run inference, you can host the model yourself
-either on your own hardware or with a cloud provider. We appreciate your
-understanding and patience as we continue to develop this project.
+there to obtain a full model from these XOR weights. Llama 2 models are not
+required to be XORed.
 
 </details>
 
@@ -102,15 +82,13 @@ inference setup and UI locally unless you wish to assist in development.
 All Open Assistant code is licensed under Apache 2.0. This means it is available
 for a wide range of uses including commercial use.
 
-The Open Assistant Pythia based models are released as full weights and will be
-licensed under the Apache 2.0 license.
-
-The Open Assistant LLaMa based models will be released only as delta weights
-meaning you will need the original LLaMa weights to use them, and the license
-restrictions will therefore be those placed on the LLaMa weights.
+Open Assistant models are released under the license of their respective base
+models, be that Llama 2, Falcon, Pythia, or StableLM. LLaMa (not 2) models are
+only released as XOR weights, meaning you will need the original LLaMa weights
+to use them.
 
-The Open Assistant data is released under a Creative Commons license allowing a
-wide range of uses including commercial use.
+The Open Assistant data is released under Apache-2.0 allowing a wide range of
+uses including commercial use.
 
 </details>
 
@@ -138,9 +116,8 @@ you to everyone who has taken part!
 
 </summary>
 
-The model code, weights, and data are free. We are additionally hosting a free
-public instance of our best current model for as long as we can thanks to
-compute donation from Stability AI via LAION!
+The model code, weights, and data are free. Our free public instance of our best
+models is not longer available due to the project's conclusion.
 
 </details>
 
@@ -151,10 +128,9 @@ compute donation from Stability AI via LAION!
 
 </summary>
 
-The current smallest (Pythia) model is 12B parameters and is challenging to run
-on consumer hardware, but can run on a single professional GPU. In future there
-may be smaller models and we hope to make progress on methods like integer
-quantisation which can help run the model on smaller hardware.
+The current smallest models are 7B parameters and are challenging to run on
+consumer hardware, but can run on a single professional GPU or be quantized to
+run on more widely available hardware.
 
 </details>
 
@@ -165,13 +141,7 @@ quantisation which can help run the model on smaller hardware.
 
 </summary>
 
-If you want to help in the data collection for training the model, go to the
-website [https://open-assistant.io/](https://open-assistant.io/).
-
-If you want to contribute code, take a look at the
-[tasks in GitHub](https://github.com/orgs/LAION-AI/projects/3) and comment on an
-issue stating your wish to be assigned. You can also take a look at this
-[contributing guide](https://github.com/LAION-AI/Open-Assistant/blob/main/CONTRIBUTING.md).
+This project has now concluded.
 
 </details>
 
@@ -190,104 +160,6 @@ well as accelerate, DeepSpeed, bitsandbytes, NLTK, and other libraries.
 
 </details>
 
-## Questions about the data collection website
-
-<details>
-<summary>
-
-### Can I use ChatGPT to help in training Open Assistant, for instance, by generating answers?
-
-</summary>
-
-No, it is against their terms of service to use it to help train other models.
-See
-[this issue](https://github.com/LAION-AI/Open-Assistant/issues/471#issuecomment-1374392299).
-ChatGPT-like answers will be removed.
-
-</details>
-
-<details>
-<summary>
-
-### What should I do if I don't know how to complete the task as an assistant?
-
-</summary>
-Skip it.
-</details>
-
-<details>
-<summary>
-
-### Should I fact check the answers by the assistant?
-
-</summary>
-
-Yes, you should try. If you are not sure, skip the task.
-
-</details>
-
-<details>
-<summary>
-
-### How can I see my score?
-
-</summary>
-
-In your [account settings](https://open-assistant.io/account).
-
-</details>
-
-<details>
-<summary>
-
-### Can we see how many data points have been collected?
-
-</summary>
-
-You can see a regularly updated interface at
-[https://open-assistant.io/stats](https://open-assistant.io/stats).
-
-</details>
-
-<details>
-<summary>
-
-### How do I write and label prompts?
-
-</summary>
-
-Check the
-[guidelines](https://projects.laion.ai/Open-Assistant/docs/guides/guidelines).
-
-</details>
-
-<details>
-<summary>
-
-### Where can I report a bug or create a new feature request?
-
-</summary>
-
-In the [GitHub issues](https://github.com/LAION-AI/Open-Assistant/issues).
-
-</details>
-
-<details>
-<summary>
-
-### Why am I not allowed to write about this topic, even though it isn't illegal?
-
-</summary>
-
-We want to ensure that the Open Assistant dataset is as accessible as possible.
-As such, it's necessary to avoid any harmful or offensive content that could be
-grounds for removal on sites such as Hugging Face. Likewise, we want the model
-to be trained to reject as few questions as possible, so it's important to not
-include prompts that leave the assistant with no other choice but to refuse in
-order to avoid the generation of harmful content.
-
-</details>
-
 ## Questions about the development process
 
 <details>

diff --git a/docs/docs/intro.md b/docs/docs/intro.md
@@ -1,3 +1,9 @@
+# Notice
+
+**Open Assistant has now concluded.** Please see
+[this video](https://www.youtube.com/watch?v=gqtmUHhaplo) for more information.
+Thanks you to all those who made this project possible.
+
 # Introduction
 
 > The FAQ page is available at

diff --git a/inference/server/oasst_inference_server/routes/chats.py b/inference/server/oasst_inference_server/routes/chats.py
@@ -153,6 +153,8 @@ async def create_assistant_message(
                 system_prompt=request.system_prompt,
                 plugins=request.plugins,
                 plugin_max_depth=settings.plugin_max_depth,
+                user_profile=request.user_profile,
+                user_response_instructions=request.user_response_instructions,
             )
             assistant_message = await ucr.initiate_assistant_message(
                 parent_id=request.parent_id,
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		This text-only dataset is crawled from [Isna news](https://isna.ir/). This is
		biggest farsi news agency and thus the text is pretty clean.