From 43cd455b5244bd0ba2be6afd694a6f93b0900e0e Mon Sep 17 00:00:00 2001 From: writinwaters <93570324+writinwaters@users.noreply.github.com> Date: Thu, 18 Jul 2024 11:20:15 +0800 Subject: [PATCH] Updated deploy a local llm using IPEX-LLM (#1578) ### What problem does this PR solve? ### Type of change - [x] Documentation Update --- README.md | 2 +- README_ja.md | 2 +- README_zh.md | 2 +- ...ploy_local_llm.md => deploy_local_llm.mdx} | 125 +++++++++++------- docs/guides/manage_files.md | 2 +- 5 files changed, 82 insertions(+), 51 deletions(-) rename docs/guides/{deploy_local_llm.md => deploy_local_llm.mdx} (65%) diff --git a/README.md b/README.md index 00729b01..16fe2421 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ Try our demo at [https://demo.ragflow.io](https://demo.ragflow.io). -## 📌 Latest Updates +## 🔥 Latest Updates - 2024-07-08 Supports workflow based on [Graph](./graph/README.md). - 2024-06-27 Supports Markdown and Docx in the Q&A parsing method. diff --git a/README_ja.md b/README_ja.md index fcac9b08..b0897951 100644 --- a/README_ja.md +++ b/README_ja.md @@ -45,7 +45,7 @@ -## 📌 最新情報 +## 🔥 最新情報 - 2024-07-08 [Graph](./graph/README.md) ベースのワークフローをサポート - 2024-06-27 Q&A解析方式はMarkdownファイルとDocxファイルをサポートしています。 - 2024-06-27 Docxファイルからの画像の抽出をサポートします。 diff --git a/README_zh.md b/README_zh.md index 53e05af7..7902c600 100644 --- a/README_zh.md +++ b/README_zh.md @@ -44,7 +44,7 @@ -## 📌 近期更新 +## 🔥 近期更新 - 2024-07-08 支持 Agentic RAG: 基于 [Graph](./graph/README.md) 的工作流。 - 2024-06-27 Q&A 解析方式支持 Markdown 文件和 Docx 文件。 diff --git a/docs/guides/deploy_local_llm.md b/docs/guides/deploy_local_llm.mdx similarity index 65% rename from docs/guides/deploy_local_llm.md rename to docs/guides/deploy_local_llm.mdx index 419bb274..ad817390 100644 --- a/docs/guides/deploy_local_llm.md +++ b/docs/guides/deploy_local_llm.mdx @@ -4,6 +4,8 @@ slug: /deploy_local_llm --- # Deploy a local LLM +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; RAGFlow supports deploying models locally using Ollama or Xinference. If you have locally deployed models to leverage or wish to enable GPU or CUDA for inference acceleration, you can bind Ollama or Xinference into RAGFlow and use either of them as a local "server" for interacting with your local models. @@ -108,7 +110,7 @@ Update your chat model accordingly in **Chat Configuration**: ## Deploy a local model using Xinference -Xorbits Inference([Xinference](https://github.com/xorbitsai/inference)) enables you to unleash the full potential of cutting-edge AI models. +Xorbits Inference ([Xinference](https://github.com/xorbitsai/inference)) enables you to unleash the full potential of cutting-edge AI models. :::note - For information about installing Xinference Ollama, see [here](https://inference.readthedocs.io/en/latest/getting_started/). @@ -129,8 +131,8 @@ $ xinference-local --host 0.0.0.0 --port 9997 ### 3. Launch your local model -Launch your local model (**Mistral**), ensuring that you replace `${quantization}` with your chosen quantization method -: +Launch your local model (**Mistral**), ensuring that you replace `${quantization}` with your chosen quantization method: + ```bash $ xinference launch -u mistral --model-name mistral-v0.1 --size-in-billions 7 --model-format pytorch --quantization ${quantization} ``` @@ -161,9 +163,9 @@ Update your chat model accordingly in **Chat Configuration**: ## Deploy a local model using IPEX-LLM -IPEX-LLM([IPEX-LLM](https://github.com/intel-analytics/ipex-llm)) is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency +[IPEX-LLM](https://github.com/intel-analytics/ipex-llm) is a PyTorch library for running LLMs on local Intel CPUs or GPUs (including iGPU or discrete GPUs like Arc, Flex, and Max) with low latency. It supports Ollama on Linux and Windows systems. -To deploy a local model, eg., **Qwen2**, using IPEX-LLM, follow the steps below: +To deploy a local model, e.g., **Qwen2**, using IPEX-LLM-accelerated Ollama: ### 1. Check firewall settings @@ -173,46 +175,69 @@ Ensure that your host machine's firewall allows inbound connections on port 1143 sudo ufw allow 11434/tcp ``` -### 2. Install and Start Ollama serve using IPEX-LLM +### 2. Launch Ollama service using IPEX-LLM #### 2.1 Install IPEX-LLM for Ollama -IPEX-LLM's support for `ollama` now is available for Linux system and Windows system. +:::tip NOTE +IPEX-LLM's supports Ollama on Linux and Windows systems. +::: -Visit [Run llama.cpp with IPEX-LLM on Intel GPU Guide](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md), and follow the instructions in section [Prerequisites](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#0-prerequisites) to setup and section [Install IPEX-LLM cpp](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#1-install-ipex-llm-for-llamacpp) to install the IPEX-LLM with Ollama binaries. +For detailed information about installing IPEX-LLM for Ollama, see [Run llama.cpp with IPEX-LLM on Intel GPU Guide](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md): +- [Prerequisites](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#0-prerequisites) +- [Install IPEX-LLM cpp with Ollama binaries](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#1-install-ipex-llm-for-llamacpp) -**After the installation, you should have created a conda environment, named `llm-cpp` for instance, for running `ollama` commands with IPEX-LLM.** +*After the installation, you should have created a Conda environment, e.g., `llm-cpp`, for running Ollama commands with IPEX-LLM.* #### 2.2 Initialize Ollama -Activate the `llm-cpp` conda environment and initialize Ollama by executing the commands below. A symbolic link to `ollama` will appear in your current directory. +1. Activate the `llm-cpp` Conda environment and initialize Ollama: -- For **Linux users**: + + ```bash conda activate llm-cpp init-ollama ``` + + -- For **Windows users**: - - Please run the following command with **administrator privilege in Miniforge Prompt**. + Run these commands with *administrator privileges in Miniforge Prompt*: ```cmd conda activate llm-cpp init-ollama.bat ``` + + -> [!NOTE] -> If you have installed higher version `ipex-llm[cpp]` and want to upgrade your ollama binary file, don't forget to remove old binary files first and initialize again with `init-ollama` or `init-ollama.bat`. +2. If the installed `ipex-llm[cpp]` requires an upgrade to the Ollama binary files, remove the old binary files and reinitialize Ollama using `init-ollama` (Linux) or `init-ollama.bat` (Windows). + + *A symbolic link to Ollama appears in your current directory, and you can use this executable file following standard Ollama commands.* -**Now you can use this executable file by standard ollama's usage.** +#### 2.3 Launch Ollama service -#### 2.3 Run Ollama Serve +1. Set the environment variable `OLLAMA_NUM_GPU` to `999` to ensure that all layers of your model run on the Intel GPU; otherwise, some layers may default to CPU. +2. For optimal performance on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), set the following environment variable before launching the Ollama service: -You may launch the Ollama service as below: + ```bash + export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 + ``` +3. Launch the Ollama service: -- For **Linux users**: + + ```bash export OLLAMA_NUM_GPU=999 @@ -224,9 +249,10 @@ You may launch the Ollama service as below: ./ollama serve ``` -- For **Windows users**: + + - Please run the following command in Miniforge Prompt. + Run the following command *in Miniforge Prompt*: ```cmd set OLLAMA_NUM_GPU=999 @@ -236,49 +262,54 @@ You may launch the Ollama service as below: ollama serve ``` + + -> Please set environment variable `OLLAMA_NUM_GPU` to `999` to make sure all layers of your model are running on Intel GPU, otherwise, some layers may run on CPU. +:::tip NOTE +To enable the Ollama service to accept connections from all IP addresses, use `OLLAMA_HOST=0.0.0.0 ./ollama serve` rather than simply `./ollama serve`. +::: - -> If your local LLM is running on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), it is recommended to additionaly set the following environment variable for optimal performance before executing `ollama serve`: -> -> ```bash -> export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 -> ``` - - -> To allow the service to accept connections from all IP addresses, use `OLLAMA_HOST=0.0.0.0 ./ollama serve` instead of just `./ollama serve`. - -The console will display messages similar to the following: +*The console displays messages similar to the following:* ![](https://llm-assets.readthedocs.io/en/latest/_images/ollama_serve.png) -### 3. Pull and Run Ollama Model +### 3. Pull and Run Ollama model -Keep the Ollama service on and open another terminal and run `./ollama pull ` in Linux (`ollama.exe pull ` in Windows) to automatically pull a model. e.g. `qwen2:latest`: +#### 3.1 Pull Ollama model + +With the Ollama service running, open a new terminal and run `./ollama pull ` (Linux) or `ollama.exe pull ` (Windows) to pull the desired model. e.g., `qwen2:latest`: ![](https://llm-assets.readthedocs.io/en/latest/_images/ollama_pull.png) -#### Run Ollama Model +#### 3.2 Run Ollama model + + + -- For **Linux users**: ```bash ./ollama run qwen2:latest ``` - -- For **Windows users**: + + + ```cmd ollama run qwen2:latest ``` -### 4. Configure RAGflow to use IPEX-LLM accelerated Ollama -The confiugraiton follows the steps in + + -Ollama Section 4 [Add Ollama](#4-add-ollama), +### 4. Configure RAGflow -Section 5 [Complete basic Ollama settings](#5-complete-basic-ollama-settings), +To enable IPEX-LLM accelerated Ollama in RAGFlow, you must also complete the configurations in RAGFlow. The steps are identical to those outlined in the *Deploy a local model using Ollama* section: -Section 6 [Update System Model Settings](#6-update-system-model-settings), - -Section 7 [Update Chat Configuration](#7-update-chat-configuration) +1. [Add Ollama](#4-add-ollama) +2. [Complete basic Ollama settings](#5-complete-basic-ollama-settings) +3. [Update System Model Settings](#6-update-system-model-settings) +4. [Update Chat Configuration](#7-update-chat-configuration) diff --git a/docs/guides/manage_files.md b/docs/guides/manage_files.md index b1d39a6b..d1c5e20a 100644 --- a/docs/guides/manage_files.md +++ b/docs/guides/manage_files.md @@ -43,7 +43,7 @@ You can link your file to one knowledge base or multiple knowledge bases at one ![link multiple kb](https://github.com/infiniflow/ragflow/assets/93570324/6c508803-fb1f-435d-b688-683066fd7fff) -## Move file to specified folder +## Move file to a specific folder As of RAGFlow v0.8.0, this feature is *not* available.