Updated deploy a local llm using IPEX-LLM (#1578)
### What problem does this PR solve? ### Type of change - [x] Documentation Update
This commit is contained in:
writinwaters
GitHub
parent
b54d5807f3
commit
43cd455b52
@ -63,7 +63,7 @@ Try our demo at [https://demo.ragflow.io](https://demo.ragflow.io).
|
||||
</div>
|
||||
|
||||
|
||||
## 📌 Latest Updates
|
||||
## 🔥 Latest Updates
|
||||
|
||||
- 2024-07-08 Supports workflow based on [Graph](./graph/README.md).
|
||||
- 2024-06-27 Supports Markdown and Docx in the Q&A parsing method.
|
||||
|
||||
@ -45,7 +45,7 @@
|
||||
</div>
|
||||
|
||||
|
||||
## 📌 最新情報
|
||||
## 🔥 最新情報
|
||||
- 2024-07-08 [Graph](./graph/README.md) ベースのワークフローをサポート
|
||||
- 2024-06-27 Q&A解析方式はMarkdownファイルとDocxファイルをサポートしています。
|
||||
- 2024-06-27 Docxファイルからの画像の抽出をサポートします。
|
||||
|
||||
@ -44,7 +44,7 @@
|
||||
</div>
|
||||
|
||||
|
||||
## 📌 近期更新
|
||||
## 🔥 近期更新
|
||||
|
||||
- 2024-07-08 支持 Agentic RAG: 基于 [Graph](./graph/README.md) 的工作流。
|
||||
- 2024-06-27 Q&A 解析方式支持 Markdown 文件和 Docx 文件。
|
||||
|
||||
@ -4,6 +4,8 @@ slug: /deploy_local_llm
|
||||
---
|
||||
|
||||
# Deploy a local LLM
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
RAGFlow supports deploying models locally using Ollama or Xinference. If you have locally deployed models to leverage or wish to enable GPU or CUDA for inference acceleration, you can bind Ollama or Xinference into RAGFlow and use either of them as a local "server" for interacting with your local models.
|
||||
|
||||
@ -108,7 +110,7 @@ Update your chat model accordingly in **Chat Configuration**:
|
||||
|
||||
## Deploy a local model using Xinference
|
||||
|
||||
Xorbits Inference([Xinference](https://github.com/xorbitsai/inference)) enables you to unleash the full potential of cutting-edge AI models.
|
||||
Xorbits Inference ([Xinference](https://github.com/xorbitsai/inference)) enables you to unleash the full potential of cutting-edge AI models.
|
||||
|
||||
:::note
|
||||
- For information about installing Xinference Ollama, see [here](https://inference.readthedocs.io/en/latest/getting_started/).
|
||||
@ -129,8 +131,8 @@ $ xinference-local --host 0.0.0.0 --port 9997
|
||||
|
||||
### 3. Launch your local model
|
||||
|
||||
Launch your local model (**Mistral**), ensuring that you replace `${quantization}` with your chosen quantization method
|
||||
:
|
||||
Launch your local model (**Mistral**), ensuring that you replace `${quantization}` with your chosen quantization method:
|
||||
|
||||
```bash
|
||||
$ xinference launch -u mistral --model-name mistral-v0.1 --size-in-billions 7 --model-format pytorch --quantization ${quantization}
|
||||
```
|
||||
@ -161,9 +163,9 @@ Update your chat model accordingly in **Chat Configuration**:
|
||||
|
||||
## Deploy a local model using IPEX-LLM
|
||||
|
||||
IPEX-LLM([IPEX-LLM](https://github.com/intel-analytics/ipex-llm)) is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency
|
||||
[IPEX-LLM](https://github.com/intel-analytics/ipex-llm) is a PyTorch library for running LLMs on local Intel CPUs or GPUs (including iGPU or discrete GPUs like Arc, Flex, and Max) with low latency. It supports Ollama on Linux and Windows systems.
|
||||
|
||||
To deploy a local model, eg., **Qwen2**, using IPEX-LLM, follow the steps below:
|
||||
To deploy a local model, e.g., **Qwen2**, using IPEX-LLM-accelerated Ollama:
|
||||
|
||||
### 1. Check firewall settings
|
||||
|
||||
@ -173,46 +175,69 @@ Ensure that your host machine's firewall allows inbound connections on port 1143
|
||||
sudo ufw allow 11434/tcp
|
||||
```
|
||||
|
||||
### 2. Install and Start Ollama serve using IPEX-LLM
|
||||
### 2. Launch Ollama service using IPEX-LLM
|
||||
|
||||
#### 2.1 Install IPEX-LLM for Ollama
|
||||
|
||||
IPEX-LLM's support for `ollama` now is available for Linux system and Windows system.
|
||||
:::tip NOTE
|
||||
IPEX-LLM's supports Ollama on Linux and Windows systems.
|
||||
:::
|
||||
|
||||
Visit [Run llama.cpp with IPEX-LLM on Intel GPU Guide](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md), and follow the instructions in section [Prerequisites](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#0-prerequisites) to setup and section [Install IPEX-LLM cpp](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#1-install-ipex-llm-for-llamacpp) to install the IPEX-LLM with Ollama binaries.
|
||||
For detailed information about installing IPEX-LLM for Ollama, see [Run llama.cpp with IPEX-LLM on Intel GPU Guide](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md):
|
||||
- [Prerequisites](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#0-prerequisites)
|
||||
- [Install IPEX-LLM cpp with Ollama binaries](https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#1-install-ipex-llm-for-llamacpp)
|
||||
|
||||
**After the installation, you should have created a conda environment, named `llm-cpp` for instance, for running `ollama` commands with IPEX-LLM.**
|
||||
*After the installation, you should have created a Conda environment, e.g., `llm-cpp`, for running Ollama commands with IPEX-LLM.*
|
||||
|
||||
#### 2.2 Initialize Ollama
|
||||
|
||||
Activate the `llm-cpp` conda environment and initialize Ollama by executing the commands below. A symbolic link to `ollama` will appear in your current directory.
|
||||
1. Activate the `llm-cpp` Conda environment and initialize Ollama:
|
||||
|
||||
- For **Linux users**:
|
||||
<Tabs
|
||||
defaultValue="linux"
|
||||
values={[
|
||||
{label: 'Linux', value: 'linux'},
|
||||
{label: 'Windows', value: 'windows'},
|
||||
]}>
|
||||
<TabItem value="linux">
|
||||
|
||||
```bash
|
||||
conda activate llm-cpp
|
||||
init-ollama
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="windows">
|
||||
|
||||
- For **Windows users**:
|
||||
|
||||
Please run the following command with **administrator privilege in Miniforge Prompt**.
|
||||
Run these commands with *administrator privileges in Miniforge Prompt*:
|
||||
|
||||
```cmd
|
||||
conda activate llm-cpp
|
||||
init-ollama.bat
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
> [!NOTE]
|
||||
> If you have installed higher version `ipex-llm[cpp]` and want to upgrade your ollama binary file, don't forget to remove old binary files first and initialize again with `init-ollama` or `init-ollama.bat`.
|
||||
2. If the installed `ipex-llm[cpp]` requires an upgrade to the Ollama binary files, remove the old binary files and reinitialize Ollama using `init-ollama` (Linux) or `init-ollama.bat` (Windows).
|
||||
|
||||
**Now you can use this executable file by standard ollama's usage.**
|
||||
*A symbolic link to Ollama appears in your current directory, and you can use this executable file following standard Ollama commands.*
|
||||
|
||||
#### 2.3 Run Ollama Serve
|
||||
#### 2.3 Launch Ollama service
|
||||
|
||||
You may launch the Ollama service as below:
|
||||
1. Set the environment variable `OLLAMA_NUM_GPU` to `999` to ensure that all layers of your model run on the Intel GPU; otherwise, some layers may default to CPU.
|
||||
2. For optimal performance on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), set the following environment variable before launching the Ollama service:
|
||||
|
||||
- For **Linux users**:
|
||||
```bash
|
||||
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
|
||||
```
|
||||
3. Launch the Ollama service:
|
||||
|
||||
<Tabs
|
||||
defaultValue="linux"
|
||||
values={[
|
||||
{label: 'Linux', value: 'linux'},
|
||||
{label: 'Windows', value: 'windows'},
|
||||
]}>
|
||||
<TabItem value="linux">
|
||||
|
||||
```bash
|
||||
export OLLAMA_NUM_GPU=999
|
||||
@ -224,9 +249,10 @@ You may launch the Ollama service as below:
|
||||
./ollama serve
|
||||
```
|
||||
|
||||
- For **Windows users**:
|
||||
</TabItem>
|
||||
<TabItem value="windows">
|
||||
|
||||
Please run the following command in Miniforge Prompt.
|
||||
Run the following command *in Miniforge Prompt*:
|
||||
|
||||
```cmd
|
||||
set OLLAMA_NUM_GPU=999
|
||||
@ -236,49 +262,54 @@ You may launch the Ollama service as below:
|
||||
|
||||
ollama serve
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
> Please set environment variable `OLLAMA_NUM_GPU` to `999` to make sure all layers of your model are running on Intel GPU, otherwise, some layers may run on CPU.
|
||||
:::tip NOTE
|
||||
To enable the Ollama service to accept connections from all IP addresses, use `OLLAMA_HOST=0.0.0.0 ./ollama serve` rather than simply `./ollama serve`.
|
||||
:::
|
||||
|
||||
|
||||
> If your local LLM is running on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), it is recommended to additionaly set the following environment variable for optimal performance before executing `ollama serve`:
|
||||
>
|
||||
> ```bash
|
||||
> export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
|
||||
> ```
|
||||
|
||||
|
||||
> To allow the service to accept connections from all IP addresses, use `OLLAMA_HOST=0.0.0.0 ./ollama serve` instead of just `./ollama serve`.
|
||||
|
||||
The console will display messages similar to the following:
|
||||
*The console displays messages similar to the following:*
|
||||
|
||||
|
||||
|
||||
### 3. Pull and Run Ollama Model
|
||||
### 3. Pull and Run Ollama model
|
||||
|
||||
Keep the Ollama service on and open another terminal and run `./ollama pull <model_name>` in Linux (`ollama.exe pull <model_name>` in Windows) to automatically pull a model. e.g. `qwen2:latest`:
|
||||
#### 3.1 Pull Ollama model
|
||||
|
||||
With the Ollama service running, open a new terminal and run `./ollama pull <model_name>` (Linux) or `ollama.exe pull <model_name>` (Windows) to pull the desired model. e.g., `qwen2:latest`:
|
||||
|
||||
|
||||
|
||||
#### Run Ollama Model
|
||||
#### 3.2 Run Ollama model
|
||||
|
||||
<Tabs
|
||||
defaultValue="linux"
|
||||
values={[
|
||||
{label: 'Linux', value: 'linux'},
|
||||
{label: 'Windows', value: 'windows'},
|
||||
]}>
|
||||
<TabItem value="linux">
|
||||
|
||||
- For **Linux users**:
|
||||
```bash
|
||||
./ollama run qwen2:latest
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="windows">
|
||||
|
||||
- For **Windows users**:
|
||||
```cmd
|
||||
ollama run qwen2:latest
|
||||
```
|
||||
### 4. Configure RAGflow to use IPEX-LLM accelerated Ollama
|
||||
|
||||
The confiugraiton follows the steps in
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
Ollama Section 4 [Add Ollama](#4-add-ollama),
|
||||
### 4. Configure RAGflow
|
||||
|
||||
Section 5 [Complete basic Ollama settings](#5-complete-basic-ollama-settings),
|
||||
To enable IPEX-LLM accelerated Ollama in RAGFlow, you must also complete the configurations in RAGFlow. The steps are identical to those outlined in the *Deploy a local model using Ollama* section:
|
||||
|
||||
Section 6 [Update System Model Settings](#6-update-system-model-settings),
|
||||
|
||||
Section 7 [Update Chat Configuration](#7-update-chat-configuration)
|
||||
1. [Add Ollama](#4-add-ollama)
|
||||
2. [Complete basic Ollama settings](#5-complete-basic-ollama-settings)
|
||||
3. [Update System Model Settings](#6-update-system-model-settings)
|
||||
4. [Update Chat Configuration](#7-update-chat-configuration)
|
||||
@ -43,7 +43,7 @@ You can link your file to one knowledge base or multiple knowledge bases at one
|
||||
|
||||
|
||||
|
||||
## Move file to specified folder
|
||||
## Move file to a specific folder
|
||||
|
||||
As of RAGFlow v0.8.0, this feature is *not* available.
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user