Xinference

Xorbits Inference(Xinference) is an open-source project to run language models on your own machine. You can use it to serve open-source LLMs like Llama-3 locally.

​

Preparation

Follow the instructions at Using Xinference to setup Xinference and run the llama-2-chat model.

​

Configuration

• API Host: http://127.0.0.1:9997
• API Key: random strings
• Model: llama-3-chat

You can find all the available models at https://inference.readthedocs.io/en/latest/models/builtin/llm/index.html

​

Troubleshooting

• Only models with chat in their name are supported.