Python SDK installation and user guide

FuriosaAI Python SDK is a software development kit for writing Python applications that use the NPU. With the Python SDK, you can utilize various tools, libraries, and frameworks of the Python ecosystem that are most widely used in the AI/ML field for developing NPU applications. Python SDK consists of various modules and provides an inference API, a quantization API, a command line tool, and a server program for serving.

Requirements

Ubuntu 18.04 LTS (Debian buster) or higher
FuriosaAI SDK required packages
Python 3.7 or higher (See Python execution environment setup for setup Python environment)
Latest version of pip

To install and use Python SDK, follow the Installing required packages guide. You need to install the required kernel driver, firmware, and runtime library.

Python execution environment setup

Python SDK requires Python 3.7 or above. Here, we describe Python execution environment configuration.

Note

If you are not using the FuriosaAI Python SDK, or if you are familiar with configuring a Python execution environment, you can skip this section.

You can check the Python version currently installed in your system with the command below.

python --version
Python 3.8.10

If the Python command does not exist, or if your Python version is below 3.7, configure the Python environment by selecting one of the methods below.

Python environment configuration with Conda (recommended): Conda allows you to configure a separate, isolated Python environment for specific Python applications. Conda therefore prevents the package dependency issues or Python version issues that users often encounter when installing Python applications.
Configure the Python execution environment directly on the Configuring Python environment using Linux packages: Linux system. You can select this option if you are not concerned about conflicts with other Python execution environments.

Python environment configuration with Conda

Conda makes it easy to configure a isolated Python environment for a specific Python application. To find out more about Conda, refer to readings available in Conda.

You can get started by downloading the installer as shown below. Select yes to all questions when running sh ./Miniconda3-latest-Linux-x86_64.sh.

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh ./Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda --version

Creating and activating isolated Python execution environment

After installing Anaconda, you can create an isolated Python execution environment and activate it as needed.

If you want to use Python 3.8, create an execution environment with the name furiosa-3.8, by using the following command.

conda create -n furiosa-3.8 python=3.8

The created Python 3.8 environment is activated with the activate command.

conda activate furiosa-3.8
# version check
python --version

Once the Python execution environment is activated, install the Python SDK as explained in Installing Python SDK package.
If you wish to terminate the Python execution environment, use the deactivate command.

$ conda deactivate

An environment created once can be used again at any time with the activate command. Packages that have been installed do not need to be reinstalled after activation.

Configuring Python environment using Linux packages

If you can configure the Python environment directly on the system, install the necessary packages as shown below.

sudo apt install -y python3 python3-pip python-is-python3

Check the Python version to ensure proper installation.

python --version
Python 3.8.10

Installing Python SDK package

FuriosaAI Python SDK package is uploaded on the pypi repository, so you can easily install it as shown by using the pip command.

pip install furiosa-sdk

The package contains a compiler command line interface and an inference API. Refer to furiosa compile and Tutorial and Code Examples for detailed usage guides.

Additional functions are provided in the form of Python extra packages, and you can select and install packages as you require from Extra packages. For example, if you need to install server` for model serving and litmus to check the compatibility between model and SDK, specify the extension package as follows.

pip install 'furiosa-sdk[server, litmus]'

Download the source code from FuriosaAI Github repository and install the packages in the following order.

git clone https://github.com/furiosa-ai/furiosa-sdk
cd furiosa-sdk/python
pip install furiosa-runtime
pip install furiosa-tools
pip install furiosa-sdk

If you wish to install extra packages, install the Python module in the subdirectory of furiosa-sdk/python. For example, if you want to install a model server, install it according to the order of dependencies as follows.

cd furiosa-sdk/python
pip install furiosa-registry
pip install furiosa-server

Extra packages

FuriosaAI Models

It can be executed directly on the NPU and provides optimized DNN model architecture, pre-trained model image, among others, in the form of a Python module. You can install them with the following command.

pip install 'furiosa-sdk[models]'

Model Server

Provides the function of accelerating DNN model with the NPU, and serving it with GRPC or Restful API.

pip install 'furiosa-sdk[server]'

Litmus

A tool to check whether the specified model is compatible with the Furiosa SDK. Here, we simulate execution of processes such as model quantization and compilation.

pip install 'furiosa-sdk[litmus]'

Quantizer

The quantizer package provides a set of APIs for converting a model into a quantized model. You can find more information about the quantization function provided by the Furiosa SDK and the NPU at Model Quantization.

pip install 'furiosa-sdk[quantizer]'