furiosa.runtime package

Submodules

furiosa.runtime.compiler module

furiosa.runtime.compiler.generate_compiler_log_path() → pathlib.Path: Generate a log path for compilation log

furiosa.runtime.consts module

furiosa.runtime.envs module

furiosa.runtime.envs.current_npu_device() → str

Return the current npu device name

Returns: NPU device name

furiosa.runtime.envs.is_compile_log_enabled() → bool

Return True or False whether the compile log is enabled or not.

Returns: True if the compile log is enabled, or False.

furiosa.runtime.envs.log_dir() → str

Return FURIOSA_LOG_DIR where the logs are stored.

Returns: The log directory of furiosa sdk

furiosa.runtime.envs.xdg_state_home() → str

Return XDG_STATE_HOME which is the base directory of furiosa logs, history, and other states

Returns: Furiosa home directory

furiosa.runtime.errors module

Nux Exception and Error

exception furiosa.runtime.errors.ApiClientInitFailed

Bases: furiosa.runtime.errors.NativeException

when api client fails to initialize due to api keys or others

exception furiosa.runtime.errors.CompilationFailed

Bases: furiosa.runtime.errors.NativeException

when Nux fails to compile a given model image to NPU model binary

exception furiosa.runtime.errors.DeviceBusy

Bases: furiosa.runtime.errors.NativeException

The device is already occupied

exception furiosa.runtime.errors.FuriosaError(message: str)

Bases: Exception

general exception caused by Furiosa Runtime

property message: str: Error message

exception furiosa.runtime.errors.IncompatibleApiClientError

Bases: furiosa.runtime.errors.NativeException

When both API client and server are incompatible

exception furiosa.runtime.errors.IncompatibleModel

Bases: furiosa.runtime.errors.NativeException

When Renegade compiler cannot recognize a given model image binary

exception furiosa.runtime.errors.InternalError(cause='unknown')

Bases: furiosa.runtime.errors.NativeException

internal error or no corresponding error in Python binding

exception furiosa.runtime.errors.InvalidInput(message: str = 'Invalid input tensors')

Bases: furiosa.runtime.errors.FuriosaError

When input tensors are invalid with any reason

exception furiosa.runtime.errors.InvalidSessionOption

Bases: furiosa.runtime.errors.NativeException

when api client fails to initialize due to api keys or others

exception furiosa.runtime.errors.InvalidYamlException

Bases: furiosa.runtime.errors.NativeException

When Renegade compiler cannot recognize a given model image binary

class furiosa.runtime.errors.NativeError(value)

Bases: enum.IntEnum

Python object correspondnig to nux_error_t in Nux C API

API_CLIENT_INIT_FAILED = 18

COMPILATION_FAILED = 14

DEVICE_BUSY = 23

DUMP_PROFILE_FAILED = 10

GET_TASK_FAILED = 9

INCOMPATIBLE_API_CLIENT_ERROR = 17

INCOMPATIBLE_MODEL = 13

INTERNAL_ERROR = 15

INVALID_BUFFER = 6

INVALID_INPUTS = 7

INVALID_INPUT_INDEX = 4

INVALID_OUTPUTS = 8

INVALID_OUTPUT_INDEX = 5

INVALID_SESSION_OPTIONS = 21

INVALID_YAML = 16

MODEL_DEPLOY_FAILED = 2

MODEL_EXECUTION_FAILED = 3

NO_API_KEY = 19

NULL_POINTER_EXCEPTION = 20

NUX_CREATION_FAILED = 1

QUEUE_NO_DATA = 12

QUEUE_WAIT_TIMEOUT = 11

SESSION_TERMINATED = 22

SUCCESS = 0

TENSOR_NAME_NOT_FOUND = 24

UNSUPPORTED_FEATURE = 25

exception furiosa.runtime.errors.NativeException(message: str, native_err: Optional[furiosa.runtime.errors.NativeError] = None)

Bases: furiosa.runtime.errors.FuriosaError

general exception caused by Nuxpy

native_error() → Optional[furiosa.runtime.errors.NativeError]: Return a native error if this exception comes from C native extension

exception furiosa.runtime.errors.NoApiKeyException

Bases: furiosa.runtime.errors.NativeException

when api client fails to initialize due to api keys or others

exception furiosa.runtime.errors.QueueWaitTimeout

Bases: furiosa.runtime.errors.NativeException

Timed out in Completion queue

exception furiosa.runtime.errors.SessionClosed

Bases: furiosa.runtime.errors.FuriosaError

Session is already terminated

exception furiosa.runtime.errors.SessionTerminated

Bases: furiosa.runtime.errors.NativeException

Session is already terminated

exception furiosa.runtime.errors.TensorNameNotFound

Bases: furiosa.runtime.errors.NativeException

When a given tensor name is not found in this model

exception furiosa.runtime.errors.UnsupportedDataType(dtype)

Bases: furiosa.runtime.errors.NativeException

Unsupported tensor data type

exception furiosa.runtime.errors.UnsupportedFeature

Bases: furiosa.runtime.errors.NativeException

Feature is not supported

exception furiosa.runtime.errors.UnsupportedTensorType

Bases: furiosa.runtime.errors.NativeException

Unsupported tensor type

furiosa.runtime.errors.into_exception(err: Union[ctypes.c_int, int]) → furiosa.runtime.errors.NativeException

Convert nux_error_t type in Nux C API to NuxException

Parameters: err (NativeError) –
Returns: NuxException

furiosa.runtime.errors.is_err(err: Union[ctypes.c_int, int]) → bool: True if NuxErr is not SUCCESS, or False

furiosa.runtime.errors.is_ok(err: Union[ctypes.c_int, int]) → bool: True if NuxErr is SUCCESS, or False

furiosa.runtime.model module

Model and its methods to access model metadata

class furiosa.runtime.model.Model

Bases: abc.ABC

NPU model binary compiled by Renegade compiler

allocate_inputs() → furiosa.runtime.tensor.TensorArray: Creates an array of input tensors with allocated buffers

allocate_outputs() → furiosa.runtime.tensor.TensorArray: Creates an array of output tensors with allocated buffers

allocate_tensors(names: List[str]) → furiosa.runtime.tensor.TensorArray: Creates an array of tensors corresponding to tensor names with allocated buffers

create_inputs() → furiosa.runtime.tensor.TensorArray: Creates an array of input tensors without allocated buffers

create_outputs() → furiosa.runtime.tensor.TensorArray: Creates an array of output tensors without allocated buffers

create_tensors(names: List[str]) → furiosa.runtime.tensor.TensorArray: Creates an array of tensors corresponding to tensor names without allocated buffers

input(idx) → furiosa.runtime.tensor.TensorDesc: Return tensor description of i-th input tensor of Model

property input_num: int: Number of input tensors of Model

inputs() → [<class 'furiosa.runtime.tensor.TensorDesc'>]: Tensor descriptions of all input tensors of Model

output(idx) → furiosa.runtime.tensor.TensorDesc: Returns tensor description of i-th output tensor of Model

property output_num: int: Number of output tensors of Model

outputs() → [<class 'furiosa.runtime.tensor.TensorDesc'>]: Tensor descriptions of all output tensors of Model

print_summary(): Prints the summary of this model

summary() → str: Returns the summary of this model

furiosa.runtime.session module

Session and its asynchronous API for model inference

class furiosa.runtime.session.AsyncSession(ref: ctypes.c_void_p)

Bases: furiosa.runtime.model.Model

An asynchronous session for a given model allows to submit predictions

close()

Closes this session

After a session is closed, CompletionQueue will return an error if CompletionQueue.recv() is called.

submit(values: Union[numpy.ndarray, numpy.generic, furiosa.runtime.tensor.TensorArray], context: Optional[object] = None) → None

Submit a prediction request

It immediately returns without blocking the caller, and If the prediction is completed, the outputs will be sent to CompletionQueue.

Parameters

values – Input values
context – an additional context to identify the prediction request

class furiosa.runtime.session.CompletionQueue(ref: ctypes.c_void_p, context_ty: type, output_descs: [<class 'furiosa.runtime.tensor.TensorDesc'>])

Bases: object

Receives the completion results asynchronously from AsyncSession

close()

Closes this completion queue.

If it is closed, AsyncSession also will stop working.

recv(timeout: typing.Optional[int] = None) -> (<class 'object'>, <class 'furiosa.runtime.tensor.TensorArray'>)

Receives the prediction results which are asynchronously coming from AsyncSession

If there are already prediction outputs, it will return immediately. Otherwise, it will be blocked until the next result are ready.

If timeout is set, recv() will be blocked only until the timeout occurs. If timed out, recv() throws QueueWaitTimeout exception.

If AsyncSession is closed earlier recv() will throw SessionTerminated exception.

Parameters

timeout (int) – How long to wait before giving up.
milliseconds. (It should be a positive interger in) –

Returns

A tuple, whose first value is the context value passed when you submit an inference task and the second value is inference output.

class furiosa.runtime.session.Session(model: Union[bytes, str, pathlib.Path], device: Optional[str] = None, worker_num: Optional[int] = None, compile_config: Optional[Dict[str, object]] = None)

Bases: furiosa.runtime.model.Model

Provides a blocking API to run an inference task with a given model

close(): Close the session and release all resources belonging to the session

run(inputs: Union[numpy.ndarray, numpy.generic, furiosa.runtime.tensor.TensorArray, List[Union[numpy.ndarray, numpy.generic]]]) → furiosa.runtime.tensor.TensorArray

Runs an inference task with inputs

Parameters: inputs – It can be a single runtime.Tensor, runtime.TensorArray or numpy.ndarray object. Also, you can pass one TensorArray or a list of numpy.ndarray objects.
Returns: Inference output

run_with(outputs: List[str], inputs: Dict[str, numpy.ndarray]) → furiosa.runtime.tensor.TensorArray

Runs an inference task with inputs

Parameters: inputs – It can be a single runtime.Tensor, runtime.TensorArray or numpy.ndarray object. Also, you can pass one TensorArray or a list of numpy.ndarray objects.
Returns: Inference output

furiosa.runtime.session.create(model: Union[bytes, str, pathlib.Path], device: Optional[str] = None, worker_num: Optional[int] = None, compile_config: Optional[Dict[str, object]] = None) → furiosa.runtime.session.Session

Creates a session for a model

Parameters

model (bytes, str, Path) – a byte string containing a model image or a path string of a model image file
device – NPU device (str) (e.g., npu0pe0, npu0pe0-1)
worker_num – Number of workers
compile_config (Dict[str, object]) – Compile config

Returns

the session for a given model, allowing to run predictions. Session is a thread safe.

furiosa.runtime.session.create_async(model: typing.Union[bytes, str, pathlib.Path], context_ty: typing.Optional[type] = None, device: typing.Optional[str] = None, worker_num: typing.Optional[int] = None, input_queue_size: typing.Optional[int] = None, output_queue_size: typing.Optional[int] = None, compile_config: typing.Optional[typing.Dict[str, object]] = None) -> (<class 'furiosa.runtime.session.AsyncSession'>, <class 'furiosa.runtime.session.CompletionQueue'>)

Creates a pair of the asynchronous session and the completion queue for a given model

Parameters

model (bytes, str, Path) – a byte string containing a model image or a path string of a model image file
context_ty (type) – Type for passing context from AsyncSession to CompletionQueue
device – NPU device (str) (e.g., npu0pe0, npu0pe0-1)
worker_num – Number of workers
input_queue_size – The input queue size, and it must be > 0 and < 2^31.
output_queue_size – The output queue size, and it must be be > 0 and < 2^31.
compile_config (Dict[str, object]) – Compile config

Returns

A pair of the asynchronous session and the completion queue. the asynchronous session for a given model allows to submit predictions. the completion queue allows users to receive the prediction outputs asynchronously.

furiosa.runtime.tensor module

Tensor object and its utilities

class furiosa.runtime.tensor.Axis(value)

Bases: enum.IntEnum

Axis of Tensor

BATCH = 3

BATCH_OUTER = 7

CHANNEL = 2

CHANNEL_OUTER = 6

HEIGHT = 1

HEIGHT_OUTER = 5

UNKNOWN = 8

WIDTH = 0

WIDTH_OUTER = 4

class furiosa.runtime.tensor.DataType(value)

Bases: enum.IntEnum

Tensor data type

BFLOAT16 = 5

FLOAT32 = 0

INT32 = 3

INT64 = 4

INT8 = 2

UINT8 = 1

property numpy_dtype: Return the numpy dtype corresponding to this DataType

class furiosa.runtime.tensor.Tensor(ref: ctypes.c_void_p, desc: furiosa.runtime.tensor.TensorDesc, allocated: bool = False)

Bases: object

A tensor which contains data and tensor description including shape

copy_from(data: Union[numpy.ndarray, numpy.generic]): Copy the contents of Numpy ndarray to this tensor

property dtype: furiosa.runtime.tensor.DataType: Data type of tensor

numpy() → numpy.ndarray: Return numpy.ndarray converted from this tensor

property numpy_dtype: Return numpy dtype

property shape: tuple

Return the tensor shape

Returns: Tensor shape. An example shape is `(1, 28, 28, 1)`.

class furiosa.runtime.tensor.TensorArray(ref: ctypes.c_void_p, descs: [<class 'furiosa.runtime.tensor.TensorDesc'>], allocated: bool = False)

Bases: object

A list of tensors

It is used for input and output values of model inferences.

is_empty() → bool: True if it has no Tensor

numpy() → [<class 'numpy.ndarray'>]: Convert TensorArray to a list of numpy.ndarray

class furiosa.runtime.tensor.TensorDesc(ref: ctypes.c_void_p)

Bases: object

Tensor description including dimension, shape, and data type

axis(idx: int) → furiosa.runtime.tensor.Axis: Axis type of i-th dimension (e.g., width, height, channel)

dim(idx: int) → int: Size of i-th dimension

property dtype: furiosa.runtime.tensor.DataType: Data type of tensor

property format: str: Tensor memory layout (e.g., NHWC, NCHW)

property length: int: Number of all elements across all dimensions

property name: Optional[str]

property ndim: int: Number of dimensions

property numpy_dtype: Return numpy dtype

property shape: tuple: tensor shape

property size: int: Size in bytes

furiosa.runtime.tensor.numpy_dtype(value): Return numpy dtype from any eligible object of Nux

furiosa.runtime.tensor.rand(tensor: furiosa.runtime.tensor.TensorDesc) → numpy.ndarray: Return a new array of given shape and type, filled with random numbers.

furiosa.runtime.tensor.zeros(tensor: furiosa.runtime.tensor.TensorDesc) → numpy.ndarray: Return a new array of given shape and type, filled with zeros.

Module contents

Provide high-level Python APIs to access Furiosa AI’s NPUs and its eco-system

furiosa.runtime.full_version() → str: Returns a full version string including the native library version