furiosa.runtime package
Submodules
furiosa.runtime.compiler module
- furiosa.runtime.compiler.generate_compiler_log_path() pathlib.Path
Generate a log path for compilation log
furiosa.runtime.consts module
furiosa.runtime.envs module
- furiosa.runtime.envs.current_npu_device() str
Return the current npu device name
- Returns
NPU device name
- furiosa.runtime.envs.is_compile_log_enabled() bool
Return True or False whether the compile log is enabled or not.
- Returns
True if the compile log is enabled, or False.
- furiosa.runtime.envs.log_dir() str
Return FURIOSA_LOG_DIR where the logs are stored.
- Returns
The log directory of furiosa sdk
- furiosa.runtime.envs.xdg_state_home() str
Return XDG_STATE_HOME which is the base directory of furiosa logs, history, and other states
- Returns
Furiosa home directory
furiosa.runtime.errors module
Nux Exception and Error
- exception furiosa.runtime.errors.ApiClientInitFailed
Bases:
furiosa.runtime.errors.NativeException
when api client fails to initialize due to api keys or others
- exception furiosa.runtime.errors.CompilationFailed
Bases:
furiosa.runtime.errors.NativeException
when Nux fails to compile a given model image to NPU model binary
- exception furiosa.runtime.errors.DeviceBusy
Bases:
furiosa.runtime.errors.NativeException
The device is already occupied
- exception furiosa.runtime.errors.FuriosaError(message: str)
Bases:
Exception
general exception caused by Furiosa Runtime
- property message: str
Error message
- exception furiosa.runtime.errors.IncompatibleApiClientError
Bases:
furiosa.runtime.errors.NativeException
When both API client and server are incompatible
- exception furiosa.runtime.errors.IncompatibleModel
Bases:
furiosa.runtime.errors.NativeException
When Renegade compiler cannot recognize a given model image binary
- exception furiosa.runtime.errors.InternalError(cause='unknown')
Bases:
furiosa.runtime.errors.NativeException
internal error or no corresponding error in Python binding
- exception furiosa.runtime.errors.InvalidInput(message: str = 'Invalid input tensors')
Bases:
furiosa.runtime.errors.FuriosaError
When input tensors are invalid with any reason
- exception furiosa.runtime.errors.InvalidSessionOption
Bases:
furiosa.runtime.errors.NativeException
when api client fails to initialize due to api keys or others
- exception furiosa.runtime.errors.InvalidYamlException
Bases:
furiosa.runtime.errors.NativeException
When Renegade compiler cannot recognize a given model image binary
- class furiosa.runtime.errors.NativeError(value)
Bases:
enum.IntEnum
Python object correspondnig to nux_error_t in Nux C API
- API_CLIENT_INIT_FAILED = 18
- COMPILATION_FAILED = 14
- DEVICE_BUSY = 23
- DUMP_PROFILE_FAILED = 10
- GET_TASK_FAILED = 9
- INCOMPATIBLE_API_CLIENT_ERROR = 17
- INCOMPATIBLE_MODEL = 13
- INTERNAL_ERROR = 15
- INVALID_BUFFER = 6
- INVALID_INPUTS = 7
- INVALID_INPUT_INDEX = 4
- INVALID_OUTPUTS = 8
- INVALID_OUTPUT_INDEX = 5
- INVALID_SESSION_OPTIONS = 21
- INVALID_YAML = 16
- MODEL_DEPLOY_FAILED = 2
- MODEL_EXECUTION_FAILED = 3
- NO_API_KEY = 19
- NULL_POINTER_EXCEPTION = 20
- NUX_CREATION_FAILED = 1
- QUEUE_NO_DATA = 12
- QUEUE_WAIT_TIMEOUT = 11
- SESSION_TERMINATED = 22
- SUCCESS = 0
- TENSOR_NAME_NOT_FOUND = 24
- UNSUPPORTED_FEATURE = 25
- exception furiosa.runtime.errors.NativeException(message: str, native_err: Optional[furiosa.runtime.errors.NativeError] = None)
Bases:
furiosa.runtime.errors.FuriosaError
general exception caused by Nuxpy
- native_error() Optional[furiosa.runtime.errors.NativeError]
Return a native error if this exception comes from C native extension
- exception furiosa.runtime.errors.NoApiKeyException
Bases:
furiosa.runtime.errors.NativeException
when api client fails to initialize due to api keys or others
- exception furiosa.runtime.errors.QueueWaitTimeout
Bases:
furiosa.runtime.errors.NativeException
Timed out in Completion queue
- exception furiosa.runtime.errors.SessionClosed
Bases:
furiosa.runtime.errors.FuriosaError
Session is already terminated
- exception furiosa.runtime.errors.SessionTerminated
Bases:
furiosa.runtime.errors.NativeException
Session is already terminated
- exception furiosa.runtime.errors.TensorNameNotFound
Bases:
furiosa.runtime.errors.NativeException
When a given tensor name is not found in this model
- exception furiosa.runtime.errors.UnsupportedDataType(dtype)
Bases:
furiosa.runtime.errors.NativeException
Unsupported tensor data type
- exception furiosa.runtime.errors.UnsupportedFeature
Bases:
furiosa.runtime.errors.NativeException
Feature is not supported
- exception furiosa.runtime.errors.UnsupportedTensorType
Bases:
furiosa.runtime.errors.NativeException
Unsupported tensor type
- furiosa.runtime.errors.into_exception(err: Union[ctypes.c_int, int]) furiosa.runtime.errors.NativeException
Convert nux_error_t type in Nux C API to NuxException
- Parameters
err (NativeError) –
- Returns
NuxException
- furiosa.runtime.errors.is_err(err: Union[ctypes.c_int, int]) bool
True if NuxErr is not SUCCESS, or False
- furiosa.runtime.errors.is_ok(err: Union[ctypes.c_int, int]) bool
True if NuxErr is SUCCESS, or False
furiosa.runtime.model module
Model and its methods to access model metadata
- class furiosa.runtime.model.Model
Bases:
abc.ABC
NPU model binary compiled by Renegade compiler
- allocate_inputs() furiosa.runtime.tensor.TensorArray
Creates an array of input tensors with allocated buffers
- allocate_outputs() furiosa.runtime.tensor.TensorArray
Creates an array of output tensors with allocated buffers
- allocate_tensors(names: List[str]) furiosa.runtime.tensor.TensorArray
Creates an array of tensors corresponding to tensor names with allocated buffers
- create_inputs() furiosa.runtime.tensor.TensorArray
Creates an array of input tensors without allocated buffers
- create_outputs() furiosa.runtime.tensor.TensorArray
Creates an array of output tensors without allocated buffers
- create_tensors(names: List[str]) furiosa.runtime.tensor.TensorArray
Creates an array of tensors corresponding to tensor names without allocated buffers
- input(idx) furiosa.runtime.tensor.TensorDesc
Return tensor description of i-th input tensor of Model
- property input_num: int
Number of input tensors of Model
- inputs() [<class 'furiosa.runtime.tensor.TensorDesc'>]
Tensor descriptions of all input tensors of Model
- output(idx) furiosa.runtime.tensor.TensorDesc
Returns tensor description of i-th output tensor of Model
- property output_num: int
Number of output tensors of Model
- outputs() [<class 'furiosa.runtime.tensor.TensorDesc'>]
Tensor descriptions of all output tensors of Model
- print_summary()
Prints the summary of this model
- summary() str
Returns the summary of this model
furiosa.runtime.session module
Session and its asynchronous API for model inference
- class furiosa.runtime.session.AsyncSession(ref: ctypes.c_void_p)
Bases:
furiosa.runtime.model.Model
An asynchronous session for a given model allows to submit predictions
- close()
Closes this session
After a session is closed, CompletionQueue will return an error if CompletionQueue.recv() is called.
- submit(values: Union[numpy.ndarray, numpy.generic, furiosa.runtime.tensor.TensorArray], context: Optional[object] = None) None
Submit a prediction request
It immediately returns without blocking the caller, and If the prediction is completed, the outputs will be sent to CompletionQueue.
- Parameters
values – Input values
context – an additional context to identify the prediction request
- class furiosa.runtime.session.CompletionQueue(ref: ctypes.c_void_p, context_ty: type, output_descs: [<class 'furiosa.runtime.tensor.TensorDesc'>])
Bases:
object
Receives the completion results asynchronously from AsyncSession
- close()
Closes this completion queue.
If it is closed, AsyncSession also will stop working.
- recv(timeout: typing.Optional[int] = None) -> (<class 'object'>, <class 'furiosa.runtime.tensor.TensorArray'>)
Receives the prediction results which are asynchronously coming from AsyncSession
If there are already prediction outputs, it will return immediately. Otherwise, it will be blocked until the next result are ready.
If
timeout
is set,recv()
will be blocked only until the timeout occurs. If timed out,recv()
throwsQueueWaitTimeout
exception.If AsyncSession is closed earlier
recv()
will throwSessionTerminated
exception.- Parameters
timeout (int) – How long to wait before giving up.
milliseconds. (It should be a positive interger in) –
- Returns
A tuple, whose first value is the context value passed when you submit an inference task and the second value is inference output.
- class furiosa.runtime.session.Session(model: Union[bytes, str, pathlib.Path], device: Optional[str] = None, worker_num: Optional[int] = None, compile_config: Optional[Dict[str, object]] = None)
Bases:
furiosa.runtime.model.Model
Provides a blocking API to run an inference task with a given model
- close()
Close the session and release all resources belonging to the session
- run(inputs: Union[numpy.ndarray, numpy.generic, furiosa.runtime.tensor.TensorArray, List[Union[numpy.ndarray, numpy.generic]]]) furiosa.runtime.tensor.TensorArray
Runs an inference task with inputs
- Parameters
inputs – It can be a single runtime.Tensor, runtime.TensorArray or numpy.ndarray object. Also, you can pass one TensorArray or a list of numpy.ndarray objects.
- Returns
Inference output
- run_with(outputs: List[str], inputs: Dict[str, numpy.ndarray]) furiosa.runtime.tensor.TensorArray
Runs an inference task with inputs
- Parameters
inputs – It can be a single runtime.Tensor, runtime.TensorArray or numpy.ndarray object. Also, you can pass one TensorArray or a list of numpy.ndarray objects.
- Returns
Inference output
- furiosa.runtime.session.create(model: Union[bytes, str, pathlib.Path], device: Optional[str] = None, worker_num: Optional[int] = None, compile_config: Optional[Dict[str, object]] = None) furiosa.runtime.session.Session
Creates a session for a model
- Parameters
model (bytes, str, Path) – a byte string containing a model image or a path string of a model image file
device – NPU device (str) (e.g., npu0pe0, npu0pe0-1)
worker_num – Number of workers
compile_config (Dict[str, object]) – Compile config
- Returns
the session for a given model, allowing to run predictions. Session is a thread safe.
- furiosa.runtime.session.create_async(model: typing.Union[bytes, str, pathlib.Path], context_ty: typing.Optional[type] = None, device: typing.Optional[str] = None, worker_num: typing.Optional[int] = None, input_queue_size: typing.Optional[int] = None, output_queue_size: typing.Optional[int] = None, compile_config: typing.Optional[typing.Dict[str, object]] = None) -> (<class 'furiosa.runtime.session.AsyncSession'>, <class 'furiosa.runtime.session.CompletionQueue'>)
Creates a pair of the asynchronous session and the completion queue for a given model
- Parameters
model (bytes, str, Path) – a byte string containing a model image or a path string of a model image file
context_ty (type) – Type for passing context from AsyncSession to CompletionQueue
device – NPU device (str) (e.g., npu0pe0, npu0pe0-1)
worker_num – Number of workers
input_queue_size – The input queue size, and it must be > 0 and < 2^31.
output_queue_size – The output queue size, and it must be be > 0 and < 2^31.
compile_config (Dict[str, object]) – Compile config
- Returns
A pair of the asynchronous session and the completion queue. the asynchronous session for a given model allows to submit predictions. the completion queue allows users to receive the prediction outputs asynchronously.
furiosa.runtime.tensor module
Tensor object and its utilities
- class furiosa.runtime.tensor.Axis(value)
Bases:
enum.IntEnum
Axis of Tensor
- BATCH = 3
- BATCH_OUTER = 7
- CHANNEL = 2
- CHANNEL_OUTER = 6
- HEIGHT = 1
- HEIGHT_OUTER = 5
- UNKNOWN = 8
- WIDTH = 0
- WIDTH_OUTER = 4
- class furiosa.runtime.tensor.DataType(value)
Bases:
enum.IntEnum
Tensor data type
- BFLOAT16 = 5
- FLOAT32 = 0
- INT32 = 3
- INT64 = 4
- INT8 = 2
- UINT8 = 1
- property numpy_dtype
Return the numpy dtype corresponding to this DataType
- class furiosa.runtime.tensor.Tensor(ref: ctypes.c_void_p, desc: furiosa.runtime.tensor.TensorDesc, allocated: bool = False)
Bases:
object
A tensor which contains data and tensor description including shape
- copy_from(data: Union[numpy.ndarray, numpy.generic])
Copy the contents of Numpy ndarray to this tensor
- property dtype: furiosa.runtime.tensor.DataType
Data type of tensor
- numpy() numpy.ndarray
Return numpy.ndarray converted from this tensor
- property numpy_dtype
Return numpy dtype
- property shape: tuple
Return the tensor shape
- Returns
Tensor shape. An example shape is
`(1, 28, 28, 1)`
.
- class furiosa.runtime.tensor.TensorArray(ref: ctypes.c_void_p, descs: [<class 'furiosa.runtime.tensor.TensorDesc'>], allocated: bool = False)
Bases:
object
A list of tensors
It is used for input and output values of model inferences.
- is_empty() bool
True if it has no Tensor
- numpy() [<class 'numpy.ndarray'>]
Convert TensorArray to a list of numpy.ndarray
- class furiosa.runtime.tensor.TensorDesc(ref: ctypes.c_void_p)
Bases:
object
Tensor description including dimension, shape, and data type
- axis(idx: int) furiosa.runtime.tensor.Axis
Axis type of i-th dimension (e.g., width, height, channel)
- dim(idx: int) int
Size of i-th dimension
- property dtype: furiosa.runtime.tensor.DataType
Data type of tensor
- property format: str
Tensor memory layout (e.g., NHWC, NCHW)
- property length: int
Number of all elements across all dimensions
- property name: Optional[str]
- property ndim: int
Number of dimensions
- property numpy_dtype
Return numpy dtype
- property shape: tuple
tensor shape
- property size: int
Size in bytes
- furiosa.runtime.tensor.numpy_dtype(value)
Return numpy dtype from any eligible object of Nux
- furiosa.runtime.tensor.rand(tensor: furiosa.runtime.tensor.TensorDesc) numpy.ndarray
Return a new array of given shape and type, filled with random numbers.
- furiosa.runtime.tensor.zeros(tensor: furiosa.runtime.tensor.TensorDesc) numpy.ndarray
Return a new array of given shape and type, filled with zeros.
Module contents
Provide high-level Python APIs to access Furiosa AI’s NPUs and its eco-system
- furiosa.runtime.full_version() str
Returns a full version string including the native library version