Model object

In furiosa-models project, Model is the first class object, and it represents a neural network model. This document explains what Model object offers and their usages.

Loading a pre-trained model

To load a pre-trained neural-network model, you need to call load() method. Since the sizes of pre-trained model weights vary from tens to hundreds megabytes, the model images are not included in Python package. When load() method is called, a pre-trained model will be fetched over the network. It takes some time (usually few seconds) depending on models and network conditions. Once the model images are fetched, they will be cached on a local disk.

Non-blocking API load_async() also is available, and it can be used if your application is running through asynchronous executors (e.g., asyncio).

Blocking APINon-blocking API

from furiosa.models.types import Model
from furiosa.models.vision import ResNet50

model: Model = ResNet50.load()

import asyncio

from furiosa.models.types import Model
from furiosa.models.vision import ResNet50

model: Model = asyncio.run(ResNet50.load_async())

Accessing artifacts and metadata

A Model object includes model artifacts, such as ONNX, tflite, calibration range in yaml format, and ENF.

ENF format is FuriosaAI Compiler specific format. Once you have the ENF file, you can reuse it to omit the compilation process that take up to minutes. In addition, a Model object has various metadata. The followings are all attributes belonging to a single Model object.

`furiosa.models.types.Model`

Represent the artifacts and metadata of a neural network model

Attributes:

Name	Type	Description
`name`	`str`	a name of this model
`format`	`Format`	the binary format type of model source; e.g., ONNX, tflite
`source`	`bytes`	a source binary in ONNX or tflite. It can be used for compiling this model with a custom compiler configuration.
`enf`	`Optional[bytes]`	the executable binary for furiosa runtime and NPU
`calib_yaml`	`Optional[str]`	the calibration ranges in yaml format for quantization
`version`	`Optional[str]`	model version
`inputs`	`Optional[List[ModelTensor]]`	data type and shape of input tensors
`outputs`	`Optional[List[ModelTensor]]`	data type and shape of output tensors
`compiler_config`	`Optional[List[ModelTensor]]`	a pre-defined compiler option

Source code in furiosa/models/types.py

class Model(ABC, BaseModel):
    """Represent the artifacts and metadata of a neural network model

    Attributes:
        name: a name of this model
        format: the binary format type of model source; e.g., ONNX, tflite
        source: a source binary in ONNX or tflite. It can be used for compiling this model
            with a custom compiler configuration.
        enf: the executable binary for furiosa runtime and NPU
        calib_yaml: the calibration ranges in yaml format for quantization
        version: model version
        inputs: data type and shape of input tensors
        outputs: data type and shape of output tensors
        compiler_config: a pre-defined compiler option
    """

    class Config(BaseConfig):
        extra: Extra = Extra.forbid
        # To allow Session, Processor type
        arbitrary_types_allowed = True
        use_enum_values = True

    name: str
    source: bytes = Field(repr=False)
    format: Format
    enf: Optional[bytes] = Field(repr=False)
    calib_yaml: Optional[str] = Field(repr=False)

    family: Optional[str] = None
    version: Optional[str] = None

    metadata: Optional[Metadata] = None

    inputs: Optional[List[ModelTensor]] = []
    outputs: Optional[List[ModelTensor]] = []

    postprocessor_map: Optional[Dict[Platform, Type[PostProcessor]]] = None

    preprocessor: Optional[PreProcessor] = None
    postprocessor: Optional[PostProcessor] = None

    @staticmethod
    @abstractmethod
    def get_artifact_name() -> str:
        ...

    @classmethod
    @abstractmethod
    def load_aux(
        cls, artifacts: Dict[str, bytes], use_native: Optional[bool] = None, *args, **kwargs
    ):
        ...

    @classmethod
    async def load_async(cls, use_native: Optional[bool] = None, *args, **kwargs) -> 'Model':
        return cls.load_aux(
            await load_artifacts(cls.get_artifact_name()), use_native, *args, **kwargs
        )

    @classmethod
    def load(cls, use_native: Optional[bool] = None, *args, **kwargs) -> 'Model':
        return cls.load_aux(
            synchronous(load_artifacts)(cls.get_artifact_name()), use_native, *args, **kwargs
        )

    def preprocess(self, *args, **kwargs) -> Tuple[Sequence[npt.ArrayLike], Sequence[Context]]:
        assert self.preprocessor
        return self.preprocessor(*args, **kwargs)

    def postprocess(self, *args, **kwargs):
        assert self.postprocessor
        return self.postprocessor(*args, **kwargs)

Inferencing with Session API

To create a session, pass the enf field of the model object to the furiosa.runtime.session.create() function. Passing the pre-compiled enf allows you to perform inference directly without the compilation process. Alternatively, you can also manually quantize and compile the original f32 model with the provided calibration range.

Info

If you want to learn more about the installation of furiosa-sdk and how to use it, please follow the followings:

Passing Model.source to session.create() allows users to start from source models in ONNX or tflite and customize models to their specific use-cases. This customization includes options such as specifying batch sizes and compiler configurations for optimization purposes. For additional information on Model.source, please refer to Accessing artifacts and metadata.

To utilize f32 source models, it is necessary to perform calibration and quantization. Pre-calibrated data is readily available in Furiosa-models, facilitating direct access to the quantization process. For manual quantization of the model, you can install the furiosa-quantizer package, which can be found at this package link. The calib_range field of the model class represents this pre-calibrated data. After quantization, the output will be in the form of FuriosaAI's IR which can then be passed to the session. At this stage, the compiler configuration can be specified.

Info

The calibration range field is actually in yaml format but serialized in string type. To deserialize the calibration range, use import yaml; yaml.full_load(calib_range).

Example

Using pre-compiled ENF binary

from furiosa.models.vision import SSDMobileNet
from furiosa.runtime import session

image = ["tests/assets/cat.jpg"]

mobilenet = SSDMobileNet.load()
with session.create(mobilenet.enf) as sess:
    inputs, contexts = mobilenet.preprocess(image)
    outputs = sess.run(inputs).numpy()
    mobilenet.postprocess(outputs, contexts)

Example

From ONNX

import yaml

from furiosa.models.vision import SSDMobileNet
from furiosa.quantizer import quantize
from furiosa.runtime import session

compiler_config = {"tabulate_dequantize": True}

image = ["tests/assets/cat.jpg"]

mobilenet = SSDMobileNet.load()
onnx_model: bytes = mobilenet.source
calib_range: dict = yaml.full_load(mobilenet.calib_yaml)

# See https://furiosa-ai.github.io/docs/latest/en/api/python/furiosa.quantizer.html#furiosa.quantizer.quantize
# for more details
dfg = quantize(onnx_model, calib_range, with_quantize=False)

with session.create(dfg, compiler_config=compiler_config) as sess:
    inputs, contexts = mobilenet.preprocess(image)
    outputs = sess.run(inputs).numpy()
    mobilenet.postprocess(outputs, contexts)

Pre/Postprocessing

There are gaps between model input/outputs and user applications' desired input and output data. In general, inputs and outputs of a neural network model are tensors. In applications, user sample data are images in standard formats like PNG or JPEG, and users also need to convert the output tensors to struct data for user applications.

A Model object also provides both preprocess() and postprocess() methods. They are utilities to convert easily user inputs to the model's input tensors and output tensors to struct data which can be easily accessible by applications. If using pre-built pre/postprocessing methods, users can quickly start using furiosa-models.

In sum, typical steps of a single inference is as the following, as also shown at examples.

Call preprocess() with user inputs (e.g., image files)
Pass an output of preprocess() to Session.run()
Pass the output of the model to postprocess()

Info

Default postprocessing implementations are in Python. However, some models have the native postprocessing implemented in Rust and C++ and optimized for FuriosaAI Warboy and Intel/AMD CPUs. Python implementations can run on CPU and GPU as well, whereas the native postprocessor implementations works with only FuriosaAI NPU. Native implementations are designed to leverage FuriosaAI NPU's characteristics even for post-processing and maximize the latency and throughput by using modern CPU architecture, such as CPU cache, SIMD instructions and CPU pipelining. According to our benchmark, the native implementations show at most 70% lower latency.

To use native post processor, please pass use_native=True to Model.load() or Model.load_async(). The following is an example to use native post processor for SSDMobileNet. You can find more details of each model page.

Example

from furiosa.models.vision import SSDMobileNet
from furiosa.runtime import session

image = ["tests/assets/cat.jpg"]

mobilenet = SSDMobileNet.load(use_native=True)
with session.create(mobilenet.enf) as sess:
    inputs, contexts = mobilenet.preprocess(image)
    outputs = sess.run(inputs).numpy()
    mobilenet.postprocess(outputs, contexts[0])

Model object

Loading a pre-trained model

Accessing artifacts and metadata

furiosa.models.types.Model

Inferencing with Session API

Pre/Postprocessing

See Also

`furiosa.models.types.Model`