Detailed Description

Defines Deep Neural Network (DNN) module for performing inference using NVIDIA^® TensorRT^™ models.

Data Structures
struct	dwDNNCustomLayer
	Specifies plugin configuration. More...

struct	dwDNNMetaData
	Specifies TensorRT model header. More...

struct	dwDNNPluginConfiguration
	Specified plugin configuration. More...

Modules
	DNN Plugin
	Provides an interface for supporting non-standard DNN layers.

Typedefs
typedef struct dwDNNObject const *	dwConstDNNHandle_t

typedef struct dwDNNObject *	dwDNNHandle_t
	Handles representing Deep Neural Network interface. More...

Functions
DW_API_PUBLIC dwStatus	dwDNN_getCUDAStream (cudaStream_t *const stream, dwDNNHandle_t const network)
	Gets the CUDA stream used by the feature list. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputBlobCount (uint32_t *const count, dwDNNHandle_t const network)
	Gets the input blob count. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputIndex (uint32_t const blobIndex, const char8_t const blobName, dwDNNHandle_t const network)
	Gets the index of an input blob with a given blob name. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputSize (dwBlobSize *const blobSize, uint32_t const blobIndex, dwDNNHandle_t const network)
	Gets the input blob size at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_getInputTensorProperties (dwDNNTensorProperties *const tensorProps, uint32_t const blobIndex, dwDNNHandle_t const network)
	Gets the input tensor properties at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_getMetaData (dwDNNMetaData *const metaData, dwDNNHandle_t const network)
	Returns the metadata for the associated network model. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputBlobCount (uint32_t *const count, dwDNNHandle_t const network)
	Gets the output blob count. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputIndex (uint32_t const blobIndex, const char8_t const blobName, dwDNNHandle_t const network)
	Gets the index of an output blob with a given blob name. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputSize (dwBlobSize *const blobSize, uint32_t const blobIndex, dwDNNHandle_t const network)
	Gets the output blob size at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_getOutputTensorProperties (dwDNNTensorProperties *const tensorProps, uint32_t const blobIndex, dwDNNHandle_t const network)
	Gets the output tensor properties at blobIndex. More...

DW_API_PUBLIC dwStatus	dwDNN_infer (dwDNNTensorHandle_t const outputTensors, uint32_t const outputTensorCount, dwConstDNNTensorHandle_t const inputTensors, uint32_t const inputTensorCount, dwDNNHandle_t const network)
	Runs inference pipeline on the given input. More...

DW_API_PUBLIC dwStatus	dwDNN_inferRaw (float32_t const const dOutput, const float32_t const const dInput, uint32_t const batchsize, dwDNNHandle_t const network)
	Forwards pass from all input blobs to all output blobs. More...

DW_API_PUBLIC dwStatus	dwDNN_inferSIO (float32_t const dOutput, const float32_t const dInput, uint32_t const batchsize, dwDNNHandle_t const network)
	Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network). More...

DW_API_PUBLIC dwStatus	dwDNN_initializeTensorRTFromFile (dwDNNHandle_t const network, const char8_t const modelFilename, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, dwContextHandle_t const context)
	Creates and initializes a TensorRT Network from file. More...

DW_API_PUBLIC dwStatus	dwDNN_initializeTensorRTFromFileWithEngineId (dwDNNHandle_t const network, const char8_t const modelFilename, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, uint32_t engineId, dwContextHandle_t const context)
	Creates and initializes a TensorRT Network from file with DLA Engine ID. More...

DW_API_PUBLIC dwStatus	dwDNN_initializeTensorRTFromMemory (dwDNNHandle_t const network, const char8_t const modelContent, uint32_t const modelContentSize, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, dwContextHandle_t const context)
	Creates and initializes a TensorRT Network from memory. More...

DW_API_PUBLIC dwStatus	dwDNN_initializeTensorRTFromMemoryWithEngineId (dwDNNHandle_t const network, const char8_t const modelContent, uint32_t const modelContentSize, const dwDNNPluginConfiguration *const pluginConfiguration, dwProcessorType const processorType, uint32_t engineId, dwContextHandle_t const context)
	Creates and initializes a TensorRT Network from memory with DLA Engine ID. More...

DW_API_PUBLIC dwStatus	dwDNN_release (dwDNNHandle_t const network)
	Releases a given network. More...

DW_API_PUBLIC dwStatus	dwDNN_reset (dwDNNHandle_t const network)
	Resets a given network. More...

DW_API_PUBLIC dwStatus	dwDNN_setCUDAStream (cudaStream_t const stream, dwDNNHandle_t const network)
	Sets the CUDA stream for infer operations. More...

Data Structure Documentation

◆ dwDNNCustomLayer

struct dwDNNCustomLayer

Data Fields
const char8_t *	layerName	Name of the custom layer.
const char8_t *	pluginLibraryPath	Path to a plugin shared object. Path must be either absolute path or path relative to DW lib folder.

◆ dwDNNMetaData

struct dwDNNMetaData

Data Fields
dwDataConditionerParams	dataConditionerParams	DataConditioner parameters for running this network.

◆ dwDNNPluginConfiguration

struct dwDNNPluginConfiguration

Data Fields
const dwDNNCustomLayer *	customLayers	Array of custom layers.
size_t	numCustomLayers	Number of custom layers.

Typedef Documentation

◆ dwConstDNNHandle_t

typedef struct dwDNNObject const* dwConstDNNHandle_t

Definition at line 63 of file DNN.h.

◆ dwDNNHandle_t

typedef struct dwDNNObject* dwDNNHandle_t

Handles representing Deep Neural Network interface.

Definition at line 62 of file DNN.h.

Function Documentation

◆ dwDNN_getCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_getCUDAStream	(	cudaStream_t *const	stream,
		dwDNNHandle_t const	network
	)

Gets the CUDA stream used by the feature list.

Parameters

[out]	stream	The CUDA stream currently used.
[in]	network	A handle to the DNN module.

Returns: DW_INVALID_HANDLE if the given network handle or the stream are NULL or of wrong type.
DW_SUCCESS otherwise.

◆ dwDNN_getInputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getInputBlobCount	(	uint32_t *const	count,
		dwDNNHandle_t const	network
	)

Gets the input blob count.

Parameters

[out]	count	A pointer to the number of input blobs.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getInputIndex()

DW_API_PUBLIC dwStatus dwDNN_getInputIndex	(	uint32_t *const	blobIndex,
		const char8_t *const	blobName,
		dwDNNHandle_t const	network
	)

Gets the index of an input blob with a given blob name.

Parameters

[out]	blobIndex	A pointer to the index of the blob with the given name.
[in]	blobName	A pointer to the name of an input blob.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle, blobIndex or blobName are null or an input blob with the given name is not found.
DW_SUCCESS otherwise.

◆ dwDNN_getInputSize()

DW_API_PUBLIC dwStatus dwDNN_getInputSize	(	dwBlobSize *const	blobSize,
		uint32_t const	blobIndex,
		dwDNNHandle_t const	network
	)

Gets the input blob size at blobIndex.

Parameters

[out]	blobSize	A pointer to the where the input blob size is returned.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getInputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getInputTensorProperties	(	dwDNNTensorProperties *const	tensorProps,
		uint32_t const	blobIndex,
		dwDNNHandle_t const	network
	)

Gets the input tensor properties at blobIndex.

Parameters

[out]	tensorProps	Tensor properties.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getInputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or the blob size are NULL or blobIndex is not in range [0 dwDNN_getInputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getMetaData()

DW_API_PUBLIC dwStatus dwDNN_getMetaData	(	dwDNNMetaData *const	metaData,
		dwDNNHandle_t const	network
	)

Returns the metadata for the associated network model.

Parameters

[out]	metaData	A pointer to a metadata structure.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or metaData are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputBlobCount()

DW_API_PUBLIC dwStatus dwDNN_getOutputBlobCount	(	uint32_t *const	count,
		dwDNNHandle_t const	network
	)

Gets the output blob count.

Parameters

[out]	count	A pointer to the number of output blobs.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle or count are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_getOutputIndex()

DW_API_PUBLIC dwStatus dwDNN_getOutputIndex	(	uint32_t *const	blobIndex,
		const char8_t *const	blobName,
		dwDNNHandle_t const	network
	)

Gets the index of an output blob with a given blob name.

Parameters

[out]	blobIndex	A pointer to the index of the blob with the given name.
[in]	blobName	A pointer to the name of an output blob.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_HANDLE - if provided network handle, blobIndex or blobName are NULL or an output blob with the given name is not found or the network handle is of wrong type
DW_SUCCESS otherwise.

◆ dwDNN_getOutputSize()

DW_API_PUBLIC dwStatus dwDNN_getOutputSize	(	dwBlobSize *const	blobSize,
		uint32_t const	blobIndex,
		dwDNNHandle_t const	network
	)

Gets the output blob size at blobIndex.

Parameters

[out]	blobSize	A pointer to the where the output blob size is returned.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_getOutputTensorProperties()

DW_API_PUBLIC dwStatus dwDNN_getOutputTensorProperties	(	dwDNNTensorProperties *const	tensorProps,
		uint32_t const	blobIndex,
		dwDNNHandle_t const	network
	)

Gets the output tensor properties at blobIndex.

Parameters

[out]	tensorProps	Tensor properties.
[in]	blobIndex	Specifies the blob index; must be in the range [0 dwDNN_getOutputBlobCount()-1].
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle is or blobSize are NULL or blobIndex is not in range [0 dwDNN_getOutputBlobCount()-1].
DW_SUCCESS otherwise.

◆ dwDNN_infer()

DW_API_PUBLIC dwStatus dwDNN_infer	(	dwDNNTensorHandle_t *const	outputTensors,
		uint32_t const	outputTensorCount,
		dwConstDNNTensorHandle_t *const	inputTensors,
		uint32_t const	inputTensorCount,
		dwDNNHandle_t const	network
	)

Runs inference pipeline on the given input.

Parameters

[out]	outputTensors	Output tensors.
[in]	outputTensorCount	Number of output tensors.
[in]	inputTensors	Input tensors.
[in]	inputTensorCount	Number of input tensors.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle, input or output are NULL.
DW_SUCCESS otherwise.

◆ dwDNN_inferRaw()

DW_API_PUBLIC dwStatus dwDNN_inferRaw	(	float32_t const const	dOutput,
		const float32_t const const	dInput,
		uint32_t const	batchsize,
		dwDNNHandle_t const	network
	)

Forwards pass from all input blobs to all output blobs.

Note: : The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().; : The umber of blobs must match the respective number of blobs returned by dwDNN_getInputBlobCount() and dwDNN_getOutputBlobCount().

Parameters

[out]	dOutput	A pointer to an array of pointers to the input blobs in GPU Memory.
[in]	dInput	A pointer to an array of pointers to the output blobs in GPU Memory.
[in]	batchsize	Batch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle , output or input are NULL.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.

Note: Inference performance might be suboptimal if the network has been generated for a larger batch size.; If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_inferSIO()

DW_API_PUBLIC dwStatus dwDNN_inferSIO	(	float32_t *const	dOutput,
		const float32_t *const	dInput,
		uint32_t const	batchsize,
		dwDNNHandle_t const	network
	)

Forwards pass from the first input blob to the first output blob (a shortcut for a single input - single output network).

Note: This method requires the network to be single input - single output.; The size of the output blob and input blob must match the respective sizes returned by dwDNN_getOutputSize() and dwDNN_getInputSize().

Parameters

[out]	dOutput	A pointer to the output blob in GPU memory.
[in]	dInput	A pointer to the input blob in GPU memory.
[in]	batchsize	Batch size for inference. Batch size must be equal or less than the one that was given at the time of model generation.
[in]	network	Network handle created with dwDNN_initialize().

Returns: DW_INVALID_ARGUMENT - if provided network handle, output or input are NULL or if provided network is not a single input - single output network.
DW_INTERNAL_ERROR - if DNN engine cannot execute inference on the given network.
DW_SUCCESS otherwise.

Note: Inference performance might be suboptimal if the network has been generated for a larger batch size.; If the processor type is DLA, the inference will always perform on maximum batch size. The only advantage of giving a lower batchsize as input is that the output to be copied will be smaller in size.

◆ dwDNN_initializeTensorRTFromFile()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFile	(	dwDNNHandle_t *const	network,
		const char8_t *const	modelFilename,
		const dwDNNPluginConfiguration *const	pluginConfiguration,
		dwProcessorType const	processorType,
		dwContextHandle_t const	context
	)

Creates and initializes a TensorRT Network from file.

Parameters

[out]	network	A pointer to network handle that will be initialized from parameters.
[in]	modelFilename	A pointer to the name of the TensorRT model file.
[in]	pluginConfiguration	An optional pointer to plugin configuration for custom layers.
[in]	processorType	Processor that the inference should run on. Note that the model must be generated for this processor type.
[in]	context	Specifies the handle to the context under which the DNN module is created.

Returns: DW_INVALID_ARGUMENT - if pointer to the network handle or the model filename are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_FILE_NOT_FOUND - if given model file does not exist.
DW_SUCCESS otherwise.

Note

The network file must be created by the TensorRT_optimization tool.

DNN module will look for metadata file named <modelFilename>.json in the same folder. If it is present, metadata will be loaded from that file. Otherwise, it will be filled with default values. Example metadata:

{
    "dataConditionerParams" : {
        "meanValue" : [0.0, 0.0, 0.0],
        "splitPlanes" : true,
        "pixelScaleCoefficient": 1.0,
        "ignoreAspectRatio" : false,
        "doPerPlaneMeanNormalization" : false
    }
    "tonemapType" : "none",
    "__comment": "tonemapType can be one of {none, agtm}"
}

\

◆ dwDNN_initializeTensorRTFromFileWithEngineId()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromFileWithEngineId	(	dwDNNHandle_t *const	network,
		const char8_t *const	modelFilename,
		const dwDNNPluginConfiguration *const	pluginConfiguration,
		dwProcessorType const	processorType,
		uint32_t	engineId,
		dwContextHandle_t const	context
	)

Creates and initializes a TensorRT Network from file with DLA Engine ID.

Parameters

[out]	network	A pointer to network handle that will be initialized from parameters.
[in]	modelFilename	A pointer to the name of the TensorRT model file.
[in]	pluginConfiguration	An optional pointer to plugin configuration for custom layers.
[in]	processorType	Processor that the inference should run on. Note that the model must be generated for this processor type.
[in]	engineId	Specifies the DLA engine id if the processorType is DW_PROCESSOR_TYPE_CUDLA.
[in]	context	Specifies the handle to the context under which the DNN module is created.

Returns: DW_INVALID_ARGUMENT - if pointer to the network handle or the model filename are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_FILE_NOT_FOUND - if given model file does not exist.
DW_SUCCESS otherwise.

◆ dwDNN_initializeTensorRTFromMemory()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemory	(	dwDNNHandle_t *const	network,
		const char8_t *const	modelContent,
		uint32_t const	modelContentSize,
		const dwDNNPluginConfiguration *const	pluginConfiguration,
		dwProcessorType const	processorType,
		dwContextHandle_t const	context
	)

Creates and initializes a TensorRT Network from memory.

Parameters

[out]	network	A pointer to network handle that is initialized from parameters.
[in]	modelContent	A pointer to network content in the memory.
[in]	modelContentSize	Specifies the size of network content in memory, in bytes.
[in]	pluginConfiguration	An optional pointer to plugin configuration for custom layers.
[in]	processorType	Processor that the inference should run on. Note that the model must be generated for this processor type.
[in]	context	Specifies a handle to the context under which the DNN module is created.

Returns: DW_INVALID_ARGUMENT - if pointer to the network handle or the model content are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_SUCCESS otherwise.

Note: The network file must be created by the TensorRT_optimization tool.; DNN module will fill metadata with default values.

◆ dwDNN_initializeTensorRTFromMemoryWithEngineId()

DW_API_PUBLIC dwStatus dwDNN_initializeTensorRTFromMemoryWithEngineId	(	dwDNNHandle_t *const	network,
		const char8_t *const	modelContent,
		uint32_t const	modelContentSize,
		const dwDNNPluginConfiguration *const	pluginConfiguration,
		dwProcessorType const	processorType,
		uint32_t	engineId,
		dwContextHandle_t const	context
	)

Creates and initializes a TensorRT Network from memory with DLA Engine ID.

Parameters

[out]	network	A pointer to network handle that is initialized from parameters.
[in]	modelContent	A pointer to network content in the memory.
[in]	modelContentSize	Specifies the size of network content in memory, in bytes.
[in]	pluginConfiguration	An optional pointer to plugin configuration for custom layers.
[in]	processorType	Processor that the inference should run on. Note that the model must be generated for this processor type.
[in]	engineId	Specifies the DLA engine id if the processorType is DW_PROCESSOR_TYPE_CUDLA.
[in]	context	Specifies a handle to the context under which the DNN module is created.

Returns: DW_INVALID_ARGUMENT - if pointer to the network handle or the model content are NULL.
DW_DNN_INVALID_MODEL - if the provided model is invalid.
DW_CUDA_ERROR - if compute capability does not met the network type's requirements.
DW_SUCCESS otherwise.

◆ dwDNN_release()

DW_API_PUBLIC dwStatus dwDNN_release ( dwDNNHandle_t const network )

Releases a given network.

Parameters

[in] network The network handle to release.

Returns: DW_INVALID_ARGUMENT - if provided network handle is NULL.
DW_SUCCESS otherwise.

◆ dwDNN_reset()

DW_API_PUBLIC dwStatus dwDNN_reset ( dwDNNHandle_t const network )

Resets a given network.

Parameters

[in] network Network handle to reset.

Returns: DW_INVALID_ARGUMENT - if provided network handle is null.
DW_SUCCESS otherwise.

◆ dwDNN_setCUDAStream()

DW_API_PUBLIC dwStatus dwDNN_setCUDAStream	(	cudaStream_t const	stream,
		dwDNNHandle_t const	network
	)

Sets the CUDA stream for infer operations.

Note: The ownership of the stream remains by the callee.

Parameters

[in]	stream	The CUDA stream to be used. Default is the stream 0, resulting in synchronous operations.
[in]	network	A handle to the DNN module to set CUDA stream for.

Returns: DW_INVALID_ARGUMENT if the given network handle is NULL.
DW_SUCCESS otherwise.

Detailed Description

Data Structures

Modules

Typedefs

Functions

Data Structure Documentation

◆ dwDNNCustomLayer

◆ dwDNNMetaData

◆ dwDNNPluginConfiguration

Typedef Documentation

◆ dwConstDNNHandle_t

◆ dwDNNHandle_t

Function Documentation

◆ dwDNN_getCUDAStream()

◆ dwDNN_getInputBlobCount()

◆ dwDNN_getInputIndex()

◆ dwDNN_getInputSize()

◆ dwDNN_getInputTensorProperties()

◆ dwDNN_getMetaData()

◆ dwDNN_getOutputBlobCount()

◆ dwDNN_getOutputIndex()

◆ dwDNN_getOutputSize()

◆ dwDNN_getOutputTensorProperties()

◆ dwDNN_infer()

◆ dwDNN_inferRaw()

◆ dwDNN_inferSIO()

◆ dwDNN_initializeTensorRTFromFile()

◆ dwDNN_initializeTensorRTFromFileWithEngineId()

◆ dwDNN_initializeTensorRTFromMemory()

◆ dwDNN_initializeTensorRTFromMemoryWithEngineId()

◆ dwDNN_release()

◆ dwDNN_reset()

◆ dwDNN_setCUDAStream()