Calibration

The module dwCalibratedCamera has been separated from dwRig renamed to dwCameraModel. It resides in the folder dw/calibration/cameramodel. It provides functions to handle calibrated intrinsic camera models.

Header file has been renamed from Camera.h to CameraModel.h.
The object handles have been renamed to dwCameraModelHandle_t and dwConstCameraModelHandle_t, respectively.
The prefixes for all functions declared in CameraModel.h have been updated from dwCalibratedCamera to dwCameraModel.
dwCameraModel_applyImageTransform() has been updated and now it takes the additional parameter bool updateMasks.
dwCameraModel_initialize has been moved here from dwRig.

For more information see Rig Configuration.

DNN

This release adds support for generating and using DLA (Deep-Learning-Accelerator) models. This change requires selection of the processor type at the time of initialization; therefore, the following functions' signatures have been updated accordingly:

dwDNN_initializeTensorRTFromFile() has been renamed to dwDNN_initializeTensorRTFromFileNew() and requires an additional argument of type dwProcessorType indicating on which processor the model shall run on. Note that the model must be generated on the corresponding processor.
dwDNN_initializeTensorRTFromMemory() has been renamed to dwDNN_initializeTensorRTFromMemoryNew() and requires an additional argument of type dwProcessorType indicating on which processor the model shall run on. Note that the model must be generated on the corresponding processor.

Note: The old APIs and the New suffix for the new APIs will be removed in the next major release.

For more information see DNN.

Egomotion

dwEgoMotionParameters has been renamed to dwEgomotionParameters.
dwEgoMotionSensorCharacteristics has been renamed to dwEgomotionSensorCharacteristics.
The dwEgomotionParameters.enableLocationFilter has been renamed to dwEgomotionParameters.disableLocationFilter.
dw/radarmotion has been moved to dw/egomotion/radar.

For more information see Egomotion.

Iterative Closest Point

ICP has been completelly moved to point cloud processing module.

For more information see Point Cloud ICP.

Image Processing

Port `dwPyramidHandle_t` to `dwPyramidImage`

Old name	New name
dwPyramid_initialize()	dwPyramid_create()
dwPyramid_release()	dwPyramid_destroy()
dwPyramid_build()	dwImageFilter_computePyramid()
dwPyramid_getLevelCount()	dwPyramidImage::levelCount

dwPyramidImage::levelImages[level] provides an dwImageHandle_t for the (level)-th image, dwImage_getProperties() acquires the image size, and dwImage_getCUDA() acquires dwImageCUDA data.

Example: Old code for creating, building and destroying a pyramid:

// Old code
dwPyramidConfig config{};
config.width = width;
config.height = height;
config.levelCount = numLevel;
config.dataType = DW_TYPE_UINT8;
dwPyramid_initialize(&pyramid, &config, 0, context);
dwPyramid_build(&singlePlaneImage, pyramid);
dwPyramid_release(pyramid);

New code for creating, building and destroying a pyramid:

// New code
dwPyramid_create(&pyramid, numLevel, width, height, DW_TYPE_UINT8, context);
dwImageFilter_computePyramid(&pyramid, &singlePlaneImage, 0, context);
dwPyramid_destroy(pyramid);

Port `dwFeatureListHandle_t` and `dwFeatureListPointers` to `dwFeatureHistoryArray`.

dwFeatureStatus has been renamed to dwFeature2DStatus.
DW_FEATURE_STATUS_XXX has been renamed to DW_FEATURE2D_STATUS_XXX.

The new dwFeatureHistoryArray struct provides pointers such as: *statuses, *ages, *ids, *locationHistory and *featureCount like the old dwFeatureListPointers struct did. It is also created and destroyed like the old handle dwFeatureListHandle_t. dwFeatureHistoryArray can be created on page-able/pinned CPU or CUDA memory.

Old name	New name
dwFeatureListPointers::statuses	dwFeatureHistoryArray::statuses
dwFeatureListPointers::ages	dwFeatureHistoryArray::ages
—	dwFeatureHistoryArray::ids
dwFeatureListPointers::locationHistory	dwFeatureHistoryArray::locationHistory
dwFeatureListPointers::featureCount	dwFeatureHistoryArray::featureCount
dwFeatureList_initialize()	dwFeatureHistoryArray_create()
dwFeatureList_release()	dwFeatureHistoryArray_destroy()
dwFeatureList_reset()	dwFeatureHistoryArray_reset()
dwFeatureList_getDataBasePointer()	dwFeatureHistoryArray::data, dwFeatureHistoryArray::bytes
dwFeatureList_getCurrentTimeIdx()	dwFeatureHistoryArray::currentTimeIdx
dwFeatureList_setCurrentTimeIdx()	dwFeatureHistoryArray::currentTimeIdx
dwFeatureList_getFeatureListSize()	dwFeatureHistoryArray::maxFeatures, dwFeatureHistoryArray::maxHistory

For new dw_imageprocessing APIs, there's no need to call selectValid(), compact() and proximityFilter() explicitly. They are automatically called within the DriveWorks module.

Example: In the old dw_feature APIs, the application needed to create a dwFeatureListHandle_t object and call dwFeatureList_getDataPointers() to get the interpretable dwFeatureListPointers.

// Old code
dwFeatureList_initialize(&featureList, maxFeatureCount, historyCapacity, width, height, 0, context);
void* d_database8;
size_t databytes;
dwFeatureListPointers featureDataGPU;
dwFeatureList_getDataBasePointer(&d_database8, &databytes, featureList);
dwFeatureList_getDataPointers(&featureDataGPU, d_database8, featureList));
 
// access the pointers of dwFeatureListPointers on GPU
featureDataGPU.statuses[...]
featureDataGPU.locationHistory[...]

If the application needed a CPU copy of the featureList handle, it first had to allocate CPU memory and call dwFeatureList_getDataPointers() to interpret it into dwFeatureListPointers.

// Old code
uint8_t* h_database8 = new uint8_t[databytes];
dwFeatureListPointers featureDataCPU;
dwFeatureList_getDataPointers(&featureDataCPU, h_database8, featureList));
 
// access the pointers of dwFeatureListPointers on CPU
featureDataCPU.statuses[...]
featureDataCPU.locationHistory[...]

It also needed the raw data pointer to copy data between the CPU and GPU.

// GPU -> CPU copy
cudaMemcpy(h_database8, d_database8, databytes, cudaMemcpyDeviceToHost);
 
// CPU -> GPU copy
cudaMemcpy(d_database8, h_database8, databytes, cudaMemcpyHostToDevice);

In the new dw_imageprocessing APIs, the application only needs to create the dwFeatureHistoryArray object on the CPU and GPU. It provides both raw pointers and interpretable pointers. It also provides the dwFeatureHistoryArray_copyAsync() API to easily copy data between two feature history arrays.

// New code
dwFeatureHistoryArray_create(&featureHistoryCPU, maxFeatureCount, historyCapacity, DW_MEMORY_TYPE_CPU, context);
dwFeatureHistoryArray_create(&featureHistoryGPU, maxFeatureCount, historyCapacity, DW_MEMORY_TYPE_CUDA, context);
 
// access the pointers on GPU
featureHistoryGPU.statuses[...]
featureHistoryGPU.locationHistory[...]
 
// access the pointers on CPU
featureHistoryCPU.statuses[...]
featureHistoryCPU.locationHistory[...]
 
// GPU -> CPU copy
dwFeatureHistoryArray_copyAsync(&featureHistoryCPU, &featureHistoryGPU, 0);
 
// CPU -> GPU copy
dwFeatureHistoryArray_copyAsync(&featureHistoryGPU, &featureHistoryCPU, 0);

The application can also create dwFeatureHistoryArray on pinned memory, it only needs to change DW_MEMORY_TYPE_CPU to DW_MEMORY_TYPE_PINNED during creation.

Like dwFeatureListHandle_t, there are also reset() and destroy() APIs. dwFeatureHistoryArray must be destroyed before the application exits.

// New code
dwFeatureHistoryArray_destroy(featureHistoryCPU);
dwFeatureHistoryArray_destroy(featureHistoryGPU);

Get particular `timeIdx` slice from `dwFeatureHistoryArray`

Similar to dwFeatureListPointers::locationHistory, dwFeatureHistoryArray::locationHistory stores historyCapacity time period of maxFeatureCount features. To get a particular feature[i] for timeIdx, it used to be:

// Old code
uint32_t currentTimeIdx;
dwFeatureList_getCurrentTimeIdx(&currentTimeIdx, featureList);
 
uint32_t histIdx = (currentTimeIdx + timeIdx) % historyCapacity;
dwVector2f* pLocation = featureDataCPU.locationHistory + histIdx * maxFeatureCount;
 
// get the particular feature[i]
pLocation[i]...

The new dw_imageprocessing modules provides a dwFeatureArray struct that contains features for a single time slice, so there's no need to compute histIdx any more.

// New code
dwFeatureArray features;
dwFeatureHistoryArray_get(&features, timeIdx, &featureHistoryCPU);
 
// get the particular feature[i]
features.locations[i]...

It can also get the most recent (top-most) time slice, or the slice that is 1 frame earlier than the top-most.

// New code
dwFeatureArray currentFeatures, previousFeatures;
dwFeatureHistoryArray_getCurrent(&currentFeatures, &featureHistoryCPU);
dwFeatureHistoryArray_getPrevious(&previousFeatures, &featureHistoryCPU);
 
// get the particular feature[i]
currentFeatures.locations[i]...
previousFeatures.locations[i]...

dwFeatureArray acquired by dwFeatureHistoryArray_getXXX APIs don't need to be destroyed as they are only shallow copies. Resources are automatically freed when the corresponding dwFeatureHistoryArray is destroyed. However, dwFeatureArray also provides creation APIs, where the usage is very similar to dwFeatureHistoryArray:

// New code
dwFeatureArray_create(&featureCPU, maxFeatureCount, DW_MEMORY_TYPE_CPU, context);
dwFeatureArray_create(&featureGPU, maxFeatureCount, DW_MEMORY_TYPE_CUDA, context);

dwFeatureArray created by dwFeatureArray_create() APIs must be destroyed by dwFeatureArray_destroy() before the application exits. Similarly, there is the dwFeatureArray_copyAsync() API to copy data between two feature array objects.

Port `dwFeatureTrackerHandle_t` to `dwFeature2DDetectorHandle_t` and `dwFeature2DTrackerHandle_t`

Harris corner detector handle `dwFeature2DDetectorHandle_t`

New imageprocessing module splits the Harris corner detector from dwFeatureTrackerHandle_t. The new Harris detector handle is dwFeature2DDetectorHandle_t.

Old name	New name
dwFeatureTracker_initialize()	dwFeature2DDetector_initialize()
dwFeatureTracker_initDefaultParams()	dwFeature2DDetector_initDefaultParams()
dwFeatureTracker_reset()	dwFeature2DDetector_reset()
dwFeatureTracker_release()	dwFeature2DDetector_release()
dwFeatureTracker_setMask()	dwFeature2DDetector_setMask()
dwFeatureTracker_detectNewFeatures()	dwFeature2DDetector_detectFromPyramid()
—	dwFeature2DDetector_detectFromImage()

The new dwFeature2DDetectorConfig parameter not only covers all settings from the old dwFeatureTrackerConfig parameter, it also provides more flexible settings.

Old dwFeatureTrackerConfig	New dwFeature2DDetectorConfig
—	type
dwFeatureTrackerConfig::imageWidth	dwFeature2DDetectorConfig::imageWidth
dwFeatureTrackerConfig::imageHeight	dwFeature2DDetectorConfig::imageHeight
dwFeatureTrackerConfig::maxFeatureCount	dwFeature2DDetectorConfig::maxFeatureCount
—	dwFeature2DDetectorConfig::harrisK
dwFeatureTrackerConfig::cellSize	dwFeature2DDetectorConfig::cellSize
dwFeatureTrackerConfig::detectorScoreThreshold	dwFeature2DDetectorConfig::scoreThreshold
dwFeatureTrackerConfig::detectorDetailThreshold	dwFeature2DDetectorConfig::detailThreshold
dwFeatureTrackerConfig::numEvenDistributionPerCell	dwFeature2DDetectorConfig::numEvenDistributionPerCell
—	dwFeature2DDetectorConfig::detectionLevel
—	dwFeature2DDetectorConfig::harrisRadius
—	dwFeature2DDetectorConfig::NMSRadius
—	dwFeature2DDetectorConfig::maskType
—	dwFeature2DDetectorConfig::isMaskAdjustmentEnabled

Example of initializing a detector with the old code:

// Old code
// Detector and Tracker share the same dwFeatureTrackerHandle_t
dwFeatureTrackerConfig trackerConfig{};
dwFeatureTracker_initDefaultParams(&trackerConfig);
trackerConfig.imageWidth                 = imageWidth;
trackerConfig.imageHeight                = imageHeight;
trackerConfig.maxFeatureCount            = maxFeatureCount;
trackerConfig.cellSize                   = cellSize;
trackerConfig.detectorScoreThreshold     = scoreThreshold;
trackerConfig.detectorDetailThreshold    = detailThreshold;
trackerConfig.numEvenDistributionPerCell = numEvenDistributionPerCell;
 
dwFeatureTracker_initialize(&tracker, &trackerConfig, 0, context);

Example of initializing a detector with the new code:

// New code
dwFeature2DDetectorConfig detectorConfig = {};
dwFeature2DDetector_initDefaultParams(&detectorConfig);
detectorConfig.imageWidth                 = imageWidth;
detectorConfig.imageHeight                = imageHeight;
detectorConfig.maxFeatureCount            = maxFeatureCount;
detectorConfig.cellSize                   = cellSize;
detectorConfig.scoreThreshold             = scoreThreshold;
detectorConfig.detailThreshold            = detailThreshold;
detectorConfig.numEvenDistributionPerCell = numEvenDistributionPerCell;
 
dwFeature2DDetector_initialize(&detector, &detectorConfig, 0, context);

LK tracker handle `dwFeature2DTrackerHandle_t`

The new imageprocessing module splits LK tracker from dwFeatureTrackerHandle_t. The new LK tracker handle is dwFeature2DTrackerHandle_t.

Old name	New name
dwFeatureTracker_initialize()	dwFeature2DTracker_initialize()
dwFeatureTracker_initDefaultParams()	dwFeature2DTracker_initDefaultParams()
dwFeatureTracker_reset()	dwFeature2DTracker_reset()
dwFeatureTracker_release()	dwFeature2DTracker_release()
dwFeatureTracker_trackFeatures()	dwFeature2DTracker_trackFeatures()
—	dwFeature2DTracker_getNewToOldMap()

The new dwFeature2DTrackerConfig parameter not only covers all settings from the the old dwFeatureTrackerConfig parameter, it also provides more flexible settings.

Old dwFeatureTrackerConfig	New dwFeature2DTrackerConfig
—	dwFeature2DTrackerConfig::algorithm
—	dwFeature2DTrackerConfig::detectorType
—	dwFeature2DTrackerConfig::isOutputCompact
dwFeatureTrackerConfig::imageWidth	dwFeature2DTrackerConfig::imageWidth
dwFeatureTrackerConfig::imageHeight	dwFeature2DTrackerConfig::imageHeight
dwFeatureTrackerConfig::maxFeatureCount	dwFeature2DTrackerConfig::maxFeatureCount
—	dwFeature2DTrackerConfig::historyCapacity
dwFeatureTrackerConfig::windowSizeLK	dwFeature2DTrackerConfig::windowSizeLK
dwFeatureTrackerConfig::iterationsLK	dwFeature2DTrackerConfig::numIterTranslationOnly
—	dwFeature2DTrackerConfig::numIterScaling
—	dwFeature2DTrackerConfig::numLevelTranslationOnly
—	dwFeature2DTrackerConfig::maxScaleChange
—	dwFeature2DTrackerConfig::nccKillThreshold
—	dwFeature2DTrackerConfig::nccUpdateThreshold

Example of initializing a tracker with the old code:

// Old code
// Detector and Tracker share the same dwFeatureTrackerHandle_t
dwFeatureTrackerConfig trackerConfig{};
dwFeatureTracker_initDefaultParams(&trackerConfig);
trackerConfig.imageWidth      = imageWidth;
trackerConfig.imageHeight     = imageHeight;
trackerConfig.maxFeatureCount = maxFeatureCount;
trackerConfig.windowSizeLK    = windowSize;
trackerConfig.iterationsLK    = numIterTranslation;
 
dwFeatureTracker_initialize(&tracker, &trackerConfig, 0, context);

Example of initializing a tracker with the new code:

// New code
dwFeature2DTrackerConfig trackerConfig = {};
dwFeature2DTracker_initDefaultParams(&trackerConfig);
trackerConfig.detectorType            = detectorConfig.type;
trackerConfig.maxFeatureCount         = maxFeatureCount;
trackerConfig.historyCapacity         = historyCapacity;
trackerConfig.imageWidth              = imageWidth;
trackerConfig.imageHeight             = imageHeight;
trackerConfig.windowSizeLK            = windowSize;
trackerConfig.numIterTranslationOnly  = numIterTranslation;
 
dwFeature2DTracker_initialize(&tracker, &trackerConfig, 0, context);

Use the new `imageprocessing` module to detect and track Harris corners

Here's a complete example showing the pipeline with the old dw_feature APIs and new dw_imageprocessing APIs.

// Old dw_features code
 
// Create feature list
dwFeatureListHandle_t featureList;
dwFeatureList_initialize(&featureList, maxFeatureCount, historyCapacity, width, height, 0, context);
 
// create pyramid
dwPyramidHandle_t pyramidCurrent, pyramidPrevious;
dwPyramidConfig config{};
config.width = width;
config.height = height;
config.levelCount = numLevel;
config.dataType = DW_TYPE_UINT8;
dwPyramid_initialize(&pyramidCurrent, &config, cudaStream, context);
dwPyramid_initialize(&pyramidPrevious, &config, cudaStream, context);
 
// create tracker and detector
dwFeatureTrackerConfig trackerConfig{};
dwFeatureTracker_initDefaultParams(&trackerConfig);
trackerConfig.imageWidth                 = width;
trackerConfig.imageHeight                = height;
trackerConfig.maxFeatureCount            = maxFeatureCount;
dwFeatureTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
 
// Old dw_feature APIs requires additional CUDA buffers for compact.
uint32_t *d_validCount, *d_validIndexes, *d_invalidCount, *d_invalidIndexes;
cudaMalloc(&d_validCount, sizeof(uint32_t));
cudaMalloc(&d_validIndexes, sizeof(uint32_t) * maxFeatureCount);
cudaMalloc(&d_invalidCount, sizeof(uint32_t));
cudaMalloc(&d_invalidIndexes, sizeof(uint32_t) * maxFeatureCount);
 
// Assuming sensor provides single channel image as input
while (image = camera->getFrame())
{
    std::swap(pyramidCurrent, pyramidPrevious);
    dwPyramid_build(&image, pyramidCurrent);
 
    // Track
    dwFeatureTracker_trackFeatures(featureList, pyramidPrevious, pyramidCurrent, nullptr, tracker);
 
    // Apply proximity filters to make features uniformly distributed
    dwFeatureList_proximityFilter(featureList);
 
    // Determine which features to throw away
    dwFeatureList_selectValid(d_validCount, d_validIndexes,
                              d_invalidCount, d_invalidIndexes, featureList);
 
    // Compact list
    dwFeatureList_compact(featureList, d_validCount, d_validIndexes,
                         d_invalidCount, d_invalidIndexes);
 
    // Detect
    dwFeatureTracker_detectNewFeatures(featureList, pyramidCurrent, tracker);
}
 
// release resources
dwFeatureTracker_release(tracker);
dwPyramid_release(pyramidCurrent);
dwPyramid_release(pyramidPrevious);
dwFeatureList_release(featureList);
 
cudaFree(d_validCount);
cudaFree(d_validIndexes);
cudaFree(d_invalidCount);
cudaFree(d_invalidIndexes);

// New dw_imageprocessing code
 
// New dw_imageprocessing APIs requires a GPU dwFeatureArray as the output of Harris corner detector.
// It also requires a GPU dwFeatureHistoryArray as output of LK tracker.
dwFeatureArray featureDetectedGPU;
dwFeatureArray_create(&featureDetectedGPU, maxFeatureCount, DW_MEMORY_TYPE_CUDA, context);
 
dwFeatureHistoryArray featureHistoryGPU;
dwFeatureHistoryArray_create(&featureHistoryGPU, maxFeatureCount, historyCapacity, DW_MEMORY_TYPE_CUDA, context);
 
// Create pyramid
dwPyramidImage pyramidCurrent, pyramidPrevious;
dwPyramid_create(&pyramidCurrent, numLevel, width, height, DW_TYPE_UINT8, context);
dwPyramid_create(&pyramidPrevious, numLevel, width, height, DW_TYPE_UINT8, context);
 
// create detector
dwFeature2DDetectorConfig detectorConfig = {};
dwFeature2DDetector_initDefaultParams(&detectorConfig);
detectorConfig.imageWidth                 = imageWidth;
detectorConfig.imageHeight                = imageHeight;
detectorConfig.maxFeatureCount            = maxFeatureCount;
dwFeature2DDetector_initialize(&detector, &detectorConfig, cudaStream, context);
 
// create tracker
dwFeature2DTrackerConfig trackerConfig = {};
dwFeature2DTracker_initDefaultParams(&trackerConfig);
trackerConfig.detectorType            = detectorConfig.type;
trackerConfig.maxFeatureCount         = maxFeatureCount;
trackerConfig.historyCapacity         = historyCapacity;
trackerConfig.imageWidth              = imageWidth;
trackerConfig.imageHeight             = imageHeight;
dwFeature2DTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
 
// Assuming sensor provides single channel image as input, all handles and buffers are initialized.
while (image = camera->getFrame())
{
    std::swap(pyramidCurrent, pyramidPrevious);
    dwImageFilter_computePyramid(&pyramidCurrent, &imageY, cudaStream, context);
 
    // Track
    dwFeatureArray featurePredicted;
    dwFeature2DTracker_trackFeatures(&featureHistoryGPU, &featurePredicted, nullptr,
                                     &featureDetectedGPU, nullptr,
                                     &pyramidPrevious, &pyramidCurrent, tracker);
 
    // Detect
    dwFeature2DDetector_detectFromPyramid(&featureDetectedGPU, &pyramidCurrent,
                                          &featurePredicted, nullptr, detector);
}
 
// release resources
dwFeature2DDetector_release(detector);
dwFeature2DTracker_release(tracker);
 
dwFeatureHistoryArray_destory(featureHistoryGPU);
dwFeatureArray_destroy(featureDetectedGPU);
dwPyramid_destroy(pyramidCurrent);
dwPyramid_destroy(pyramidPrevious);

Advanced detector and tracker

The new dwFeature2DDetectorHandle_t and dwFeature2DTrackerHandle_t provides advanced detecting/tracking options. To enable advanced mode, the application needs to set the detector type as DW_FEATURE2D_DETECTOR_TYPE_EX and the tracker algorithm as DW_FEATURE2D_TRACKER_ALGORITHM_EX:

dwFeature2DDetectorConfig detectorConfig = {};
dwFeature2DDetector_initDefaultParams(&detectorConfig);
detectorConfig.type = DW_FEATURE2D_DETECTOR_TYPE_EX;
// ... Other settings
 
 
dwFeature2DTrackerConfig trackerConfig = {};
dwFeature2DTracker_initDefaultParams(&trackerConfig);
trackerConfig.algorithm    = DW_FEATURE2D_TRACKER_ALGORITHM_EX;
trackerConfig.detectorType = detectorConfig.type;
// ... Other settings

Please refer to documentation for more details regarding available configuration.

Advanced mode also requires an additional device buffer nccScore. It can be left as nullptr, but a valid nccScore buffer is recommended for better tracking/detecting quality.

float32_t* nccScore = nullptr;
cudaMalloc(&nccScore, sizeof(float32_t) * maxFeatureCount);
 
// Initialization code
// ...
 
// Start detecting/tracking loop
while (image = camera->getFrame())
{
    std::swap(pyramidCurrent, pyramidPrevious);
    dwImageFilter_computePyramid(&pyramidCurrent, &imageY, cudaStream, context);
 
    // Track
    dwFeatureArray featurePredicted;
    dwFeature2DTracker_trackFeatures(&featureHistoryGPU, &featurePredicted, nccScore,
                                     &featureDetectedGPU, nullptr,
                                     &pyramidPrevious, &pyramidCurrent, tracker);
 
    // Detect
    dwFeature2DDetector_detectFromPyramid(&featureDetectedGPU, &pyramidCurrent,
                                          &featurePredicted, nccScore, detector);
}

The nccScore buffer is recommended only when both the detector and tracker are in advanced mode. It's valid to connect the DW_FEATURE2D_DETECTOR_TYPE_STD normal detector with the DW_FEATURE2D_TRACKER_ALGORITHM_EX advanced tracker, or to connect the DW_FEATURE2D_DETECTOR_TYPE_EX advanced detector with the DW_FEATURE2D_TRACKER_ALGORITHM_STD advanced tracker. The application only needs to set dwFeature2DTrackerConfig::detectorType with the correct value of its corresponding detector.

Working with SFM APIs

Below are the recommended steps for using imageprocessing APIs in SFM:

Set dwFeature2DTrackerConfig:.algorithm = DW_FEATURE2D_TRACKER_ALGORITHM_SFM to allow sparse tracking results output.
Perform tracking and reconstruction. DO NOT detect new features immediately after tracking.
Call dwFeature2DTracker_compact() explicitly after SFM reconstruction is done. Fill the d_newToOldMap device array by dwFeature2DTracker_getNewToOldMap().
Call dwReconstructor_compact* by d_newToOldMap received in step 3.
Detect new features when compact is done.

Pseudo code for a typical workflow:

// Track features
dwFeature2DTracker_trackFeatures();
 
// Reconstruction
// ...
 
// Compact data
dwFeature2DTracker_compact();
dwFeature2DTracker_getNewToOldMap();
 
dwReconstructor_compactFeatureHistory();
dwReconstructor_compactWorldPoints();
 
// Detect new features
dwFeature2DDetector_detectFromPyramid();

For the full implementation refer to Structure from Motion (SFM) Sample.

Port `dwScalingFeatureListHandle_t` and `dwScalingFeatureListPointers` to `dwTemplateArray`

Similar to dwFeatureHistoryArray, template tracker provides a dwTemplateArray to replace the old dwScalingFeatureListHandle_t and dwScalingFeatureListPointers.

The new dwTemplateArray struct provides pointers such as: *bboxes, *statuses, *ids, *ages, *scaleFactors and *templateCount like the old dwScalingFeatureListPointers did. It is also created and destroyed like old handle dwScalingFeatureListHandle_t. dwTemplateArray can be created on page-able/pinned CPU or CUDA memory.

Old name	New name
dwScalingFeatureListPointers::locations	dwTemplateArray::bboxes
dwScalingFeatureListPointers::sizes	dwTemplateArray::bboxes
dwScalingFeatureListPointers::statuses	dwTemplateArray::statuses
dwScalingFeatureListPointers::ids	dwTemplateArray::ids
dwScalingFeatureListPointers::ages	dwTemplateArray::ages
dwScalingFeatureListPointers::scaleFactors	dwTemplateArray::scaleFactors
dwScalingFeatureListPointers::bNewTemplate	—
dwScalingFeatureListPointers::featureCount	dwTemplateArray::templateCount
dwScalingFeatureList_initialize()	dwTemplateArray_create()
dwScalingFeatureList_release()	dwTemplateArray_destroy()
dwScalingFeatureList_reset()	dwTemplateArray_reset()
dwScalingFeatureList_getDataBasePointer()	dwTemplateArray::data, dwTemplateArray::bytes

For new dw_imageprocessing APIs, there's no need to call addEmptyFeature, selectValid(), compact() and applySizeFilter() explicitly. They are automatically called within the DriveWorks module.

Example: In the old dw_feature APIs, the application needed to create a dwScalingFeatureListHandle_t object and call dwScalingFeatureList_getDataPointers() to get the interpretable dwScalingFeatureListPointers.

// Old code
dwScalingFeatureList_initialize(&templateList, 0, maxTemplateCount, pxlType, context);
void* d_database8;
size_t databytes;
dwScalingFeatureListPointers templateDataGPU;
dwScalingFeatureList_getDataBasePointer(&d_database8, &databytes, templateList);
dwScalingFeatureList_getDataPointers(&templateDataGPU, d_database8, templateList));
 
// access the pointers of dwFeatureListPointers on GPU
templateDataGPU.locations[...]
templateDataGPU.sizes[...]

If the application needed a CPU copy of the templateList handle, it first had to allocate CPU memory and call dwScalingFeatureList_getDataPointers() to interpret it into dwScalingFeatureListPointers.

// Old code
uint8_t* h_database8 = new uint8_t[databytes];
dwScalingFeatureListPointers templateDataCPU;
dwScalingFeatureList_getDataPointers(&templateDataCPU, h_database8, templateList));
 
// access the pointers of dwScalingFeatureListPointers on CPU
templateDataCPU.locations[...]
templateDataCPU.sizes[...]

It also needed the raw data pointer to copy data between the CPU and GPU.

// GPU -> CPU copy
cudaMemcpy(h_database8, d_database8, databytes, cudaMemcpyDeviceToHost);
 
// CPU -> GPU copy
cudaMemcpy(d_database8, h_database8, databytes, cudaMemcpyHostToDevice);

In the new dw_imageprocessing APIs, the application only needs to create the dwTemplateArray object on the CPU and GPU. It provides both raw pointers and interpretable pointers. It also provides the dwTemplateArray_copyAsync() API to easily copy data between two template arrays.

// New code
dwTemplateArray_create(&templateCPU, maxFeatureCount, DW_MEMORY_TYPE_CPU, context);
dwTemplateArray_create(&templateGPU, maxFeatureCount, DW_MEMORY_TYPE_CUDA, context);
 
// access the pointers on GPU
templateGPU.statuses[...]
templateGPU.bboxes[...]
 
// access the pointers on CPU
templateCPU.statuses[...]
templateCPU.bboxes[...]
 
// GPU -> CPU copy
dwTemplateArray_copyAsync(&templateCPU, &templateGPU, 0);
 
// CPU -> GPU copy
dwTemplateArray_copyAsync(&templateGPU, &templateCPU, 0);

The application can also create dwTemplateArray on pinned memory, it only needs to change DW_MEMORY_TYPE_CPU to DW_MEMORY_TYPE_PINNED during creation.

Like dwScalingFeatureListHandle_t, there are also reset() and destroy() APIs. dwTemplateArray must be destroyed before the application exits.

// New code
dwTemplateArray_destroy(templateCPU);
dwTemplateArray_destroy(templateGPU);

Port `dwScalingFeatureTrackerHandle_t` to `dwTemplateTrackerHandle_t`

Old name	New name
dwScalingFeatureTracker_initialize()	dwTemplateTracker_initialize()
dwScalingFeatureTracker_initDefaultParams()	dwTemplateTracker_initDefaultParams()
dwScalingFeatureTracker_reset()	dwTemplateTracker_reset()
dwScalingFeatureTracker_release()	dwTemplateTracker_release()
dwScalingFeatureTracker_trackAsync()	dwTemplateTracker_trackImage()
—	dwTemplateTracker_trackPyramid()
—	dwFeature2DTracker_getNewToOldMap()

For new dw_imageprocessing APIs, there's no need to call dwScalingFeatureTracker_updateTemplateAsync() explicitly. It is automatically called within the DriveWorks module.

The new dwTemplateTrackerParameters parameter covers all settings from the old dwScalingFeatureTrackerConfig and dwScalingFeatureListParameters parameters.

Old dwScalingFeatureXXX settings	New dwTemplateTrackerParameters
—	dwTemplateTrackerParameters::type
dwScalingFeatureListParameters::maxFeatureCount	dwTemplateTrackerParameters::maxTemplateCount
dwScalingFeatureListParameters::imageWidth	dwTemplateTrackerParameters::imageWidth
dwScalingFeatureListParameters::imageHeight	dwTemplateTrackerParameters::imageHeight
dwScalingFeatureListParameters::pyramidLevelCount	dwTemplateTrackerParameters::maxPyramidLevel
dwScalingFeatureTrackerConfig::numIterCoarse	dwTemplateTrackerParameters::numIterationsCoarse
dwScalingFeatureTrackerConfig::numIterFine	dwTemplateTrackerParameters::numIterations
dwScalingFeatureTrackerConfig::thresholdUpdate	dwTemplateTrackerParameters::thresholdUpdate
dwScalingFeatureTrackerConfig::thresholdKill	dwTemplateTrackerParameters::thresholdKill
dwScalingFeatureTrackerConfig::thresholdStop	dwTemplateTrackerParameters::thresholdStop
dwScalingFeatureTrackerConfig::maxScaleChange	dwTemplateTrackerParameters::maxScaleChange
—	dwTemplateTrackerParameters::validWidth
—	dwTemplateTrackerParameters::validHeight

Here's a complete example showing the template tracker pipeline with the old dw_feature APIs and new dw_imageprocessing APIs.

old dw_feature

// Old dw_features code
 
// Create template list
dwScalingFeatureListHandle_t templateList;
dwScalingFeatureList_initialize(&templateList, cudaStream, maxTemplateCount, DW_TYPE_UINT8, context);
 
// create pyramid
dwPyramidHandle_t pyramidCurrent, pyramidPrevious;
dwPyramidConfig config{};
config.width = width;
config.height = height;
config.levelCount = numLevel;
config.dataType = DW_TYPE_UINT8;
dwPyramid_initialize(&pyramidCurrent, &config, cudaStream, context);
dwPyramid_initialize(&pyramidPrevious, &config, cudaStream, context);
 
// create tracker and detector
dwScalingFeatureTrackerConfig trackerConfig{};
dwScalingFeatureTracker_initDefaultParams(&trackerConfig);
dwScalingFeatureTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
 
// Old dw_feature APIs requires additional CUDA buffers for compact.
uint32_t *d_validCount, *d_validIndexes, *d_invalidCount, *d_invalidIndexes;
cudaMalloc(&d_validCount, sizeof(uint32_t));
cudaMalloc(&d_validIndexes, sizeof(uint32_t) * maxFeatureCount);
cudaMalloc(&d_invalidCount, sizeof(uint32_t));
cudaMalloc(&d_invalidIndexes, sizeof(uint32_t) * maxFeatureCount);
 
// add bboxes to track, assuming there're nBoxToTrack, bounding box info is stored in hBBoxes array
uint8_t* d_database8;
size_t databytes;
dwScalingFeatureList_getDataBasePointer(&d_database8, &databytes, templateList);
 
uint8_t* h_database8 = new uint8_t[databytes];
dwScalingFeatureListPointers templateDataCPU;
dwScalingFeatureList_getDataPointers(&templateDataCPU, h_database8, templateList));
 
// assuming there're nBoxToTrack stored in hBBoxes array
for (uint32_t i = 0; i < nBoxToTrack; i++)
{
    templateDataCPU.locations[i]    = {hBBoxes[i].x, hBBoxes[i].y};
    templateDataCPU.sizes[i]        = {hBBoxes[i].width, hBBoxes[i].height};
    templateDataCPU.statuses[i]     = DW_FEATURE_STATUS_DETECTED;
    templateDataCPU.ages[i]         = 1;
    templateDataCPU.bNewTemplate[i] = true;
}
*templateDataCPU.featureCount = nBoxToTrack;
cudaMemcpy(d_database8, h_database8, databytes, cudaMemcpyHostToDevice);
dwScalingFeatureList_addEmptyFeature(nBoxToTrack, dwScalingFeatureListHandle_t obj);
 
// Assuming sensor provides single channel image as input
while (image = camera->getFrame())
{
    std::swap(pyramidCurrent, pyramidPrevious);
    dwPyramid_build(&image, pyramidCurrent);
 
    dwImageCUDA *imageCurrent, *imagePrevious;
    dwPyramid_getLevelImageCUDA(&imageCurrent, 0, pyramidCurrent);
    dwPyramid_getLevelImageCUDA(&imagePrevious, 0, pyramidPrevious);
 
    // Update template
    dwScalingFeatureTracker_updateTemplateAsync(templateList, imagePrevious, tracker);
 
    // Track
    dwScalingFeatureTracker_trackAsync(templateList, imageCurrent, tracker);
 
    // Apply proximity filters to make features uniformly distributed
    dwScalingFeatureList_applySizeFilter(maxWidth, maxHeight, templateList);
 
    // Determine which features to throw away
    dwScalingFeatureList_selectValid(d_validCount, d_validIndexes,
                                     d_invalidCount, d_invalidIndexes, templateList);
 
    // Compact list
    dwScalingFeatureList_compact(templateList, d_validCount, d_validIndexes,
                                 d_invalidCount, d_invalidIndexes);
 
}
 
// release resources
delete[] h_database8;
dwScalingFeatureList_release(featureList);
dwScalingFeatureTracker_release(tracker);
dwPyramid_release(pyramidCurrent);
dwPyramid_release(pyramidPrevious);
 
cudaFree(d_validCount);
cudaFree(d_validIndexes);
cudaFree(d_invalidCount);
cudaFree(d_invalidIndexes);

new dw_imageprocessing

// New dw_imageprocessing code
 
// create template array
dwTemplateArray templateGPU, templateCPU;
dwTemplateArray_create(&templateGPU, maxTemplateCount, DW_MEMORY_TYPE_CUDA, context);
dwTemplateArray_create(&templateCPU, maxTemplateCount, DW_MEMORY_TYPE_CPU, context);
 
// Create pyramid
dwPyramidImage pyramidCurrent, pyramidPrevious;
dwPyramid_create(&pyramidCurrent, numLevel, width, height, DW_TYPE_UINT8, context);
dwPyramid_create(&pyramidPrevious, numLevel, width, height, DW_TYPE_UINT8, context);
 
// create template tracker
dwTemplateTrackerParameters trackerConfig = {};
dwTemplateTracker_initDefaultParams(&trackerConfig);
trackerConfig.maxTemplateCount  = maxTemplateCount;
trackerConfig.imageWidth        = width;
trackerConfig.imageHeight       = height;
trackerConfig.validWidth        = maxWidth;
trackerConfig.validHeight       = maxHeight;
dwTemplateTracker_initialize(&tracker, &trackerConfig, cudaStream, context);
 
 // assuming there're nBoxToTrack stored in hBBoxes array
for (uint32_t i = 0; i < nBoxToTrack; i++)
{
    templateCPU.bboxes[i]   = hBBoxes[i];
    templateCPU.statuses[i] = DW_FEATURE2D_STATUS_DETECTED;
    templateCPU.ages[i]     = 1;
}
*templateCPU.templateCount = nBoxToTrack;
dwTemplateArray_copyAsync(&templateGPU, &templateCPU, 0);
 
// Assuming sensor provides single channel image as input, all handles and buffers are initialized.
while (image = camera->getFrame())
{
    std::swap(pyramidCurrent, pyramidPrevious);
    dwImageFilter_computePyramid(&pyramidCurrent, &imageY, cudaStream, context);
 
    dwImageCUDA *imageCurrent, *imagePrevious;
    dwImage_getCUDA(&imageCurrent, pyramidCurrent.levelImages[0]);
    dwImage_getCUDA(&imagePrevous, pyramidPrevious.levelImages[0]);
 
    // Track
    dwTemplateTracker_trackImage(&templateGPU, imageCurrent, imagePrevious, tracker);
}
 
// release resources
dwTemplateArray_destroy(templateGPU);
dwTemplateArray_destroy(templateCPU);
dwTemplateTracker_release(tracker);
 
dwPyramid_destroy(pyramidCurrent);
dwPyramid_destroy(pyramidPrevious);

Lane Graph

The APIs previously contained in dw/world/LaneGraphProducer.h are no longer available, while the data structures are now part of LaneGraph.h.

Lidar Accumulator

The following API has been removed:

dwStatus dwLidarAccumulator_hasFullSweepAvailable(bool* available, dwLidarAccumulatorHandle_t obj);

dwLidarAccumulator_addPacket() will return DW_SUCCESS if the full sweep is ready.

For more information see Lidar Accumulator (replaced with dwPointCloudAccumulator).

Maps

The following APIs have been renamed:

Old name	New name
dwMaps_transformLaneDividersToLocalLinesPerLocalLayout()	dwMaps_transformLaneDividersToLocalLines()
dwMaps_transformRoadFeaturesToLocalLinesPerLocalLayout()	dwMaps_transformRoadFeaturesToLocalLines()
dwMaps_computeBearing()	dwMaps_computeBearingFromGeoPoints()
dwMaps_computeBearingFromRotMatrix()	dwMaps_computeBearingFromRotation()
dwMaps_computeLocalToENUMatrix()	dwMaps_computeRotationFromBearing()
dwMapTracker_updateCurrentPose()	dwMapTracker_updateWithGlobalPose()

The following API has been removed:

dwStatus dwMaps_transformLaneDividersToLocalLines(dwMapsLaneDividerLineBuffer* lineSegments,
                                                 const dwMapsLaneDividerBuffer* laneDividers,
                                                 const dwMapsGeoPoint* localOrigin,
                                                 const dwMatrix3d* localToENURotation33,
                                                 const dwMapsLocalBounds* bounds,
                                                 const dwVector3f* directionFilterVector,
                                                 float32_t directionFilterAngleRadian,
                                                 bool ignoreLaneDirection);

dwMaps_transformLaneDividersToLocalLines() should be used instead.

Object Detection

The dwObjectDetector API has been simplified. The functions

dwObjectDetector_inferDeviceAsync()
dwObjectDetector_interpretHost()
dwObjectDetector_getClusteredObjects()
dwObjectDetector_getRawDetections()
dwObjectDetector_bindInput()
dwObjectDetector_bindOutput()
dwObjectDetector_processDeviceAsync()
dwObjectDetector_processHost()

have been removed in favor of dwObjectDetector_detectObjects().

Objects

The following data structures have been renamed:

Old name	New name
dwObjectData	dwObjectGeneric
dwObjectDataCamera	dwObjectCamera
dwObjectDataRadar	dwObjectRadar
dwObjectDataLidar	dwObjectLidar
dwObjectDataFused	dwObjectFused

dwObjectHandle_t has been removed and replaced with plain structs. In place of dwObjectHandleList, use dwObjectArray. Becase object handles are no longer part of the code base, the following APIs have been removed:

Removed APIs
dwStatus dwObject_createCamera(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataCamera* otherData)
dwStatus dwObject_createRadar(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataRadar* otherData)
dwStatus dwObject_createLidar(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataLidar* otherData)
dwStatus dwObject_createFused(dwObjectHandle_t* handle, const dwObjectData* data, const dwObjectDataFused* otherData)
dwStatus dwObject_destroy(dwObjectHandle_t handle)
dwStatus dwObject_reset(dwObjectHandle_t handle)
dwStatus dwObject_getData(dwObjectData* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataCamera(dwObjectDataCamera* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataRadar(dwObjectDataRadar* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataLidar(dwObjectDataLidar* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_getDataFused(dwObjectDataFused* info, uint32_t index, dwConstObjectHandle_t handle)
dwStatus dwObject_setData(const dwObjectData* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataCamera(const dwObjectDataCamera* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataRadar(const dwObjectDataRadar* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataLidar(const dwObjectDataLidar* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_setDataFused(const dwObjectDataFused* info, uint32_t index, dwObjectHandle_t handle)
dwStatus dwObject_addDataCamera(const dwObjectData* info, const dwObjectDataCamera* otherInfo, dwObjectHandle_t handle)
dwStatus dwObject_addDataRadar(const dwObjectData* info, const dwObjectDataRadar* otherInfo, dwObjectHandle_t handle)
dwStatus dwObject_addDataLidar(const dwObjectData* info, const dwObjectDataLidar* otherInfo, dwObjectHandle_t handle)
dwStatus dwObject_addDataFused(const dwObjectData* info, const dwObjectDataFused* otherInfo, dwObjectHandle_t handle)

Because dwObjectHandle_t is no longer part of the code base, calling create to create an object is no longer necessary. For example, the old way of creating an object required calling:

dwObjectHandle_t objHandle = DW_NULL_HANDLE;
dwObjectData data{};
dwObjectDataCamera cameraData{};
dwObject_createCamera(&objHandle, &data, &cameraData);

The new way to do this is written as:

dwObjectCamera obj{};

Object history is no longer stored as part of the object, but is now a separate history array: dwObjectHistoryArray. The old way to get a historical object from an object handle was:

uint32_t historicalIndex = 10; // Arbitrary historical index, 0 being the most recent

dwObject_getData(&data, historicalIndex, objHandle);

The new way to do this is written as:

uint32_t historicalIndex = 10;
uint32_t objectIndex = 0; // This would be the index of the specific object in the history array
dwObjectArray objArray{};
...
dwObjectCamera* objects = static_cast<dwObjectCamera*>(objArray.objects);
objects[historicalIndex * objArray.maxObjectCount + objectIndex];

In order to add data to the history, you no longer call an API but use the indices into the objects array to insert into the history table.

Object arrays have also been introduced as dwObjectArray. These work very similarly to dwObjectHistoryArray except that it is a 1D contiguous array of objects.

Point Cloud Processing

dwPointCloudMemoryType has been replaced by the following data structure:

dwMemoryType

dwPointCloudRangeImage structure has been removed. Range image creator now accepts dwImageHandle_t instead.

For more information see Point Cloud Processing.

Rig Configuration

The module dwRigConfiguration has been renamed dwRig.

Header file has been renamed from RigConfiguration.h to Rig.h.
The object handles have been renamed to dwRigHandle_t and dwConstRigHandle_t, respectively.
The prefixes for all functions declared in Rig.h have been updated from dwRigConfiguration to dwRig.
dwRig_initializeCalibratedCamera is now dwCameraModel_initialize in calibration/cameramodel/CameraModel.h.

SFM

Feature status has been changed from dwFeatureStatus to dwFeature2DStatus.
Prototype for dwReconstructor_compactWorldPoints and dwReconstructor_compactFeatureHistory has changed. The input has changed from const uint32_t* d_validIndexCount, const uint32_t* d_validIndexes, const uint32_t* d_invalidIndexCount, const uint32_t* d_invalidIndexes to const uint32_t* d_validIndexCount, const uint32_t* d_newToOldMap, while d_newToOldMap is obtained with dwFeature2DTracker_getNewToOldMap.
Please refer to the "Working with SFM APIs" section in Image Processing for how to work with new dw_imageprocessing APIs.

For more information see Structure from Motion (SFM) and Features.

VehicleIO

This release adds support for multiple simultanous VehicleIO backends, and the following changes are required:

dwVehicleIO_addCANSensor() has to be called before send/consume operations.
VehicleIO can automatically detect to which backend to send commands to, therefore dwVehicleIO_sendVehicleCommand() can be just be replaced with dwVehicleIO_sendCommand().
The same is not true when receiving messages and dwVehicleIO_consume() needs be replaced with dwVehicleIO_consumeCANFrame() with sensorId equal to the sensor added with dwVehicleIO_addCANSensor().

For more information see VehicleIO.

Wait Conditions

The previous API header file waitcondition/Classifier.h has been split into waitcondition/camera/TrafficLightSignClassifier.h and waitcondition/camera/TrafficLightSignClassifier_processPipeline.h.

dwObjectClassifierHandle_t has been renamed to dwTrafficLightSignClassifierHandle_t.