NVIDIA Isaac Sim ? NVIDIA TAO? ??? AI ?? ?? ??, ????

Reading Time: 7 minutes

??? ??? ??? ??, ???? ?? ??? ???? ? ??? ??? ???? ?? ??? ??? ?? ??? ? ?? ?????. ?? AI? ???? ?? ??? ? ? ???? ??? ??? ???? ???? ??? ?? ? ?? ?? ? ???? ??? ?? ? ????.

?? ?? ???? ???? ? ??? ?? ??? ??? ? ???? ????? ??? ????. ? ??? ??? ?? ??? ????, ?? ?? ?? ?? ???? ???? ???, ??? ??? ?? ?? ???? ?? “??”?? ??? ??? ???.

???? ??? ?????. ?? ??? ? ???? ????? ?????? ???? AI ??? ?? ???? ??????? ???? ????. ?? ???? ??? ?? ??? ???? ???? ??????? ??? ? ?? ?? ??? AI ??? ???? ???. ?? ????? ?? ???? ??????? ???? ? ?? ?? ??? ?????.

? ?????? NVIDIA Isaac ??? ? TAO ?????? ?? ??? ??? ?? ?? ??? ?? ??? ? ?? ??? ????? ???. ?? ??? ??? ?? ???? ???? ?? ???? ????? ??????? NVIDIA Isaac Sim? ?????. NVIDIA TAO Toolkit? ???? ????? ???? ?? ?? ???? ???? ?? ????? ??? ?? ???? Transfer Learning ??? ??? ?? ?? AI ?? ?? ??????. ??? NVIDIA Isaac ROS? ???? ???? ??? ??? ???? ?? ??? ?????.

Diagram shows an overview of workflow with synthetic data generation using NVIDIA Isaac Sim and training a pretrained model with TAO in a simulated environment, evaluate the results and collect more data if necessary. The model is then pruned and retrained again in the simulated environment. The model then is moved to the next phase where it is fine-tuned on real-world data and eventually deployed.? — *?? 1. ?? ?? ??? ???? ?? NVIDIA Isaac Sim? ???? ?? ????? TAO Toolkit ??? ?????? ?????? ??.*

?? ??

???? ?? ???? ? ??? ?? ??? ?? ???? ??? ???.

NVIDIA GPU ???? ??: >470
NVIDIA Docker: 2.5.0-1
???? ?? ?????? NVIDIA GPU:
- NVIDIA A100
- NVIDIA V100
- NVIDIA T4
- NVIDIA RTX 30×0 (NVIDIA Isaac Sim? NVIDIA RTX 20 ???? ??)
- NVIDIA Jetson Xavier ?? Jetson Xavier NX
NVIDIA TAO Toolkit: 4.22. ??? ??? TAO Toolkit ?? ?? ???? ?????
NVIDIA Isaac Sim ? Isaac ROS

NVIDIA Isaac Sim? ?? ?? ??? ??

? ????? NVIDIA Isaac Sim?? ?? ???? ???? ??? ???? ?????. ?? ???? ??? ????? ?? ????? ???? ??? ??? ?????. ?? ???? ?? ???? ????? ???? ??? ?? ?? ??? ??? ??? ???? ? ??? ? ? ????.

NVIDIA Isaac Sim? ?? ???? ???? ??? ? ?? ??? ?????.

Replicator composer
Python scripts
GUI

? ????? Python ????? ???? ??? ????? ???? ????? ????. ??? ????? ?? ? ??? ??? ??? ???, ?????? ??? ??, ??? ?? ? ??? ? ????? ???? ??? ??? ???? ????? ??? ???.

?? ??? ??? ?????? ??? ????? ?? ??? ????? ???? ????? ??? ????, ?? ???? ? ? ?? ??? ??? ????? ???? ??? ??? ?????.

? ??, ??? ?? ???? ? ?? ??? ???? ???? ????? ? ????. ?? ???? ?? ??? ??? ??? ??? ???? ??? ?????. NVIDIA Isaac Sim? ?? ??? ?????, ???? YCB ?????? ???? ???? ?????.

Picture of cans, and bottles randomly placed on the floor. Another picture of a box placed underneath the cart along with some additional boxes placed in a warehouse type environment. — *?? 2. ??? ?? ?? ??? ?? ????? ???*

NVIDIA Isaac Sim? ???? ?? Isaac Sim ? ???? python.sh ????? ??? ???? ?? ??? ?????. ? ??? ??? ??? ?? ????? ???? ? ?????.

??? ??? ?? ???? ?????.

??? ???? ?? ??? ???? ?????.

def add_camera_to_viewport(self):
  # Add a camera to the scene and attach it to the viewport
  self.camera_rig = UsdGeom.Xformable(create_prim("/Root/CameraRig", "Xform"))
  self.camera = create_prim("/Root/CameraRig/Camera", "Camera")

Semantic ID? ??? ??

def add_floor_semantics(self):
  # Get the floor from the stage and update its semantics
  stage = kit.context.get_stage()
  floor_prim = stage.GetPrimAtPath("/Root/Towel_Room01_floor_bottom_218")
  add_update_semantics(floor_prim, "floor")

?? ??? ????? ???? ??? ??:

def load_single_asset(self, object_transform_path, object_path, usd_object):
  # Random x, y points for the position of the USD object 
  translate_x , translate_y = 150 * random.random(), 150 * random.random()
  # Load the USD Object
  try:
      asset = create_prim(object_transform_path, "Xform",
               position=np.array([150 + translate_x, 175 + translate_y, -55]), 
               orientation=euler_angles_to_quat(np.array([0, 0.0, 0]),
               usd_path=object_path)
	# Set the object with correct physics
      utils.setRigidBody(asset, "convexHull", False)

??? ???? ?? ??? ???:

def create_camera_randomization(self):
  #  A range of values to move and rotate the camera       
  camera_tranlsate_min_range, camera_translate_max_range = (100, 100, -58),
                                                            (220, 220, -52)      
  camera_rotate_min_range, camera_rotate_max_range = (80, 0, 0), (85, 0 ,360)

  # Create a Transformation DR Component for the Camera
  self.camera_transform = self.dr.commands.CreateTransformComponentCommand(
                              prim_paths=[self.camera.GetPath()], 
                              translate_min_range=camera_tranlsate_min_range,
                              translate_max_range=camera_translate_max_range,
                              rotate_min_range=camera_rotate_min_range,
                              rotate_max_range=camera_rotate_max_range,
                              duration=0,5).do()

?????? ??? ??? ??? ?? ??? ???? ?????. ??? ?? ?? ?? ???? ????? ?? ID? ??? ???? ???. ?? ????? ?? ??? sim2real ??? ???? ?? ??? ????? ???????.

NVIDIA Isaac Sim ??? ??? ???? ??? ?? ??? ????? ??????. ? ?? ???? ??? ??? ????? ???? ??? ????, ??? ????? ??????, ??? ??? ???? ?? ?? ??? ???????. NVIDIA? ?????? ?? ?? ???? ???? ? 3? ?? ???? ??????.

TAO Toolkit?? ????, ?? ? ???

? ????? TAO Toolkit? ???? ?? ??? ??? ?? ???? ?? ?????. ? ??? ?? NGC?? ???? UNET ??? ??? ????? ????.

!ngc registry model list nvidia/tao/pretrained_semantic_segmentation:*

??? ?? ???, ?? ??(TAO ??) ? ?? ????? ?????.

%set_env KEY=tlt_encode
%set_env GPU_INDEX=0
%set_env USER_EXPERIMENT_DIR=/workspace/experiments
%set_env DATA_DOWNLOAD_DIR=/workspace/freespace_data
%set_env SPECS_DIR=/workspace/specs

The next step is to pick the model.

?? ????? ??? ?????.

???? ?? ????? ?? ??

?? ????? AI ? ? ?? ??? ?? ?????? ?? ?????? ???? ???? ?? ??? ?????. ???? ????? ???? ?? ?? ???? ??? Transfer Learning? ???? ?? ????? ??? ?? ??? ?? ??? ? ????.

?? ????? ?? ???? ??, ???, ?? ??? ?? ?? ??? ???? ???? ????.

?? ResNet10 ? ResNet18 ??? ?? U-Net ??? ??????. ???? ?? ??? ?? ??? ? ?? ?? ???? ??, ?? ????? ?? ???? ??? ?? ???????. ????? ????? ??? ??? ?? ??? ???? ????? ???? ?? ???????.

BackBone	Pruned	Dataset Size	Image Size	Training Evaluations
		Train	Val		F1 Score	mIoU (%)	Epochs
RN10	NO	25K	4.5K	512×512	89.2	80.1	50
RN18	NO	25K	4.5K	512×512	91.1	83.0	50

? 1. TAO? NGC ????? ?? ??? ?? ?? ?? ????? ??? ?? ??.

????(FPS)? ???? ??? ???? ?? ??? ??? ??? ??? ??????. ?? ??? ?? ??? ????(UNET) ??? ????.

Images of a carpeted floor with objects scattered around. The blue areas represent blocked space and the red areas are free space. — *?? 3. ResNet18 ??? ??. (??) ????? ???, (???) ?? ???.*

?? ??? ??? ? ?, ?? ??? ? ??? ?? ??? ????? ?? ??????. ??? NGC ?????? ?? ??? PeopleSemSeg ??? ??????. ? ??? ??? ??, ?? ?? ? ?? ??(FOV)? ???? ??? ?????? ?? “??” ???? ?? 500? ?? ??? ?? ?? ?????????. ? ??? ??? ?? ??? ? ?? ?? ???? ??? ?? ????.

??? ?????? ? ??? ????? ? ?? IOU? 10% ?? ????? ? ?? ??? ???? ??? ? ?? ? ?? ??? ???? ?????.

BackBone	Pruned	Dataset Size	Image Size	Training Evaluations
		Train	Val		F1 Score	mIoU (%)	Epochs
PeopleSemSegNet	NO	25K	4.5K	512×512	98.1	96.4	50
PeopleSemSegNet	NO	25K	4.5K	960×544	99.0	98.1	50

? 2. ???? ??? PeopleSemSegNet ??? ??? ??

Images shows a carpeted floor with objects scattered around. — *?? 4. ?? ???(??)? ?? ???(???)? ??? PeopleSemSeg TAO ???? Transfer Learning? ?? ?? ??.*

?? 4? ?? ???? PeopleSemSeg ??? ?? ???? ?, ?? ????? ????? ???? ?? ???? ?? ?? ??? ?????. ?, ??? NVIDIA Isaac Sim ???? ?? ????? ??? ??? ?????.

?? ??? ? ??? ??? ? ?? ?? ????? ??? ?? ? ??? ?? ??????? ?? ??? ??? ???? ?? ????? ????. ???? TAO? ?? ?? ??? ?????.

!tao unet train --gpus=1 --gpu_index=$GPU_INDEX \
              -e $SPECS_DIR/spec_vanilla_unet.txt \
              -r $USER_EXPERIMENT_DIR/semseg_experiment_unpruned \
              -m $USER_EXPERIMENT_DIR/peoplesemsegnet.tlt  \
              -n model_freespace \
              -k $KEY

??? ????? ? ?? ???? ?? ?? ??? ?????.

!tao unet evaluate --gpu_index=$GPU_INDEX -e$SPECS_DIR/spec_vanilla_unet.txt \
-m $USER_EXPERIMENT_DIR/semseg_experiment_unpruned/weights/model_freespace.tlt \
    -o $USER_EXPERIMENT_DIR/semseg_experiment_unpruned/ \
    -k $KEY

NVIDIA Isaac Sim ???? ?? ?? ??? Sim2Sim ?? ??? ?????? ?? ??? ?????.

?? ????? ? ??? ????? ?? GPU?? ????? ??? ??????. ??? ??? ???? ???? ?? ? ??? ????.

??(Pruning): TAO Toolkit? ?? ??? ?? ?? ???? ??? ???? ???? ??? ??? ????? ?????. ?? ?? ??? ???? ????? ??? ?? ?????? ???.
???? ? ???: TAO Toolkit? ? ?? ??? ??? ? ??? ???? ?? ??? ? ?? ? ????. ??? ?? FP32?? INT8? ???? ???? ???? ????? ?? ??? ?? ? ????.

?? ??? ??(prune)???.

!tao unet prune \
    -e $SPECS_DIR/spec_vanilla_unet.txt \
    -m $USER_EXPERIMENT_DIR/semseg_experiment_unpruned/weights/model_freespace.tlt \
    -o $USER_EXPERIMENT_DIR/unet_experiment_pruned/model_unet_pruned.tlt \
     -eq union \
     -pth 0.1 \
     -k $KEY

??? ?? ?????? ??(prune)???.

!tao unet train --gpus=1 --gpu_index=$GPU_INDEX \
              -e $SPECS_DIR/spec_vanilla_unet_retrain.txt \
              -r $USER_EXPERIMENT_DIR/unet_experiment_retrain \
              -m $USER_EXPERIMENT_DIR/unet_experiment_pruned/model_unet_pruned.tlt \
              -n model_unet_retrained \
              -k $KEY

???(pruned) ??? Sim2Sim ?? ??? ?????? ?? ??? ???? ?? ???? ?? ?????.

!tao unet train --gpus=1 --gpu_index=$GPU_INDEX \
              -e $SPECS_DIR/spec_vanilla_unet_domain_adpt.txt \
              -r $USER_EXPERIMENT_DIR/semseg_experiment_domain_adpt \
              -m $USER_EXPERIMENT_DIR/semseg_experiment_retrain/model_unet_pruned.tlt\
              -n model_domain_adapt \
              -k $KEY

??

? 1?? ???? ?? ??? ??? ?? ?? ?? ??? ?????. ?? ???? ??? ?? ?? ? ???? ??? ?? ??? ??? ??? 1/17? ???? NVIDIA Jetson Xavier NX?? ?? ? ?? ???? 5? ? ?? ?? ??? ??????.

Model	Dataset	Training Evaluations	Inference Performance
Pruned	Fine-Tune on Real World Data	Training Set	Validation Set	F1 Score (%)	mIoU (%)	Precision	FPS
NO	NO	Sim	Sim	0.990	0.981	FP16	3.9
YES	NO	Sim	Sim	0.991	0.982	FP16	15.29
YES	NO	Sim	Real	0.680	0.515	FP16	15.29
YES	YES	Real	Real	0.979	0.960	FP16	15.29
YES	YES	Real	Real	0.974	0.959	INT8	20.25

? 3. Sim2Sim ? Sim2Real? ??

????? ???? ?? ???? ?????? 25K ???? ???? ??, ?? ??? ?? ?? ???? ???? ???? 44?? ????? ?????. ?? ???? ?? ?????? 56?? ????? ?????. ?? ???? ??, ? ?? ?? ?????? ??? ?????? ??????. ??? ?? ??? ??? 960×544???. ?? ??? NVIDIA TensorRT trtexec ??? ???? ?????.

Four images showing the results obtained from the model chosen for deployment. The images show clear delineation between the floor, walls, and obstacles directly in front of the robot. — *?? 5. ?? ???? ?? ??? ? ???? ?? ?? ??? ??*

NVIDIA Isaac ROS? ??? ??

? ????? ???? ? ???? ??? ??? Jetson Xavier NX? ???? iRobot? Create 3 ??? NVIDIA Isaac ROS? ???? ?? ??? ???? ??? ?????. Create 3 ??? NVIDIA Isaac ROS ??? ?? ??? ?? ROS2?? ?????.

? ????? ?? ?? ??? ???? ?? /isaac_ros_image_segmentation/isaac_ros_unet GitHub ?????? ?????.

Picture of a carpeted floor with objects on it and the same picture using free space identification results. — *?? 6. ROS2? rqt_image_viewer? ??? ??? ? ?? ???. (??) Create 3 ??? USB ???? ?????. (???) isaac-ros-image-segmentation ??? ?????.*

?? ?? ?? ??? ????? /NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation GitHub ??????? ?? ??? ?????.

Docker ????? ?? ?? ???:

$isaac_ros_common/scripts/run_dev.sh your_ws

?? ??? ???? ??:

isaac_ros_dnn_encoders
isaac_ros_nvengine_interfaces
?? ???(? ??? ??? ? ??)
isaac_ros_tensor_rt
isaac_ros_triton

?? ?? ?? ? ??:

$cd /workspaces/isaac_ros-dev
$colcon build && . install/setup.bash

?? ????? ????? ?? ?? ??(.etlt) ??? ????:

$scp <your_machine_ip>:<etlt_model_file_path> <ros2_ws_path>

???? TLT ??(.etlt)? ??? TensorRT ?? ???? ?????. INT8 ??? ?? ?? ??? ?????.

tao converter -k tlt_encode  \
               -e  trt.fp16.freespace.engine \
               -p input_1,1x3x544x960,1x3x544x960,1x3x544x960 \
               unet_freespace.etlt

Isaac ROS image segmentation? ??? ????:

TensorRT ?? ?? ??? ??? ????? ?????.
config.pbtxt? ?????.
isaac_ros_unet ?? ??? ?? ?? ?? ? ??? ???????.
?? ??? ?? ???? ?????.

$ colcon build --packages-up-to isaac_ros_unet && . install/setup.bash
$ ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py

??

? ?????? NVIDIA Isaac Sim? ?? ??? ???? ??? TAO Toolkit? ??? ?? ??, NVIDIA Isaac ROS? ??? ?? ?? ?? ???? ?? ? ?? ?????? ???????.

NVIDIA Isaac Sim? TAO Toolkit? ?? AI ????? ???? ????? ?????, ?? ???? ???? AI ?? ??? ???? AI ?? ?? ??????? ???? ??? ? ????.

/NVIDIA-AI-IOT/robot_freespace_seg_Isaac_TAO GitHub ????? ???? ? ??? ?????.

NVIDIA Isaac Sim ? NVIDIA TAO? ??? AI ?? ?? ??, ????

?? ??

NVIDIA Isaac Sim? ?? ?? ??? ??

TAO Toolkit?? ????, ?? ? ???

???? ?? ????? ?? ??

??

NVIDIA Isaac ROS? ??? ??

??

Tags

??? ??

??

?? ??? ?? ??

Related posts

?????? NVIDIA TensorRT 10.0? ???, ??, AI ?? ??

NVIDIA TensorRT Model Optimizer? ??? AI ?? ?? ???

NVIDIA TensorRT-LLM ? NVIDIA Triton Inference Server? Meta Llama 3 ?? ??

8-bit ??? ???? ???? ???? ??? 2? ? ??? ????? NVIDIA TensorRT

NVIDIA AI ?? ????? ???? Diffusion XL? ?? ??? ????