• <xmp id="om0om">
    •
    • <table id="om0om"><noscript id="om0om"></noscript></table>
  • Related Resources

    Pony AV ?? ??? ?? ????? ???

    2022? 5? 26?
    By
    +2
    Discuss (0)
    Reading Time: 10 minutes

    ??? ??? ??? ???? ??? ???? ???? ??? ???? ??? ?????.??? ??? ?? ???? ????, ?? ???? ??? ?? ??? ??? ??? ? ??? ???? ??? ??? ??? ?????.? ??? ???? ??? ??? ?? ???? ?? ????? ? ????? ??? ? ??????.

    ??? ?? ??? ??? ???? ????? ?? ?????? ???? ????? ?? ?????.? ?????? Pony.ai? ?? ? ?? ??? ?? ?????? ??? ?????.

    Pony.ai? ?? ???? ?? ?? ???, LiDAR ? ???? ?????. ???? ??? ??? ?????, ???? ??? ???? ?????, ?? ???? ??? ???, ?? ? ???? ????? ??? ?????.

    ? ??? ?? ????? ?? ??? ?? ? ??? ??? ????? ?? ?? ?? ???? ??? ? ????.

    Diagram shows a vehicle emitting camera, radar, and lidar-sensing modalities leading to the data processing pipeline for perception, prediction, planning, and control.
    ?? 1. Pony? ???? ?? ??? ?? ?????

    ??? ?? ?????? ?? ??? ????? ????? ???. ??? ??? ??? ?????. ?? ??? ?? ???? ? ?? ???? ??? ??? ???.

    ??, ??? ???? ?? ? ??? ???? ???? ?? ???? ???? ?????. ?? ? ??? ????? ?? ???? ?? ??? ?? ???? ??? ???? ??? ?? ??? ????.

    ??, ???? ??? ???? ?? HW/SW ???? ?????? ???. ???? ?? ? ? ? ?? ??? ??? ???? ??? ????? ???? ?? ?? ????. ?? ?? ???? ?? ?????.

    ?? ???? ??? ??

    Easing the b?? ?? ?????? ?? ??? ????? ??, GPU ???? ? GPU ???? ??? ?? ?? ??? ??????.

    ???? GPU?

    Pony.ai ?? ?? NVIDIA? ?? ?? ??? ??? ?? ??? ??????. ??? ?? ? ???? ?? ??? ???? USB ? ??? ?? ??? ????? CPU? USB/??? ??????? ???? ?? ??? ????.

    Block diagram showing the pipeline from the camera, to the CPU, to the GPU.
    ?? 2. ????? CPU, GPU??? ?????? ???? ?? ?????

    ???/USB? ?? ??? ???? ???? ????? ? ????? CPU ???? ?????.

    ? ??? ? ????? ???? ???? ??? ?????. USB? ??? ??? ?????(GigE-camera)? CPU? ??????. ?? ? ?? ???? ???? ????? CPU? ??? ???? ?? I/O ??? ??? ? ?? ?????. ??? ?? ????? ????? ? ??? ????? ??????.

    ??? ???? LiDAR? FPGA ?? ?? ?????? ???? ??? ??????.

    Block diagram showing the same camera/CPU/GPU setup, but adding an FPGA between the camera and CPU.
    ?? 3. ?? ???????? FPGA(???? ???? ??)

    ?? ?????? FPGA? ???? CPU I/O ??? ??????? PCIe? ?? DMA? ???? ?? ????? ?? ??? ? ?? ???? ??? ? ????.

    FPGA? ??? ??? ? ??? ??? ???? ? ?? ?? ??? ?????. ?? ??? ??? ??? ??? ???? PCIe ??? ?? FPGA?? ? ???? ???? ???? ?? DMA ??? ??????. DMA ??? FPGA?? ?? ???? CPU? ?????. CPU? I/O ???? ??? ?? ??? ??? ?? ????? ??? ?? ?? ??? ?? ??? ?????.

    ??? ???? GPU?? ???? ?? ??? ??? ?? ????? FPGA?? CPU? DMA? ?? ??? ??? GPU ???? ????? ???. ??? ???? CUDA HostToDevice ??? ??? ????, FHD ??? ???? ??? ??? ?? 1.5ms? ?????.

    ??? ??? ? ????? ?? ???? ????. ?????, ??? ???? CPU? ?? ????? ?? GPU ???? ?? ????? ???.

    Same block diagram showing the camera/FPGA/CPU/GPU pipeline, but removing the CPU by adding a PCIe switch between the FPGA and GPU.
    ?? 4. ????? ???/FPGA/CPU/GPU ?????? ????? RDMA? ???? FPGA? GPU ?? ???? ?? ?????.

    ??? ??? ?? FPGA-GPU RDMA? ???? PCIe ???? ??????. ? ????? CPU-GPU ??? ??? ???????. ?? ??? NVIDIA GPU Direct RDMA ??? Xilinx? XDMA ????? ???? PCIe Gen3 8??? FPGA-GPU ??? ?? 6GB/s? ??????.

    NVIDIA GPU Direct RDMA? ???? ? ??? ??????.GPU Direct RDMA? ???? PCIe BAR(PCIe ?? ??? ?? ?? ???? ?? ?? ????)? ?? PCIe ??? ???? ? ?? CUDA ??? ??? ?? ??? ? ????.

    ?? ?? ?? ????? GPU ??? ??? ??? ?? ? ??? ??? ?? ?? API? ?????. ??? API? ?? ??? DMA ??? ? ???? ???? ??? ?? ??? GPU ???? ?? ???? ??? ?? ??? ???? ???.

    GPU Direct RDMA? CPU ? ??? ???? ????? ??? ???? ??? 8GB/s? PCIe Gen3 8??? ?? 6GB/s? ???? ?????.

    GPU ???? ??

    ??? ????? ???? ?? ?? ??? GPU? ??????. ???? GPU? ?? ? ?? ????? GPU ? ????? ?? ??? ??? ? ????. ???? ??? ?? CPU? ???? CPU ??? ???? ?? ???? ?????.

    A close-up comparison of how the PCIe switch better facilitates communication from the CPU to multiple GPUs.
    ?? 5.PCIe ???? ?? GPU-GPU ??

    ??? ??? ? ??? ?? ? ?? ?? ??? ???? PCIe ???? ??????.?? ? ?? ??? ?? ? PCIe ?? ??? ??? ? ???? ?? GPU?? ?? ? ?? ???? ?????.

    ?? ????? ??? ????

    ?? ???? CUDA ???? ???? ??? ?? ????? ?????? ?? ??? ??? ???????.

    ?? ?? FHD ??? ???? JPEG ???? ????? ?? NvJPEG ?????? RTX5000 GPU? ?? ?? CPU ????? ?? 4ms? ??? ???. NvJPEG? ??? ???? ?? ?? ??? ??? CPU? ?? ? ???? CPU ? GPU ???? ??? ? ????.

    Block diagram of CPU/PCIe switch/GPU with JPEG encoding added to the GPU for better resource management.
    ?? 6. GPU? NvJPEG ?????? ???? JPEG ???? ?? ??? ??? ???? ?? ?????.

    GPU??? JPEG ???? GPU ? CPU ???? ???? ?????. NvJPEG GPU ????? ??? CPU(??? ??? ? ???)?? ?? ??? ?????. CPU ??? 1?? NVIDIA RTX 5000? 4ms ?? ???? ??? FHD ???? JPEG? ??????. ?? ?? ?? ?? GPU ??? ?????.

    ??? CPU ? GPU(CUDA ??)? ??? ??? ? ???? ???? ??? ?? ? ??? ?? NVIDIA ??? ??? ??????. ? ??? GPU? ?? ???? ???? ?????. ?? GPU? ????? ?? ? ? ?? ??? ???? ? ???? ?? CUDA ???? ???? ????.

    ?? NVIDIA GPU? ?? ???? ??? ???? ???? ??? ?? ??? JPEG?? HEVC(H.265)? ??????????. ??? ??? ??? ?? ??? ?? CPU ? GPU ???? ?? ??????.

    CUDA ??? ??? ??? ?? GPU?? FHD ???? ??? ????? ?? ?? 3ms? ?????. ??? I-??? ?? ???? ????, ?? ??? ?? ??? ??? ?? ????? ?????.

    Same CPU/PCIe switch/GPU block diagram, with HEVC encoding added to the GPU, which avoids consuming CUDA cores or the CPU.
    ?? 7. CUDA ?? ?? CPU ??? ???? HEVC ???? ?? ??? ??? ???? ?? ?????.

    NVIDIA ??? ??? CUDA ?? ?? CPU? ???? ?? GPU ?? ?? ????? ???? ?????. NVENC? H264/H265? ?????. ??? FHD ???? HEVC? ????? ? ???? ??? ? 3ms??? GPU? CPU? ?? ??? ??? ? ????. I-??? ?? ??? ???? ? ???? ??? ??? ???? ??? ??? ????.

    GPU?? ??? ??

    ? ?? ??? ??? ??? ???? ????? ?????? ???? ??? ??????.

    ??? Google? protobuf? ???? ???? ?????. CameraFrame ???? ?? ?? ?????. ??? ?? ? ??? ???? ?? ?????. protobuf? ?? ??? ?? ????(??? ???)? ? ??? ???? ??? ??? ???? ???.

    Block diagram showing the camera module linking to the perception module through a CameraFrame message.
    ?? 8. CameraFrame ???? ??

    ?? ?? ??? ???? ??????. Protobuf? ???? ?? ???? ? ???? ??? ???.

    message CameraFrame {
      optional string device_name = 1;
      optional int32 width = 2;
      optional int32 height = 3;
      optional int32 pixel_format = 4;
      optional bytes data = 5;
    };

    ??? ???? DL ??? ????? CUDA H2D ??? ?????.

    ??? ??? ???? ?? ?? ?? ?? ?? ??? ??? ?? ????-??? ??? ?????. ? CameraFrame ???? ?? ??? ??? ??? ???? ???? ? ?? ??? ???.

    ?? ?????? ??? ??? ???? ??? ? ?? ?? ??? ???? GPU? ???? ?? CUDA HostToDevice ??? ??? ???? ????.

    Block diagram showing camera module communicating with the data recorder, perception, localization, and camera quality monitor modules via CameraFrame message.
    ?? 9. ??? ??? ?? ??? ??? CameraFrame ???? ???? ????-??? ??? ???? ?? ?????. ? ??? ??? CPU?? GPU?? ??? ??? ???? ???.

    ?? ??? ??? ???? ?? H2D ??? ?? ??, ? ???? ??? ?? ????! ?? ?? ????? GPU? ?? CPU?? ???? ?? ?? ???? ?????.

    message CameraFrame {
      optional string device_name = 1;
      optional int32 width = 2;
      optional int32 height = 3;
      optional int32 pixel_format = 4;
      optional bytes data = 5;
    };

    ?? ??? ???? ???? ?? ???? CUDA HostToDevice ??? ???? ???. ?? ?? ??? ?? ?????? CPU?? ? ????? ??? CPU-GPU ??? ??? ?????.

    Block diagram showing camera module communicating with the data recorder, perception, localization, and camera quality monitor modules through CameraFrame message with additional GPU memory support.
    ?? 10. GPU ??? ??? ?? ?? ????-??? ??? ??

    ??? GPU ???? ??? ??? ????? ?? Protobuf codegen ????? ??????. ?? ?? ????? GPU?? ???? ?? ?? ???? ?????. GPUData ??? GPU ???? ????.

    message CameraFrame {
      optional string device_name = 1;
      optional int32 width = 2;
      optional int32 height = 3;
      optional int32 pixel_format = 4;
      optional GpuData data = 5;
    };

    ??? ??? ???? GpuData ??? Protobuf? ???? API? ?? Protobuf ?? ???? ???? ? ??? ??????. GpuData? CPU ??? bytes ??? ?? ?? resize ??? ?????. ??? ??? ??? ????? GPU? ????.

    ??? ??? ???? ??? ?? ???? ?? GPU ??? ???? ??? ? ????. ??? ?? ?? ??????? ?? ?? ??? ??????.

    GPU ??? ?? ??

    GpuData ???? resize ??? ????, ?? CUDA cudaMalloc? ?????. GpuData proto ???? ???? cudaFree? ?????.

    ? ? API ??? GPU? ??? ?? ???? ?? ??? ??? ???? ????. ? ???? ?? 0.1ms? ??? ? ????.

    ? ??? ???? ???? ?? ?? ???? ???? ?? ????? ????? GPU ??? ????alloc/free ??? ????? ???.

    ? ??? ???? ?? ?? ?? ??? GPU ???? ??????. ????? ?????. ??? ??? ??? ??? ?? ??? ?? ?? ??? GPU ??? ??? ??? ???? ????. alloc? ??? ??? ???? ??? ?? ?????. free? ??? ??? ??? ?? ?????. GPU ???? ?? ?????? alloc/free ??? 0? ??????.

    Block diagram showing the flow of the fixed slot size GPU memory pool, with the first camera frame entering the last one destructing to preserve memory.
    ?? 11. ?? ?? ???? ???? GPU ??? ?
    camera_frame.mutable_gpu_data()->Resize(size);
ptr = pool->Alloc();
    camera_frame destructs
pool->Free(ptr);

    ?? ?? ???? ???? ????? ??? ?? ???? ? ?? ?? ??? ?? ???? ??? ? ?? ? ??? ????? ?? ??? ?? ??? ?? ??? ????? ???. ?? ??? ??? ?????.

    CUDA 11.2? ??? ??? ? ??? ??????. ????? cudaMemPool? ?????. ?? ?? ???? ??? cudaMalloc ? free? ?????. ?? ??? ???? ? ?? ?? ??? ??? ???. ?? ?? ?? ?? ??(??? ?? 2us)?? ???? ?? ??? ? ????.

    Block diagram showing dynamic size GPU memory pool, which uses CUDA 11.2 to handle any allocation size, compared with fixed slot size.
    ?? 12. ?? ?? ??? ???? GPU ??? ?
    camera_frame.mutable_gpu_data()->Resize(size);
pool->cudaMallocFromPoolAsync(&ptr, pool, ...);
    camera_frame destructs
pool->cudaFreeAsync(ptr);

    ? ????? resize  ??? ??? ?? ?????? ?? cudaMalloc ? free? ??????.

    YUV ? ???? ? ??? ??? ??

    ??? ???? ?? ? ??? ????? ????? ???? ?? ???? ?? ?? ???? ??? ??? ??????. ?? ??? ??? ?? ??? ????? ????.

    NVIDIA? ???? RGB ? ???? ??? ???? ???? ? ??????. ??? ???? ISP ??? YUV ? ??? ??? GPU?? YUV? RGB? ???? ?? 0.3ms? ?????. ?? ?? ?? ???? ? ??? ???? ????. RGB ?? ??? ???? ?? ?????.

    Block diagram of camera data to the GPU, with removal of the colorspace conversion module using the YUV format.
    ?? 13. YUV ??? ???? ? ?? ?? ??

    ??? ??? NVIDIA? RGB ??? ????? YUV ????? ??????????. ??? ??? ???? ??? ???? ??? YUV420 ?? ??? ????? ??????.

    YUV420 ?? ??? ?????? GPU ??? ???? ???? ??????. ?? ?? ??? ???? ?? ?? ?? ??? Y ??? ?? ? ?? ?? RGB? ?? GPU ??? ???? 3?? 2? ??????.

    GPU?? ??? ??? ??

    ??? ???? ??? ?? ??? ??? ???? ??? GPU?? ?????. ???? ??? ????? ????? ? ?????. ??? ??? ???? ??? ? ? ?? ???? ??????.

    • ??? ?? ????? ?? ??? ??? ???? ????, ??? ????? ???? ?? ?? ??(Array of Structure) ?? GPU ???? ?? ??(Structure of Array)? ???? GPU ??? ??? ??? ???? ?? ? ???? ??????.
    • CPU? GPU ?? ?? ??? ???? ?? ?? ?? ???? ?? ??? ?? ???? ?????.
    • NVIDIA CUB ?????? ?? ?????, ?? ??/?? ??? ????? ?????.
    Block diagram showing pipeline from lidar sensor, to downsample, position transform, filtering, and finally, point cloud processing on the GPU.
    ?? 14. ??? ???? GPU? ??? ???? ????? ?????? ???? ?? ?????.

    GPU? ??? ??? ?? ?????? ?? ??? ?????.

    • GPU ???? ?? ?? ??? ????.
    • CPU-GPU ??? ????? ??? ?? ???.
    • ??? ??? ?? 58% ?? NVIDIA CUB ????? ??.

    ??? ???? ?? ?? ????? ????? ?? ???? ?? 4ms ?? ? ?????.

    ?? ????

    ??? ?? ???? ?? ?? ???? ??? ??? ???? ??? ????? ? ? ????.

    Timeline of data processing on CUDA, GPU 0, and GPU 1, showing higher utilization for perception on GPU 0, and GPU 1 showing more downtime gaps.
    ?? 15. ?? ????? DL ????? ?? ????

    ?? ????? ??? ??? ??? ????? GPU? ????? ????? ?????. ? GPU? ?? 80%? ?? ?? ????? GPU0 ? GPU1 ????? ??? ????? ????? ????. GPU 0? ?? ?? ?? ?? ??? ?? ?????. GPU 1? ?? ?? ?? ??? ?? ??? ? ????.

    ???? GPU ??? ?? ???? ? ??? ????.

    ?? ??

    ?? ??? FPGA? ?? ???? ?? ?? ??? ?? ???? ????? ?? ??? ? ?????. ?? ??? ?? ??? ?? ? ???? ?? ??? SoC(System-on-a-Chip)? ???? ??? ? ?? ?? ??? ??? ?? ??? ????? ??? ???? ???? ????.

    ??? ??? ?? NVIDIA DRIVE Orin SoC? ?? ??? ???? ????? ??? ??????. ASIL ????? ?? ???? ????? ?? ?????. 

    FPGA?? NVIDIA DRIVE Orin?? ??????

    ?? ??? FPGA? ?? ???? ?? ?? ??? ?? ???? ????? ?? ??? ? ?????.

    ?? ??? ?? ??? ?? ? ???? ?? ??? SoC(System-on-a-Chip)? ???? ??? ? ?? ?? ??? ??? ?? ??? ????? ??? ???? ???? ????.

    ??? ??? ?? NVIDIA DRIVE Orin SoC? ?? ??? ???? ????? ??? ??????. ASIL ????? ?? ???? ????? ?? ?????. ???? ??? ??? ???? ???? ???? ??? ?? ??? ???? ??? ?? ???? ????? ??? ? ????.

    NVIDIA Orin? ???? ?? ?? ?? ??, ???, ?? ??, ??? ??? ???? ??? ?????. ? ???? ?? ???? ???? ???? ? BOM ??? ?? 70% ??? ??? ??????.

    Block diagram showing pipeline from camera, to DRIVE Orin SoC, to PCIe Switch, to CPU and GPU.
    ?? 16. NVIDIA DRIVE Orin SoC? ??? ?? ?????? ??

    Orin SoC? FPGA? ?? ?????? ???? ??? ??? 10? ??? ???? ???? ?????? ?? 70%? ?? ??? ?????.

    NVIDIA?? ??? ?? Orin-CPU-GPU ?? ?? ?? ?? ??? NvStream? ?? DMA ??? ?? PCIe ??? ??? ????.

    • ??? ???? DL ??? ?? NVIDIA Orin SoC? NvStream? ???? ?? ???? ?? GPU? ???? ?????.
    • GPU ??? ?? ?? NVIDIA Orin SoC? NvStream? ???? ???? ???? ?? ??? CPU? ?????.

    ?? 2/3 ??? ??? ??????

    Block diagram showing camera/Orin/GPU pipeline, using resource sharing to achieve L2/L3 driving capabilities.
    ?? 17. X86 CPU? ???? ??? L2/L3 ??? ??? ?????? ??? ?? ?????.

    Orin SoC? ?? 250 TOPS DL ??? ????? L4? L2/L3 ?? ??? ?? ? ????. ??? NVIDIA RTX 5000? ??? ???? ????? L4 ????? ??? ??? Orin?? ???? ??? ????? ??? ? ????. ??? ???? ??? L4? L2/L3 ??? ?? ???? ??? ? ????.

    ? ??? ??? ??? L2/L3 ??? ????? ??? ??? ??? ????.

    NVIDIA Orin? ?? ?? 4 ???? ??? ??? ????? ???? RTX5000 ?? GPU? ??? ????? ??? ? ?? ?? 254? ? ?? ??? ??? ?????. ??? NVIDIA Orin SoC? ???? ??? ???? ???? ??? ?? ?? ?? ???? ?????.

    • ??? sparse ????
    • DLA ??
    • ?? NVIDIA Orin SoC?? ??

    ??

    Pony? ?? ??? ?? ????? ??? ? ?? ?? ??? ???? ? ??? ?? ??? ??? ?? ????? ? ??? ??? ???? ?? ??? ???? ?? ??? ??????. ? ?? ??? ??? ??? ????.

    • ??? ??? ???? ???? ????. ???? ?? ????? ????? ???? ???? ??? ?? ????? ???.
    • ??? ???? ??? ?? ????? ???? ?? ??? ?? ?? ??? ???? ?????.
    • ?? 4? ?? 2 ??? ?? ??? ??? ??? ??? ????? ??? ?????.

    ??? ?? ??? ?????? ???? ?????? ??? ? ??? ????? ? ???? ?? ??? ??? ? ? ???? ?????. ?? ??? ???? ??? ?? ??? ?? ??? ??? ???? ? ?? ????? ?????.

    ?? ?? ??

    ? ????? Pony? ?? ??? ??????? ?? ?? ??? ??? ???? ????. ? ??? ???? ?? ??? ????? ??? ????? ??? ? ?? ?? ?????? ??????.

    Discuss (0)
    +2

    Tags

    Automotive / Transportation | DRIVE | featured | Orin SoC | technical walkthrough

    ??? ??

    Ke Li ???
    Ke Li is a senior software engineer working on GPU performance optimization at Pony.ai. Before joining Pony.ai, Ke was an HPC developer technology engineer at NVIDIA, focusing on scientific computing and medical imaging. Ke is interested in novel compute platforms and system software for autonomous driving.

    ??

    ?? ???

    人人超碰97caoporen国产