Logging

Note: The nvlog feature is currently under development. You can enable this feature for testing purposes, but do not enable the feature in production systems until it is enabled by default in NVIDIA DRIVE? OS releases.

NVIDIA DRIVE? OS has a logging component that allows logs from multiple NVIDIA DRIVE OS components to be obtained. Currently, support is only available for applications within NVIDIA DRIVE OS Linux Guest VM to send logs that are stored in a file that is created and accessible from Linux Guest VM. This support will be enhanced in future releases to support more execution environments outside of Guest VM.

Terms

Logging Entity : Application threads(s) trying to log, each thread is an entity.

Entity Logging library : entity specific logging library build that is linked to Guest VM entities (libnvlog_guestvm_entity)

nvlogclient_fast : logging client/server combined application.

Logging library : full build of logging library that is linked to nvlogclient_fast (libnvlog.so)

Logging Pipeline

The logging pipeline is as follows:
(Logging Entity + libnvlog_guestvm_entity) -> (nvlogclient_fast + libnvlog.so) -> log storage file

Components

The following lists the source and binary locations within the SDK:
  1. Pre-built binaries:
    <top>/drive-linux/lib-target/libnvlog.so
<top>/drive-linux/filesystem/contents/bin/nvlogclient_fast
<top>/drive-linux/filesystem/contents/bin/nvlogtest
<top>/drive-linux/filesystem/contents/bin/nvlogtest_stdin
  2. Source code:
    Source files for the test applications (nvlogtest*) and the Logging Client (nvlogclient_fast) can be found under:
    <top>/drive-linux/samples/tools/nvlog/
  3. Include files and user API:

    Any application needing to send logs using this new framework must use the <top>/drive-linux/samples/tools/nvlog/include/nvlog.h header file and link to libnvlog_guestvm_entity library. API documentation can be found in the same header file.

In a booted DRIVE OS Linux Guest VM console, the following logging-related components are available:
  1. /usr/local/bin/

    nvlogclient_fast: Application component responsible for collecting logs from different Logging Entities within Linux Guest VM and storing them.

    nvlogtest: Test application that has two threads sending log messages every second.

    nvlogtest_stdin: Test application that takes input from stdin and sends each line as a log message.

  2. /usr/local/lib/

    libnvlog.so: Logging library that provides logging APIs.

Execution Steps

Default Behavior

The nvlog feature is enabled by default on AV + L for all the platforms. The following messages appear during boot:
Dec 21 00:54:11 tegra-ubuntu systemd[1]: Starting nvlog client logging service...
Dec 21 00:54:11 tegra-ubuntu bash[1140]: Launched /usr/local/bin/nvlogclient_fast !!!
Dec 21 00:54:11 tegra-ubuntu nvlogclient_fast: SHMDiscoveryClientThread: Discovery client thread starting..
Dec 21 00:54:11 tegra-ubuntu nvlogclient_fast: NvLogDiscoveryPlugin_nvsciipc_Init - Entry
Dec 21 00:54:11 tegra-ubuntu systemd[1]: Started nvlog client logging service.
For the standard build, the output log file is created in the /home/nvidia/ directory:
nvidia@tegra-ubuntu:~$ ls -al /home/nvidia/nvlog_system_log.txt
-rw-r--r--. 1 root root 20480 Dec 21 00:54 /home/nvidia/nvlog_system_log.txt

nvidia@tegra-ubuntu:~$ cat /home/nvidia/nvlog_system_log.txt
      EUID       PID ENTITY ID    SEQ     TIMESTAMP  MSG
         0      1305      1305              3      346841060    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1305      1305              2      346840076    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk11_ivc_d2
         0      1305      1305              1      346838406    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk11_ivc_d1
         0      1291      1304              1      346765176    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  pkcs11-keystore-tk10
         0      1291      1291              3      346758455    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1291      1291              2      346757262    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk10_ivc_d2
         0      1291      1291              1      346755441    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk10_ivc_d1
         0      1284      1299              1      346717953    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  pkcs11-keystore-tk1
         0      1284      1284              3      346711198    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1284      1284              2      346710136    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk1_ivc_d2
         0      1284      1284              1      346708419    Module_id 51 Severity 5 : Server trying IVC endpoint:  pkcs11_keystore_tk1_ivc_d1
         0      1272      1281              1      346543194    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  nvmacsec
         0      1272      1272              3      346537165    Module_id 51 Severity 5 : created TA endpoint: 1, created CA endpoints: 2
         0      1272      1272              2      346536002    Module_id 51 Severity 5 : Server trying IVC endpoint:  nvmacsec_ta_d2
         0      1272      1272              1      346534344    Module_id 51 Severity 5 : Server trying IVC endpoint:  nvmacsec_ta_d1
         0      1260      1270              1      346428824    Module_id 51 Severity 5 : Waiting for Suspend Request from Kernel:  gp-se
…

For the production build, the output log file is created in the /var directory.

Disable nvlog

To disable nvlog,

  1. Change the status of the nvlog node to disabled in the tegra234-linux-gos.dtsi file.

    nvlog {
       status = "disabled";
       guest_vm {
                 ...
       }
}
  2. Recompile and flash.
Note:
  • Raw output from Logging Client is not sequential with regard to generated logs. This is due to the underlying implementation where readers and writers are lockless and the reader retrieves messages in bursts (in reverse order for each burst). The raw output file must be sorted on the basis of timestamp (column 5) to get the logs from multiple sources in order (example: sort -k5 -n).
  • Log messages can be lost if entity (or producer) is spewing many log messages before nvlogclient_fast (or consumer) is able to read the messages.
  • The first log message is not currently retrieved or stored by the Logging Client. This is a known issue.