NVIDIA Riva? ??? ?? ???? ?????? ????

Reading Time: 9 minutes

?? ??? ?? ?? ??? ???? ?? ??? ??? ?? ???????. ?? ??? ????? ??? ?? ??? ??? ?? ???? ?? ???? ???? ????. ?? ??? ?? ?? ??????, ???? ? ???? ???? ????? ?????.

??? AI? ??? ??(transcript)??, ???? ?? ??? ????? ??????? ??? ? ????. NVIDIA Riva? ? ?? ??? ??? ??? ? ?? ??? ??? ??? ???? ??? ???? SDK???.

??1. ?? ???? ??? ??? ??(NER)? ???? NVIDIA Riva

?? ?????? ??? ?? ???? ??? ???? ???? ?? ??? ??? ???? ? ??????? ???? ?????. ??? ?? ??? WebRTC ?? ??-?-??(peer-to-peer) ??? ?? ?? ?? ?????? PeerJS? ?????. ??? ??? ??, Riva? ?? ?? ??(ASR)? ?????. ???? ?? ?? ??? ???? ??? ??(NER)? Riva? ?????. ?? ??? ?? ?? ???? NER ??? ???? ??? ?? ???????. ? ???? ?? ??? ???? ??? ??? ???? ???? ??? ?? ?? ??? ??????? Riva ??(Riva Samples) ?? ????? ??? ?? ?????.

? ??????? ??? ??-?-?? ?? ?? ? ???????? ?????. ???? ??? ???? ?????.

HTML ??? 1?
????? ?????? ?? 1?
?? ?????? ?? 1?(?? ???? ??-?-?? ?? ???)

????? ?? ???? ??? ??? ???, ?? ??????? ?? ? ???? ??? ?? ?? ???. ID? ?? ??, ??, ??, ?? ??? ???? ?? ?? ???? ??? ????. ?

? ???? ? ??????? ASR? ??? ??(NLP) ??? ???? ?? ????? ???, ?? ?????? ??? ?? ?? ??? ?????. ? ??????? ??? ?????, Node.js? ???? ?????(Express)? ? ??? ????? ??? ?????. ??-?-?? ??? WebRTC ?? ???? ??????? ??? PeerJS? ????. ?????? ????? ? ???? ??? ??, ??? ??? ??? ?? ??? ????. ?? ?? ??? ??? ? ?????? ?? ?? ???? ???. ??? ?? ?? ???? ?? ?? ??.

?? ???? ???? ????? ????, ?? ???? ????, ??? ?? ??? ??? ? ????.

ASR? NLP ????

NVIDIA Riva? ??? ??? AI ???? ??? ??? ???? SDK???. Riva? ?? ?? ???? ???? ??? ??? ?? Riva ?? ??? ???? ??? ??? ? ????. ? ???? ??? ????? ? ??? ?? ??? ?????.

config.sh? ??? ?????.
riva_init.sh? ??? ??? ????? ???, ??? ?????.
riva_start.sh.? Riva ?? ??? ?????.

??? ???? ??????? Riva? ??? ?? gRPC ?????? ?????. ?? ?? ??? ?? ??? ????? Riva? ??? ??? ????? ????? ??????. riva_start_client.sh ??? ?? ?????? ???? ???? ????? Riva? ???? ???? ??? ? ????.

??3? Riva? ??? ??????? ?? ?? ??? ?????. ?? ?? ??(Node.js ??????)? ?? ??? ??? ????? ?? Riva ???? ?????.

? ???????? Riva? ??? ? ?????. ??? ???? ???? ??? ? ???? ?? ??(???)? ??? ???? ???. ?? ?? ??????? ??? ???? ??? ??, ? ???? Node.js ??? ?????. ? ??? gRPC? Riva? ??? ???? ???? ???? ? ??? ??????? ?? ?????. ?????? ?????? ???? ???? ??-?-?? ??? ???? ? ???? ?? ??? ??? ? ? ?? ???.

? ??????? ??? ????

?? ??? ?? ??? ???? ?? WebRTC ???? ???? ??????? ??? ???? ???? ? ????. ????? ?????? ???? ???? Start? ???? ??? Riva ??? ??????. ?? ??? ?? ??? ???? ???? ???? ??? ?? ?????? ?????.

socket = socketio.on('connect', function() {
    console.log('Connected to speech server');
});

WebRTC ???? ???? ???(graph)? ???? ?????. ?????? ??? ?? ??? ????? ??? ?????.

??? ??? ?????. ? ??? ???? ?? ??-?? ????? ??? ????.
? ???? ??? ???? ??? ?????.
? ??? ?? ???? ?? ???? ???? ?????.

??? ???? ??? ? ??? ???? ??? ??? ? ??(web worker)? ?? ????(resampling)?? ?? ??? ?? ??? ????. ??? ?? ??? ???? ????(resampler)? ??????.

audio_context = new AudioContext();
sampleRate = audio_context.sampleRate;
let audioInput = audio_context.createMediaStreamSource(localStream);
let bufferSize = 4096;
let recorder = audio_context.createScriptProcessor(bufferSize, 1, 1);
let worker = new Worker(resampleWorker);
worker.postMessage({
    command: 'init',
    config: {
        sampleRate: sampleRate,
        outputSampleRate: 16000
    }
});

????? ??? ??? ? ??? ???? ??????. ?? ??? ??? ??? ???? ??? ??????. ??? ?? ???? ????? ? ?? ??? ??? ??? ?????.

recorder.onaudioprocess =  function(audioProcessingEvent) {
    let inputBuffer = audioProcessingEvent.inputBuffer;
    worker.postMessage({
        command: 'convert',
        // You only need the first channel
        buffer: inputBuffer.getChannelData(0)
    });
    worker.onmessage = function(msg) {
        if (msg.data.command == 'newBuffer') {
            socket.emit('audio_in', msg.data.resampled.buffer);
        }
    };
};

???? Riva? ??? ?? ????? ??? ????. Riva? ?? ????? ??? ? ?? ????. ??? ????? ?????? ???? ??? ?? ??? ?? ??? ??? ??? ?? ?? ??????. ?? ? ???? ??? ??? ???(?? ???? ???)? ??? ? ????.

audioInput.connect(recorder);
recorder.connect(audio_context.destination);

?? ??????? ???? ????? ???? ??? ???? ????? ? ??? ?? ??? ????. ???? ? ???? ???? ???? ??? ???????.

???? Riva? ?????

Node.js? ??? ?? ?? ????? ????? ??? Socket.IO? ??? ?? ??(incoming connection)? ?????. ??? ? ?? ?? Riva ??? ??????.

io.on('connect', (socket) => {
    console.log('Client connected from %s', socket.handshake.address);
    // Initialize Riva
    socket.handshake.session.asr = new ASRPipe();
    socket.handshake.session.asr.setupASR();
    socket.handshake.session.asr.mainASR(function(result){
        var nlpResult;
        if (result.transcript == undefined) {
            return;
        }
        // Final transcripts also get sent to NLP before returning
        if (result.is_final) {
            nlp.getRivaNer(result.transcript)
            .then(function(nerResult) {
                result.annotations = nerResult;
                socket.emit('transcript', result);
            }, function(error) {
                result.annotations = {err: error};
                socket.emit('transcript', result);
            });
        } else {
            socket.emit('transcript', result);
        }
    });
});

? ???? ??? ? ?? ??? ????. ASRPipe? ?? ???? ??? handshake.session ????? ???? ? ?????? ??? ??? ??? Riva ???? ????. setupASR? ???? Riva ??? ??? ? ASR ?? ??(loop)? ?????.

ASR ?? ??? ????(asynchronous)???. ???? ??? ??? ??(batch)? ????? ???? ? ??? ?? ??? ???? ????. ??? mainASR? ???? ?????. ???? ???? Riva? ??? ??? ? ?????. ? ?? ???? ???(?, ? ?? ??? ???)? ?? ???? ?? ???? ?? ??????. ??? ?? ?? ?? ??? ???? ?? ??? ??? ???? ‘??’?? ???? ???? ??? ????. ? ??? ?? ??? ??????? ?????, ?? ??? ?? ?? ???? NLP ????? ?? NER? ?????. ? ? ?? ??? ??? ??????? ?? ??? transcript ???? ?? ??? ?? ??? ?????.

Socket.IO? ?? ???? ???(listener)? ??? ? ????. ??? ???? ??? ??? ??? ? ??. ?????? ??? ??? ??? ?? ??? ???? audio_in???????. ?? ???? ???? Riva ???? ??? ?? ??? io.on(‘connect’)????? ??????.

socket.on('audio_in', (data) => {
    socket.handshake.session.asr.recognizeStream.write({audio_content: data});
});

? ??? ??? ???? ?? ?? ??? ???? ????. ??? ??? ? Riva ???? ?? ?????? ??? ??? ???? ?? ??? ?? ???.

?? ?? ?? ???

?? ASR? ???? gRPC ?????? ???????. Node.js? ???? gRPC ????? ??? ?? ??? 3??? ????.

???? ??? Riva API ????.
? API ??? ?? ??(convenience function) ????.
?????? Riva ?? ??? ??? ?????.

?? ??? ASRPipe ???? ???? asr.js ???? ?? Riva API? ???? ???. ?

const jAudio = require('./protos/src/riva_proto/audio_pb');
var asrProto = 'src/riva_proto/riva_asr.proto';
var protoOptions = {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true,
    includeDirs: [protoRoot]
};
var asrPkgDef = protoLoader.loadSync(asrProto, protoOptions);
var jAsr = grpc.loadPackageDefinition(asrPkgDef).nvidia.riva.asr;

ASRPipe????? ?? ?? ??? ?? ??? ??????.

class ASRPipe {
    setupASR() {
        // the Riva ASR client
        this.asrClient = new jAsr.RivaSpeechRecognition(process.env.RIVA_API_URL, grpc.credentials.createInsecure());
        this.firstRequest = {
            streaming_config: {
                config: {
                    encoding: jAudio.AudioEncoding.LINEAR_PCM,
                    sample_rate_hertz: 16000,
                    language_code: ‘en-US’,
                    max_alternatives: 1,
                    enable_automatic_punctuation: true
                },
                interim_results: true
            }
        };
    }
}

??? Riva ASR ?????? ???? ?? ?? ????? ????? ???? ?? ? ?? ???? ?????. ??? ??? ???? mainASR ??? ??? ?? ASR ???? ?????.

async mainASR(transcription_cb) {
    this.recognizeStream = this.asrClient.streamingRecognize()
    .on('data', function(data){
        if (data.results == undefined || data.results[0] == undefined) {
            return;
        }
        // transcription_cb is the result-handling callback
        transcription_cb({
            transcript: data.results[0].alternatives[0].transcript,
            is_final: data.results[0].is_final
        });
    })
    .on('error', (error) => {
        console.log('Error via streamingRecognize callback');
        console.log(error);
    })
    .on('end', () => {
        console.log('StreamingRecognize end');
    });
    // First request to the stream is the configuration
    this.recognizeStream.write(this.firstRequest);
}

streamingRecognize ??? ???????. ??? ???? Riva ? ??? ??? ?? ??? ????, ? ??? ????(repackage)? ?? ??? ?? ??? ?????.

NER ?? ???

Riva NER ???? ??? ? ?????. ??? ????? NLP API? ??? ??, ClassifyTokens???? ?? ??? ???? ? ??? ?????. ?? ???? ???? ??? Riva? ??? ??? ?????. ??? ?? computeSpans ??? ???? ??? ? ??? ?????.

function getRivaNer(text) {
    var entities;
    req = { text: , model: {model_name: process.env.RIVA_NER_MODEL} };
    return new Promise(function(resolve, reject) {
        nlpClient.ClassifyTokens(req, function(err, resp_ner) {
            if (err) {
                reject(err);
            } else {
                entities = computeSpans(text, resp_ner.results[0].results);
                resolve({ner: entities});
            }
        });
    });
};

? ???? Riva?? gRPC ??? ?????. ??????? ???? ???? ???? ??? Riva? ??? ???? ???? ??, ??? ? ???? ??? ??? ? ??. Riva? ??? ??? transcript ???? ?? ??? ?? ???? ? ?????? ?????. ?? ?????? ? ??? ??? ?? ??(circuit)? ?????.

??? ?????? ?????

????? ? ?????? ???? ???? ????? ?????. ?????-?? ?? ??? Socket.IO ??? ?? ????? ??? ???? transcript????? ???? ???? ??? ??? ?? ???.

socket.on('transcript', function(result) {
    document.getElementById('input_field').value = result.transcript;
    if (result.is_final) {
        // Erase input field
        $('#input_field').val('');
        showAnnotatedTranscript(username, result.annotations, result.transcript);
        // Send the transcript to the peer to render
        if (peerConn != undefined && callActive) {
            peerConn.send({from: username, type: 'transcript', annotations: result.annotations, text: result.transcript});
        }
    }
});

input_field? ? UI?? ?? ???? ????? ??? ???, ??? ?? ????? ???????. ?????? ???? ??? ??? ??? ??? ?? ??? ????. ???? ‘??’?? ???? ?? ??? ??? ???? ?? ???? ???? ?? ???? ?? ? ??? ?? ??? ??? ? ? ?? ???.

??? ??? ???? ?? HTML? CSS? ?????. ??? ?? ??? ??? ??? displaCy-ENT? ???? ?? ???? ?????.

?? NER? ?? ?? ????

????? Riva? ??(Location), ??(Person), ???(Organization), ??(Time) ?? ???? ???? NER ??? ?????. ? ??? ?? ??? ??? ?? ?? ?? ??? ??????? ?? ????. ? ???? ????? ??? ??? AI? ?? ?? ??????? ??? ? ??? ?????. ???? ???(Rivato) ?? ??? ??? ?? NER ??? ???? ???????.

??? ??? ???? ?? ?????? ?? ??? ??? ?????. ??? ??? ???? ?? ?? ??? ?? ?? ??? ??? ??? ???? ?? ???? ??? ????. NVIDIA TAO Toolkit? ???(Python) ??? AI ????, ?? ?? ??? ?? ??? ??????? ?? ??? ????? ?? ??????.

?? ???? ??? ??? ???? ??? ????? ?? ?? ?? ????. 2010? i2b2/VA ??? ?? ?? ??? NER ?????? ??????. ???? ??(?? ?? ??)? ??(?? ?? ??), ??? ??? ???? ???? ??? ??? ???? ??? ????. ? ?????? ?? NLP ?????? ?? ????? ???? ??? ???? ???? ??? ? ????.

NER ???? ?? IOB ?? ??? ????, ??? ? ? ??(token)? ?? ??(Beginning an entity)?? ??(? ?? ??) ?(Inside an entity), ?? ?(Outside) ? ??? ??????. ?? ???? ?? ??? ?? ??? ?? ? ????.

???:

DISCHARGE DIAGNOSES :
Coronary artery disease , status post coronary artery bypass graft .

??:

O O O
B-problem I-problem I-problem O O O B-treatment I-treatment I-treatment I-treatment O

??? TAO Toolkit? ???? ??? ??????. TAO? ??? NER ??? ???? ??? ?? ?? ??? TAO-Riva NER ???? ??? ???? ?????. ? ??? ?? ?? ??? ?? ?? ?????? ‘bert-base-uncased’?? ????, ???? ??? i2b2 ???? NER ??? ?? ?? ?????.

??? ??? ??? ??? ? ?? ??? ????. ?? ??? ?????? ??? ?? TAO Toolkit? ??? ???? ?? ?????. ?? TAO? ??? ? ??? Riva? ???? ??? ???? ? ?? ??. Riva? ???? ?? ??? ?? ??? ??? ???? ???? ?????. ?? ?? ? ???? ??? ?? Riva ??? ?????. ? ??? ??? NVIDIA Riva Speech Skills? ?????.

?? ??? TAO Toolkit? ??? ?? ?? ??? ??? ???? ?? ?????. ?

!tlt  token_classification train \
      -e $SPECS_DIR/train.yaml \      # Specification file
      -g 1 \
      -k $KEY \
      -r $RESULTS_DIR/medical_ner \
      data_dir={destination_mount}/data/i2b2 \
      model.label_ids={destination_mount}/data/i2b2/label_ids.csv \
      trainer.max_epochs=10

? ??? ??? ? TAO Toolkit? ??? trained-model.tlt?? ??? ??? ?????. ?? ????? ? ??? Riva? ??? ??? ? ?? .riva????? ???? ???.

!tlt  token_classification export \
      -e $SPECS_DIR/export.yaml \      # Specification file
      -g 1 \
      -m $RESULTS_DIR/medical_ner/checkpoints/trained-model.tlt \
      -k $KEY \
      -r $RESULTS_DIR/medical_ner \
      export_format=RIVA

? ??? ?? Riva?? ??? ? ?? exported-model.riva? ???? ???.

Riva ServiceMaker Docker ???? ??? ??? ??? ??/?????.

docker pull nvcr.io/riva/riva-speech:1.0.0b1-rc5-servicemaker
docker run --gpus all -it --rm
      -v $RESULTS_DIR/medical_ner:/servicemaker-dev
      -v $RIVA_REPO_DIR:/data
      --entrypoint="/bin/bash"
      nvcr.io/ea-riva-stage/riva-service-maker:1.0.0b1-rc5
riva-build token_classification
      --IOB=true
      /data/med-ner.jmir
      /servicemaker-dev/exported-model.riva
riva-deploy /data/med-ner.jmir /data/models -f

–IOB ???? ?? ???? ‘IOB ?? NER ??’? ????? ??? ?? ???? ??????. $RIVA_REPO_DIR? ?? ?? ?????? riva_init.sh? ??? ? ??? Riva ???? ?????. ? ????? ??? ??? ?? ??? NER ? ?? ?? ??? ?? ?? ????? ??? ????. riva-deploy? ???? Riva? ? ??? ??? NER ??? ?????.

? ??? NER ??? ???? ?? ???????? ?? ?? ????? ???? ?? ??? ??? ??? ? ????.

???? ??

Riva? ??? ???? ????? ?????, Riva SDK? ??? ???????? ????? ????? ??? ????? ????? ??? ?????. ?? ?? ??? ??? ?? ??(Helm chart) ??? ?? ?????.

???? ????? ?????, ?? 3.0, ?????? NVIDIA GPU Operator? ?????. ?? ?? NGC?? Riva AI ???? ?? ??? ???????. ?

export NGC_API_KEY=<your_api_key>
helm fetch https://helm.ngc.nvidia.com/ea-riva/charts/riva-api-0.1-ea.tgz --username='$oauthtoken' --password=<YOUR API KEY>

??? ??? ? ? /riva-api?? ??? ??? ???? ????.

riva-api
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   └── service.yaml
└── values.yaml

Chart.yaml ???? ??? ?? ? ?? ?? ??? ?? ????. ?? ??? ????? values.yaml ???? ??? ?? ??? ?????.

replicaCount: Riva ??? ???? ?.
speechServices [asr | nlp | tts]: ?? ??? ???? ?? ? ?? ? ??(Boolean) ????.
ngcModelConfigs: NGC?? ????? ?? ??.
service: ????? ??? ?? ??? ???.

values.yaml ???? ?? ????? ????? ?? ??? ??? ??? ????. ????? ????? Riva ?? ??? ??? ??? ????.

GPU ??? ?? ?? ?? ??? ??? Riva Speech Docker ????? ???? ???.
?? ????? ??? ?? ??? ??????.
Triton Inference Server? ??? ??/?????.
???? ?? ??? ????? ??? ?? ??? ???.
GPU? ?? ???? ???? ?? ??????(Prometheus) ???? ?????.

????? Riva ??? ???? ?? ??? ??? ?????.

helm install riva_server riva-api

?? –set???? ??? values.yaml ??? ?? ?? ??? ?? ????. NGC_API_KEY? ngcCredentials.email, model_key_string??? ??? ????? ?????. ????? model_key_string???? tlt_encode? ??? ????.

helm install riva-api --set ngcCredentials.password=`echo -n $NGC_API_KEY | base64 -w0` --set ngcCredentials.email=your_email@your_domain.com --set modelRepoGenerator.modelDeployKey=`echo -n model_key_string | base64 -w0`

> NAME: riva-api
LAST DEPLOYED: Thu Jan 28 12:05:36 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

??? ??? Riva ??? ?? ?? ????? ?????.

kubectl get pods
kubectl logs <pod name>

Riva ??? ??? ????? ?? ???? IP ??? ???? ???.

kubectl get services
> NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                                                                        AGE
riva-api   LoadBalancer   10.100.194.170   ac51c23e62d094aa68ac2adb98edb7eb-798330929.us-east-2.elb.amazonaws.com   8000:30034/TCP,8001:31749/TCP,8002:30708/TCP,50051:30513/TCP,60051:31739/TCP   2m19s
kubernetes   ClusterIP      10.100.0.1       <none>                                                                   443/TCP                                                                        123m

EXTERNAL-IP ?? env.txt?? external endpoint? ??? ? ????.

?RIVA_API_URL= <external-IP>?

???? ??????? ?? ?????? ?? ? ??????? ?? ???? ???? ???. ??? ? ???? ???? ????? Node.js ??????? ?????. ?? ???????? ?? ??? ???? IP ??? ??? Riva ASR? NLP ??? ??? ?? ?????.

??

???? ?? ??? ?? ??????? ??? AI ??????? ??? ??? ????? ????? ?? ?? ????. ?? ????? ??? NVIDIA Riva? ??? ??? ??????? ??? ??? ??? ?? ??? ??? ???? ??? ?????. ?? TAO Toolkit? ??? ??????? ????????, ?? ??? ??? ?? ???? ??? ???????. ?? Riva? ?????? ??? ?????.