Jetson? NVIDIA Metropolis ???????? ?? AI ?? ??? ????? ???? ??? ?? AI ??????? ??? ??? ? ?? ???? ?? ??? ???? ?????.
? ?????? ?? ?? ??? ?? ???? ???? ??? ? ?? ?? ??? ?? NVIDIA Jetson ?? AI ????? Metropolis ???????? ???? ??? AI ?? ??????? ?? ? ???? ??? ?????.
? ???? ????? ??? ?? ? ?? NanoOwl ??????? ????, ?? Jetson? Metropolis ???????? ???? ???? ???????? ????? ??? ? ??? ???.
??? AI? ?????? ??
??? AI? ??? ?? ???? ? ???? ???? ??? ??? ? ??? ???? ??? ??? ?? ?????.
???? ??? AI? ??? ??? ??? ???? ??? ????? ?? ?????. ??? ??? ?? ???? ?? ?? ? ??? ????? ??? ??? ???? ??? ? ????. ??? ??? ??? CLIP, Owl, Llama, GPT, Stable Diffusion? ?? ??? ??? ??? ??? ? ????. ?? ?? ? ? ??? ?????.
Jetson? ??? AI ??? ?? ??? ??? NVIDIA Jetson ??? AI ? ? NVIDIA Jetson?? ??? AI ??? ?????.
Jetson? Metropolis ???????
Metropolis ???????? Jetson?? ????? ?? ??? ? ?? AI ??????? ??? ???? ? ??? ? ????. Metropolis ???????? ??? ??, ??? ????, IoT ???? ??, ????, ???? ?? ?? ?? ??? ? ?? ??? Docker ???? ?????. ??? ????? ?? ?? ??? ??????? ?? ? ????. ?? 2? ?? ??? ???????? ?????.
??? ??? Jetson? Metropolis ??????? ??? ?????.
Metropolis ???????? ??? AI ? ??
Metropolis ???????? ??? AI? ???? ??? ?? ?? ?? ???? ?? ??? ??? ? ????. ?? 3? Jetson? Metropolis Microservices? ???? ??? AI ?? ??????? ???? ???? ???? ??? ? ?? NanoOwl ?? ??? ?????? ?????.?
Metropolis ???????? ?? ?????? ??? ??
GitHub?? ?? ?? ?? ??? AI ??? ???, ??? Jetson?? ??? ????? ????? ????. ??? ?? ? ??? NVIDIA Jetson ??? AI ??? ?? ? ????.
??? ??? ???? ???? ????. ????? ???? ???? ???? ?? ? ????. ??? ??? ?? ?? ??? ???? ???? ???? ???. ?? ?? ???? ???? ???? ?? ??? ??? ???? ??? ??? ? ????.
Python ?? ????? ??? AI ??? NanoOwl? ??????. ??? ?? ??? ???? ???? ?? ?? ??? AI ??? ??? ? ????.
Metropolis ???????? ??? AI ??? ????? ?? ?? ???????? ??? ??? ???? ???(?? 4).
???? ???? ??, ??? ??? RTSP ????? ?????. RTSP? ??? ?? ? ?? ???????? ??? ???? ??(VST)?? ???????. ??? ????? ?? ??? ?? RTSP? ?? ???????. ?? ?????? ?? ??????? ???? ?? ? ?? Redis ????? ?????. ??? ??? Metropolis ???????? ??? ??? ???? ?? ?? ???? ?????.
??, ??? AI ???????? ????? ?? ?? ?? ?????? ????? ??????? REST API ??? ??? ? ??? ???.
?????, ??????? ?? ???????? ???? ??? ? ??? ???????? ???. ?? 5? Redis?? NanoOwl ??? ???? ?????? ???? ??? ?????.
? ?? ????? NanoOwl? ?????. ??? Python?? ??? ? ?? ?? ? ?? ??? ?? ?? ??? ?? ? ??? ?? ? ????. ? ????? ??? AI? Metropolis ???????? ???? ??? ?? ????? ???? ?? ? ?? Python ?? ??? ???? ???, ???? ???? ???? ?? ?? ??? ???????. ?? ??? ?? ??? ??? /NVIDIA-AI-IOT/mmj_genai GitHub ????? ?? ??? ??????.
Metropolis ????????? ??? ?? ??? AI ??? ????? ?? ??? ????:
- ?? ??? ??
predict?? ?? jetson-utils?????? ???? RTSP I/O? ?????.- ????? ??? ????? ?? REST ?????? ?????.
mmj_utils? ???? ????? ?????.mmj_utils? ???? VST? ?? ???? ???? ?????.mmj_utils? ???? Redis? ?????? ?????.
?? ??? ?? ?? ?? ??
NanoOwl? ??? AI ??? OwlPredictor ???? ?????. ? ???? ??????? ??? ???? ?????. ???? ??? ??? ?? ??? ????? predict ??? ???? ??? ?????.
? ?? ??? ??? ??? ?? ?? ?? ? ??? ?????.
import PIL.Image
import time
import torch
from nanoowl.owl_predictor import OwlPredictor
image = PIL.Image.open("my_image.png")
prompt = ["an owl", "a person"]
#Load model
predictor = OwlPredictor(
"google/owlvit-base-patch32",
image_encoder_engine="../data/owlvit_image_encoder_patch32.engine"
)
#Embed Text
text_encodings = predictor.encode_text(text)
#Inference
output = predictor.predict(
image=image,
text=prompt,
text_encodings=text_encodings,
threshold=0.1,
pad_square=False)???? ??? AI ??? ??? Python ?????? ??? ????. ???? ??? ??? ??, ??? ???? ??, ??? ????? ????? ???? ?? ??? ?? ? ????. ?? ??? AI ??? ????? ???? ???? OwlPredictor ???? ??? ?????? ???? ???.
jetson-utils ?????? ???? RTSP I/O ????
jetson-utils ?????? ???? RTSP ??? ??? ??? ??? ? ????. ? ?????? RTSP ????? ???? ???? ? RTSP ????? ???? ???? ? ??? ? ?? videoSource ? videoOutput ???? ?????.
from jetson_utils import videoSource, videoOutput
stream_input = "rtsp://0.0.0.0:8554/input"
stream_output = "rtsp://0.0.0.0:8555/output"
#Create stream I/O
v_input = videoSource(stream_input)
v_output = videoOutput(stream_output)
while(True):
image = v_input.Capture() #get image from stream
output = predictor.predict(image=image, text=prompt, ...)
new_image = postprocess(output)
v_output.Render(new_image) #write image to stream ? ?? ??? RTSP ????? ???? ??? ?? ?? ?? ??? ??? ? ????. ?? ???? ? ???? ???? ?? RTSP ???? ??????.
Flask? ??? ????? ?? REST ????? ????
?? ??? AI ??? ??? ???? ?? ??? ??? ?????. ???? ?? ???? ????? ???? ????? ? ??? ??? ???? ????? ???? ?? ??? ???? Flask? ???? REST ?????? ?????.
Flask ??? ??? ? ?? ????? ?? ????? Flask ??? ???? ?? ??? ? ?? ?? ???? ????. ??? ??? /NVIDIA-AI-IOT/mmj_genai GitHub ????? ?????.
from flask_server import FlaskServer
#Launch flask server and connect queue to receive prompt updates
flask_queue = Queue() #hold prompts from flask input
flask = FlaskServer(flask_queue)
flask.start_flask()
while(True):
...
if not flask_queue.empty(): #get prompt update
prompt = flask_queue.get()
output = predictor.predict(image=image, text=prompt, ...)???? ???? ????? ???? ?? ?? ?? ????? ???? ?????? ?????. GET ??? REST ?????? ????, Flask ??? ????? ????? ???? ????. ??? ?? ??? ??? ? ????? ??? ???? ????? ???? ?? ??? ?? ??? ??? ? ????.
mmj_utils? ??? ???? ????
??? ?? ??? ??, ?? ??? ??? ????? ?? ?? ????(?? 6). ?? ?? ??? ??, ?? ???? ??? ??? ?? ??? ???? ?????? ??? ? ??? ??? ??? ??? ? ????.
? ??? ????? mmj_utils ?????? DetectionGenerationCUDA?? ???? ???? ?????. ? ?????? ????? ???? ? ???? CUDA ?? ??? ???? jetson_utils? ?????.
from mmj_utils.overlay_gen import DetectionOverlayCUDA
overlay_gen = DetectionOverlayCUDA(draw_bbox=True, draw_text=True, text_size=45) #make overlay object
while(True):
...
output = predictor.predict(image=image, text=prompt, ...)
#Generate overlay and output
text_labels = [objects[x] for x in output.labels]
bboxes = output.boxes.tolist()
image = overlay_gen(image, text_labels, bboxes)#generate overlay
v_output.Render(image)?? ??? ??? ???? DetectionGenerationCUDA ????? ??????? ??? ??, ?? ?? ?? ? ??? ??? ?? ??? ? ????. mmj_utils? ??? ???? ??? ?? ??? ??? /NVIDIA-AI-IOT/mmj_utils GitHub ?????? ?????.
????? ????? ????? ???? ?? ???, ??? ??, ???? ??? ??? ??? ?????. ?? ?? ?? ???? ???? ??? ??? ??? ????? ??? ??? ???? ?????. ?? ?? ? ??? ???? RTSP ????? ???? ? ????.
mmj_utils? ???? VST? ?? ???? ??? ????
VST? RTSP ???? ???? ?? ? ?? ???? ? ? ?? ?? ? UI? ??? ? ????. VST? ????? VST REST API? ?? ????? VST REST API? ??? mmj_utils? VST ???? ?????.
Python ?????? RTSP ?? ???? ?????? ?? VST?? RTSP ??? ??? ?????. ? ??? IP ??? ?? VST? ?? ???? ?? ??? ??? ???? ??? ? ????.
from mmj_utils.vst import VST
vst = VST("http://0.0.0.0:81")
vst_rtsp_streams = vst.get_rtsp_streams()
stream_input = vst_rtsp_streams[0]
v_input = videoSource(stream_input)
...??? VST? ???? ? ?? ??? RTSP ??? ?????. ??? ? ??? ??? ???? ?? ??? ????? ??? ???? ??? ? ????.
mmj_utils? ???? Redis? ????? ????
??? AI ??? ?? ????? ?? ? ????? ???? ?? ??????? ??? ? ?? ?????? ?????.
? ??, NanoOwl? ??? ??? ?? ??? ??? ?????. ? ??? ?? ????? ??? ? ?? Redis ???? Metropolis ????? ??? ? ????. mmj_utils ??????? Redis?? ?? ?????? ???? ? ??? ?? ???? ????.
from mmj_utils.schema_gen import SchemaGenerator
schema_gen = SchemaGenerator(sensor_id=1, sensor_type="camera", sensor_loc=[10,20,30])
schema_gen.connect_redis(aredis_host=0.0.0.0, redis_port=6379, redis_stream="owl")
while True:
...
output = predictor.predict(image=image, text=prompt, ...)
#Output metadata
text_labels = [objects[x] for x in output.labels]
schema_gen(text_labels, bboxes)?? ??? ???? ?? ??? SchemaGenerator ??? ??????? Redis? ??? ? ????. ?? ?? ???? ??? ??? ???? ??? ??? ???? ??? ??? ? ????. ?? ??? Metropolis ???? ???? ?? ????????? ??? ? ??? Redis? ?????.
?????? ??
??????? ???? ?? Ingress ? Redis? ?? ??? ???? ??? ? ????. ?? ??, docker compose? ?? ??? ?? ?? AI ????? VST? ?? ?????? ???? ?????. ??? ?? I/O ? ??????? ??? ??? ?? ??????? ????(?? 7), ??????? ???? Metropolis ???????? ??? ? ????.
- ??? AI ??????? ????????.
- ??? ??? ???? ?????.
docker compose? ??????? ?????.- ????? ??? ?????.
??? AI ?????? ?????
??? ?? ? ?? ??? Docker? ???? ??? AI ??????? ??????? ????.
?? ?? ?? ??? jetson-containers ????? ???? ????. ? ????? ??? AI ??? ??? ?? ?? ??????? ???? ?? Jetson? Docker ????? ?? ??? ? ?? ??? ?????. jetson-containers? ???? ??? ???? ?? ????? ?? ??, ?????? ??? ??? AI ??? ???? ? ??? ?? ???? ????? ????? ??? ??? ?????.
NanoOwl ??? ????? ???? ??? ?? ??? ??? GitHub ????? /src/readme ??? ?????.
??? ??? ??? ??
???? Metropolis ????????? ???? ??? ??? ???? ?????. ??? ??? ???? Metropolis ???????? ?? ??????? ???? ? ??? ?? ??? ?????.
? ?? ?? AI ???????? ????, Redis, ???? ??? ???? ?????. ??? ???? APT? ?? ??? ???? systemctl? ??? ? ????.
??? ??? ???? ???? ???? ??? ?? ??? ??? Jetson? Metropolis ??????? ?? ?? ???? ?????.
docker compose? ?????? ??
??????? ??????? ??? ??? ???? ?????? docker compose? ???? VST ?? Analytics? ?? ?? ?????? ???? ?? ??????? ??? ? ????.
??? ??? ??? ?? ??? ?? ??? ????? ???? docker-compose.yaml ??? ????. docker ?? ??? ??? ??? docker ?? up ? docker ?? down ??? ???? ??????? ????? ??? ? ????.
docker ??? ?? ??? ??? GitHub ????? /deploy/readme ??? ?????.
????? ?? ??
??????? ??? ??? VST? ?? RTSP ???? ???? REST API? ?? ??? AI ??? ?? ???? ??? ????? ???? RTSP ??? ??? ????? ?? ?? ??? ??? ? ????. ?? Redis?? ????? ??? ??? ?? ????.
??
? ?????? ??? AI ??? ???? Jetson? Metropolis ???????? ???? ??? ??????. ??? AI? Metropolis ???????? ???? ???? ??? ??? ??? ?? ??????? ??? ??? ? ????.
???? ???? ?? ??? ??? Jetson? Metropolis ??????? ?? ???? ??????. ?? ?? ??????? ?? ?? ???? ???? ??? ?? ??? ??? ??? /NVIDIA-AI-IOT/mmj_genai GitHub ????? ??????.
