• <xmp id="om0om">
    •
    • <table id="om0om"><noscript id="om0om"></noscript></table>
  • Related Resources

    Megatron? DeepSpeed? ? ???? ???? ?? ? ?? ?? ?? Megatron-Turing NLG ????(2)

    2022? 1? 24?
    By and
    +1
    Discuss (0)
    Reading Time: 4 minutes

    Megatron? DeepSpeed? ? ???? ???? ?? ? ?? ?? ?? Megatron-Turing NLG ????(1)? ?????.

    ??? ??

    ?? ??(LM) ??? ?? ??? ??? ???? ?? ????? ??? ??, ?? ??? ??? ???? NLP ??? ????? ??? ? ??? ?????.

    LM? ????? ??? ?? ?? ?? ??? ???? ??? ????? MT-NLG? ??? ??, ? ??? ?? ??? NLP ????? ??? ??? ???? ???? ??????. ???? ??? ?? 5? ??? ?? 8?? ???? ??????.

    • ??? ?? ???? ???(LAMBADA)?? ?? ??? ??? ??? ??? ??? ?????.
    • ?? ???? ???-h(RACE-h)? ?Q(BoolQ)?? ?? ??? ??? ??? ???? ???? ??? ?? ?? ?????.
    • ?? ?? ???? ?????(PiQA)? ?????(HellaSwag), ?????(Winogrande)? ??? ???? ???? ??? ??? ??? ???? ??? ??? ???????.
    • ??? ??? ??, ???? ????? ANLI-R2? ??(HANS)? ?? ???? ???? ?? ??? ?????.
    • ??? ??? ?? ???? ??(WiC)? ???? ??? ???? ?????.

    ???(reproducibility)? ???? ???? ?? ?? ????? ‘lm-evaluation-harness’? ??? ?? ?? ??? ???? ??, ????? ??? ??? ?? ??? ?? ??? ?? ??? ?? ??? ????? ????. ??? ? ?? ?? ??? ?? ???? ?? ???, ??, ?? ???? MT-NLG ??? ??????.

    Table 2? ??? ???(metric)? ?????. ????? ??? ? ?? ?? ??? ??? ?? ??? ?????, ??? ?? ?? ?? ??? ??? ??????. ?? ?? ???? ???-h, ANLI-R2? ??? ??, ?? ????? ?? ??? ??? ??????.

    ?? ??? ?? ????. ??? ??? ??? ??? MT-NLG? ???? ???? ?????. ? ?? ??? ? ?? ??? ??? ????? ?? ???? ???(?? ??? ANLI)?? ?? ???????. ?? ??? ???? ??? ?? ????? ??. ???? ?? ??? ??? ?????? ??? ??? ??????. MT-NLG? ?? ???? ?? ?? ???? ??????? ???? ???? ???. ??? ??? ?? ??? ?? ? ??? ???? ????.

    ?? ?????? ??, ????? ??? ???? ???? ?? ??(baselines)? ?? ?????. ?? ???? ??? ??? MNLI?? ????? BERT ?? ??? ?? ??? ???? ??? ??? ??? ??? ??, ??? ????? ??? ??? ??? ?????. ?? ?? ???? ?? ???? ??? ???? ?? ????? ?? ???? ??? ?????.

    MT-NLG ?? ?? ? ??? ???? ??? ??? ???? ?? ??? ??? ??? ????, 1? ?? 4?? ?(??)? ??? ?? ??? ?? ?????. ????? ???, ??, ?? ???? MT-NLG ??? ????? ?? ??? ??? ??? ?? 1?? ??????.

    ???? ???? ???? ??? ??? ?????? ?? ???? ?? ??? ??? ?? ???? ?? ??????(?3 ??). MT-NLG ??? ???? ???? ?? ??? ??? ? ??? ?(?? 1)? ??? ????. ??? ???? ?? ?? ??? ??(?? 2)? ????????. ? ??? ?? ??? ???? ??? ??? ??? ??? ??? ???? ??? ???? ? ????.

    ?? ???? ??(?3? ??? ?)? ????? ?????. ? ????? ??? ??? ?? ??? ??? ??? ???? ???? ??? ??? ????. ??? ????? ??? ??? ??(NLI) ??? ? ?? ??? ??? ?? ???. ?? ??? ???? NLI ?????? ???? ??(bias) ??? ??? ?? ??? ? ?? ?? ??? ?? ???? ??. ?? ?? MT-NLG? ?? ?? ?? ???? ?????.

    ?? ??? ??

    ?? ?? ???? ??? ?? ?? ?????? ?????? ??? ??(toxicity) ?? ??? ??? ??? ???. ?? ??? ? ?? ??? ??, ????? NVIDIA? ???????? ??? AI ????? ???? ??? ???? ????.

    ??? MT-NLG? ????? ???? ????? ???? ???? ??? ????? ??? ?????? ????. NVIDIA? ???????? ?? ??? ??? ???? ??, ??? ??? ????? ??? ????? ???? ?????.

    ?? ???? ????? MT-NLG? ???? ???? ????? ????? ?? ??? ??? ??, ????? ??? ???? ???? ??? ???? ???. ??? ??? ‘???????? ?? ?? AI? ?? ??(Microsoft Responsible AI Principles)’? ??? ???? ???? ????. ? ???? ???, ???? ???, ?????? ??, ???, ???, ??? ?? ???? ?? AI? ??? ??? ?? ?? ?? ?? ?? ???? ??? ??? ???? ????.

    ??

    ?? ??? AI? ??? ??? ??(Moore’s law)? ?? ???? ??? ?? ????. ? ?? GPU?? ??? ???? ?? ?? ??? ?? ????? ?? ??? ??? ??? ???? ??? ????? ???? ??. ??? AI ??? ???????(hyperscaling)? ? ??? ???? ???? ???? ? ?? ?? ??? ???? ???.

    ? ? ?? ??? ???? ????? ??? ???? ???? ??? ????. MT-NLG? NVIDIA Selene?? ??????? ?? NDv4 ?? ?????? Megatron-LM? ???? ?? ????? ??? ???? ??? ?? AI ??? ????? ? ?? ???? ???? ???? ??? ? ? ??.

    ??? ??? ??? ??? ??? ??? ???? AI? ?? ???? ??? ????? ?? ??? ?? ??? ????. Megatron-LM? ?????? ??? ???, ??? ??? AI ?? ??? ???? ??? ??? AI ??? ????? ?? ???? ???? ???? ????.

    ??? MT-NLG? ?? ???? ????? ?? ????? ??? ??? NLP? ??? ?? ????? ?????. ? ??? ??? ?? ?? ??? ????, ??? ???? ??? ?? ??? ?? ????.

    ??

    ? ?????? ??? ????? ??????.

    NVIDIA: ???? ????(Mostofa Patwary), ???? ???(Mohammad Shoeybi), ??? ?????(Patrick LeGresley), ???? ?????(Shrimai Prabhumoye), ??? ???(Jared Casper), ??? ????(Vijay Korthikanti), ??? ?(Vartika Singh), ?? ????(Julie Bernauer), ??? ???(Michael Houston), ???? ????(Bryan Catanzaro).

    ???????: ?? ???(Shaden Smith), ??? ??(Brandon Norick), ?? ?????(Samyam Rajbhandari), ? ??(Zhun Liu), ?? ????(George Zerveas), ?? ?(Elton Zhang), ?? ???? ?????(Reza Yazdani Aminabadi), ?? ?(Xia Song), ??? ?(Yuxiong He), ??? ?(Jeffrey Zhu), ??? ???(Jennifer Cruzan), ??? ??(Umesh Madan), ??? ????(Luis Vargas), ???? ???(Saurabh Tiwary).

    Discuss (0)
    +1

    Tags

    DGX | Conversational AI / NLP | featured | HPC / Supercomputing | Megatron | technical walkthrough

    ??? ??

    Paresh Kharya ???
    Paresh Kharya is the senior director of product management and marketing for accelerated computing at NVIDIA. He focuses on NVIDIA data center products, including server GPUs, CPUs, and NVIDIA computing software platform for AI and HPC. Previously, Paresh held a variety of business roles in the high-tech industry, including product management at Adobe and business development at Tech Mahindra. Paresh has an MBA from the Indian Institute of Management and a BS in computer science from the National Institute of Technology, India.
    Ali Alvi ???
    Ali Alvi is the group program manager for the Microsoft Turing team, which is a cross-company deep learning/AI initiative. They are developing and productizing high ambition, deep learning capabilities across Microsoft’s suite of products (Outlook, Word, PowerPoint, SharePoint, Bing, Dynamics, Maps, and so on). His focus includes QA over unstructured and semi-structured data, relevance models for search, machine reading comprehension, natural language representation, natural language generation, document and query representation through vector space embeddings, dialog and conversational AI, and multimodal AI models. Previously, he was on the team that introduced Microsoft’s first ever wearable, Microsoft Band. He lead the application, experiences and developer platform for Microsoft Band and Health. Ali has a bachelor's in computer science from Lahore University of Management Sciences (LUMS) with minors in mathematics and economics. Ali has been at Microsoft since 2001.

    ??

    ?? ???

    人人超碰97caoporen国产