Scribd Logo
Upload

Welcome to Scribd!

  • GPT3

    Uploaded by

    Netlab IT usluge
    24 views11 pages
    The document discusses GPT3, which stands for Generative Pre-trained Transformer 3 and is a large language model developed by OpenAI that uses deep learning to generate human-like text. It provides details on GPT3's specifications including that it has been trained on a massive dataset using 117 billion parameters. Applications that GPT3 can be used for are discussed such as creating resumes, building machine learning models, answering SQL queries, and powering a search engine.

    Original Description:

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document discusses GPT3, which stands for Generative Pre-trained Transformer 3 and is a large language model developed by OpenAI that uses deep learning to generate human-like text. It provides details on GPT3's specifications including that it has been trained on a massive dataset using 117 billion parameters. Applications that GPT3 can be used for are discussed such as creating resumes, building machine learning models, answering SQL queries, and powering a search engine.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    24 views11 pages

    GPT3

    Uploaded by

    Netlab IT usluge
    The document discusses GPT3, which stands for Generative Pre-trained Transformer 3 and is a large language model developed by OpenAI that uses deep learning to generate human-like text. It provides details on GPT3's specifications including that it has been trained on a massive dataset using 117 billion parameters. Applications that GPT3 can be used for are discussed such as creating resumes, building machine learning models, answering SQL queries, and powering a search engine.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 11

    GPT3

    Generative Pre-trained Transformer 3

    Generative Pre-trained Transformer 3


    • What is GPT3?
    • GPT3 Specifications
    • Models Vs Parameters
    WHAT WILL YOU • Datasets used to train GPT3
    • GPT3 Accuracy
    LEARN HERE • Applications of GPT3

    20XX PITCH DECK 2


    GPT3 GPT3 SPECIFICATIONS
    Stands for Generative Pre-trained • GPT3 has been trained with 175B
    WHAT IS Transformer. It’s a 3 rd gen language parameters

    GPT3 model developed by open AI now in • GPT3 has been trained with 45TB
    beta text data which includes sources
    from Wikipedia, books, coding
    tutorials etc
    WHAT CAN IT DO FOR YOU? • 60% of data for pre training GPT3
    model was taken from a common
    GPT3 can understand your
    crawl ( Common Crawl is an
    problem and generate human organization that crawls web
    like text in less than a second and provides data as open
    source to users)
    PRE-TRAINED MODEL • GPT3 has 96 decoder layers and
    Pre-Trained models are large is built on a system with 285K
    CPU cores, 10K GPU’s and 400
    networks trained on massive
    Gbps network connectivity for
    datasets without supervision
    each GPU server
    • Was trained on a Supercomputer
    developed by Open AI and
    Microsoft jointly
    2022 GPT3 3
    MODELS VS BERT
    PARAMETERS Google’s BERT pre-trained
    model uses 110M parameters
    where as BERT-large uses 340M
    parameters

    GPT
    Previous GPT2 model had 1.5B
    parameters where as GPT3 was
    largest among all with 117B
    parameters

    TURNING NLG
    Microsoft’s Turning NLG model
    had 17B parameters

    20XX GPT3 4
    Datasets used to train GPT3

    Dataset Quantity (tokens) Weight in Training Epochs elapsed


    in B mix when training for
    300B tokens
    Common Crawl 410 60% 0.44
    WebText2 19 22% 2.9
    Books1 12 8% 0.43
    Books2 55 8% 0.43
    Wikipedia 3 3% 3.4

    2022 GPT3 5
    GPT3 ACCURACY

    ACCURACY OF MODEL
    INCREASES WITH NUMBER
    OF PARAMETERS

    NO NEED TO DO GRADIENT
    UPDATE OR
    HYPERPARAMETER TURNING

    INTERACT WITH MODEL


    WITH NATURAL LANGUAGE
    OR PROVIDE EXAMPLES OF
    TASK

    20XX GPT3 6
    APPLICATIONS OF GPT3

    CREATING
    SEARCH ENGINE BUILD ML MODELS RESUMES SQL QUERIES
    Inbuilt
    https://chat.openai.c Create resume by Write SQL queries
    Build ML model looking at few lines given a line of text
    om/
    automatically of text based on and table in place
    Or work experience and
    education
    Create a fully background
    functional search
    engine with GPT3
    2022 GPT3 7
    CREATE RESUME

    CREATE RESUME
    SPREAD SHEET FUNCTION
    BUILDING ML
    MODELS

    2022 GPT3 10
    SQL QURIES

    2022 GPT3 11

    You might also like