On the 11th of June, 2020, the world was not ready for OpenAI to release a beta version of their latest language training model, GPT-3. Here’s a quick tour of this powerful language model.
What is GPT-3?
GPT-3, or Generative Pre-trained Transformer 3 is the latest breakthrough in language generators. It uses deep learning technologies to generate human-like texts. It’s the largest language model to exist so far, with over 175 billion parameters. In comparison, its predecessor, GPT-2 has only 1.5 billion parameters.
Here’s a graph comparing the number of parameters of other pre-trained NLP (Natural Language Processing) models. The number of parameters GPT-3 can handle compared to other popular language models
How does GPT-3 work?
With over 175 billion parameters, it’s no surprise that GPT-3 is one of the most sophisticated text predictors to date.
But how is this possible?
Let’s go back to basics for a second. Remember when we first got introduced to transformer models back in 2017? They essentially used a deep learning technique called neural networks, to handle natural language data for translations and text summarisations. This transformer became the standard for all language generators, so GPT-3 followed suit. GPT-3 models are based on the same deep-learning transformer-based neural network architecture.
It’s the same standard, but it’s larger. And is pre-trained by an abundance of resources via datasets, such as Common Crawl, Wikipedia, WebText2, Books1, and Books2. The amount of training removes the need for human intervention and saves a lot of time. It’s like having the entire internet’s history in your generator.
Every article, every resource, every piece of information, all in on one powerful machine.
And with an abundance of information, GPT-3 can generate the most statistically likely responsible for any given input, based on what it has learned from the internet. Rich insights can be extracted from patterns in large data sets, and it can identify and apply linguistic patterns at a speed beyond what any human can do. Think of GPT-3 as a sort of virtual assistant.
What can GPT-3 accomplish?
The short answer is a lot. But here’s a narrow list of what we believe GPT-3 can do.
It could possibly write creative fiction researchers found that GPT-3's creative work samples were creative, thought-provoking, and were stylish, in a literary sense.
It could make chatbot technology more sophisticated what makes GPT-3 stand out is that it requires minimal context and input. After feeding the generator input, GPT-3 best guesses what the next chunk of text should be. It then repeats this process- taking the original input, plus the newly generated chunk, and subsequently generating another chunk of text. It makes it the perfect virtual assistant for customers and employees.
GPT-3 is currently in beta, but we at piMonk are excited about how things will evolve in the future, so stay tuned with us! Also, read our blog on understanding git just got easier.