Why it’s an Exciting Time for NLP-based Startups
The following is a guest post from Sierra Ventures portfolio company, QuillBot.
***
It’s an exciting time to be an NLP-based company. Advances in the field have reduced costs and the time needed for processing and training. Additionally, overall model output quality has increased due to more robust pre-trained models. All of these factors have created a level of accessibility for entrepreneurs and startups previously unseen.Three trends that are driving the advancement of the field include:
- Transfer Learning
- Hardware Advancements
- Increased Accessibility
Transfer LearningTransfer learning has enabled the rise of large, pre-trained language models by leveraging the existing natural language data on the internet. Tasks such as fill-in-the-blank, next sentence prediction, or unscrambling are learned by the model, making it easier for the model to learn another type of task. This had led to the rise of large, pre-trained language models, such as BERT and GPT-3. Once the large models are trained, they can be fine-tuned for more specific applications by new users.Next in the evolution of transfer learning is zero shot learning, which is the ability for a model to complete a task for which it hasn’t been trained. An example of this would be a German translation model which is tasked to translate words into Spanish, without any reference seed examples.These advances are enabling deep learning capabilities or small-scale startup applications. With models like BERT, GPT-3, and T5, startups can iterate without having to build and pre-train models of their own. Hardware AdvancementsRecent advances in hardware make it easier to train existing models due to reduced costs that allows for increased access to GPU RAM and more robust CUDA cores. When coupled with deployments in the cloud, a user can work with larger datasets in a cost-efficient manner while maintaining high output quality.Increased AccessibilityWith companies like OpenAI, Google, and Facebook championing open-source models (with the exception of OpenAI’s GPT-3), there is now ample data available. These tech giants are doing the computational heavy lifting, covering the building and training costs, and then releasing their pre-trained models to the public. Open-source frameworks, like HuggingFace, are becoming increasingly available so that those who want to use BERT, for example, and train it themselves, don’t need to program the underlying model training or inference code. Startups like QuillBot, a company that has used NLP technology to build models to summarize and paraphrase text, can now run, use, and retrain models for their specific needs. These advancements allow for the creation of scalable cutting edge products while allowing for iterating on new ideas and product features. The technological advances in transfer learning, hardware, and accessibility continue to facilitate even the smallest-scale ideas being brought to fruition, by teams both large and small. Sierra Ventures looks forward to meeting the next wave of talented entrepreneurs leveraging NLP technology to build impactful businesses!
Learn more about our investment QuillBot, a company leveraging NLP to improve writing skills.