Introduction to Natural Language Processing (NLP)
This field studies two main aspects:
a) Natural Language Understanding, giving meaning to a combination of letters (serach engines, self driving cars intstructions)
b) Natural Language Generation, which generate language (translations, summaries, chatbots…)
We are able to read because we can assign meaning to symbols. However, how many times have we gotten confused about a meaning because it had a different context than wich we were used to? Or how many times have we said the same sentence with a different tone?
A machine needs to learn meaning of words and our potential mistakes. For this, we use morphology for some words like ‘thinking’ and ‘thinker’, ‘swim’ and swimmer’. For the rest, we can use distributional semantics, which looks into which words appear in the same sentence.
Nevertheless… how will the computer understand these words? We need to translate these into math! How? Through count vectors:
It counts the number of times a same word appears in an article or sentence. An example provided by Jabril Ashe are, for example, Car, Cat, and Felidae. The stop words (a, in, the, of, and…) can change the meaning of some words and usually don’t mean much by themselves.
A problem with this method is that we need a massive dataset, which is not realistic. The ideal scenario is to ‘learn a representation for words that captures all the same relationships and similarities as count vectors in a much more compact way.’
An Encoder-Decoder Model allows us to build internal representation and make predictions with words. The encoder says what we can think and remember from the text and the decoder says what we can do with that data.
Language Model
The example provided by Jabril Ashe is ‘I’m hungry, I’d like chocolate ____’ where he tries to make the machine guess what to place in the blank space.
The encoder might be solely focusing on the word chocolate, which then uses to access its cluster of ‘chocolate-related words’ to place one there.
Another example includes: ‘Diana, my friend from CA, loves physics and is having a birthday party next week, I want to find a present for _____’
What might be the problem with this? We talked about Diana 27 words before the blank and this name can be replaces by ‘her’, hence, we want an encoder that can capture these pieces of information and allow the decoder to place the correct word.
However, there’s a limit to how much a model can remember. We cannot fit the whole meaning of a sentence into one vector, at least just yet.
Let’s try now to predict the next word through a neural network. Our data will be gathered from texts. Each word in the sentence can be used in the fill-in-the-blank game and train the model to encode up to the blank to predict the word that goes there.
To define the encoder, we will use a Recurrent Neural Network since it has a loop that reuses a single hidden layer, which gets updated as the model reads. Since we’re working with words and not images, we cannot assign pixel values to them. To solve this, we train the model through unsupervised learning to learn the representation of each word.
We assign each word a random list of numbers — or vector. The encoder will take these and put them into a single one.
The decoder will then be a single hidden layer that outputs a score for every word after analysing the sentece representation vector.
It will compare the score to every word in our vocabulary and that with the highest score will be selected — see image below, Messi.
Now you know how predicting the next word works in your gmail but this is just the beginning. The way in which, for example, Google Translate works is much more complicated since it has a translation system.
If we use the vectors and make a plot, we can visualize word representations. The representation of words from one model might not work for another. Since we use language for everything, this field is truly exciting.
You can access the course in which this article is based on here:
You can follow me on LinkedIn here:
Read more about technolgy here:
