cs224d assignment1

CS 224D: Deep Learning for NLP

1

1

Course Instructor: Richard Socher

Lecture Notes: Part I

2

2

Authors: Francois Chaubard, Rohit

Mundra, Richard Socher

Spring 2016

Keyphrases: Natural Language Processing. Word Vectors. Singu-

word vectors.

1 Introduction to Natural Language Processing

Natural Language Processing tasks

come in varying levels of difficulty:

Easy

• Spell Checking

• Keyword Search

• Finding Synonyms

Medium

• Parsing information from websites,

We begin with a general discussion of what is NLP. The goal of NLP

is to be able to design algorithms to allow computers to "understand"

natural language in order to perform some task. Example tasks come

in varying level of difficulty:

Easy

• Spell Checking

• Keyword Search

• Finding Synonyms

Medium

• Parsing information from websites, documents, etc.

The first and arguably most important common denominator

across all NLP tasks is how we represent words as input to any and

all of our models. Much of the earlier NLP work that we will not

cover treats words as atomic symbols. To perform well on most NLP

tasks we first need to have some notion of similarity and difference

cs 224d: deep learning for nlp 2

between words. With word vectors, we can quite easily encode this

ability in the vectors themselves (using distance measures such as

Jaccard, Cosine, Euclidean, etc).

2 Word Vectors

our language. Each dimension would encode some meaning that

we transfer using speech. For instance, semantic dimensions might

indicate tense (past vs. present vs. future), count (singular vs. plural),

and gender (masculine vs. feminine). One-hot vector: Represent every word

as an R

1 at the index of that word in the sorted

english language.

So let’s dive into our first word vector and arguably the most

1

0

0

from digital circuit design, meaning "a

group of bits among which the legal

combinations of values are only those

with a single high (1) bit and all the

others low (0)".

We represent each word as a completely independent entity. As

we previously discussed, this word representation does not give us

directly any notion of similarity. For instance,

as word vectors), we first loop over a massive dataset and accumu-

decom-

cs 224d: deep learning for nlp 3

As our first attempt, we make the bold conjecture that words that

are related will often appear in the same documents. For instance,

"banks", "bonds", "stocks", "money", etc. are probably likely to ap-

pear together. But "banks", "octopus", "banana", and "hockey" would

haps we can try something better.

co-occurrences of words thereby becoming an affinity matrix. In this

method we count the number of times each word appears inside a

window of a particular size around the word of interest. We calculate

this count for all the words in corpus. We display an example below.

Let our corpus contain just three sentences and the window size be 1: Using Word-Word Co-occurrence

Matrix:

•