Columbia University自然语言处理课程作业1（代码实现）

pg2482@columbia.edu Time it takes programs to complete: Part A takes 1m 29s to complete. Part B takes 10m 22s to complete. == Part A #2 Perplexity measurements == $ python perplexity.py A2.uni.txt Brown_train.txt The perplexity is 1104.83292814 $ python perplexity.py A2.bi.txt Brown_train.txt The perplexity is 57.2215464238 $ python perplexity.py A2.tri.txt Brown_train.txt The perplexity is 5.89521267642 == Part A #3 Perplexity == $ python perplexity.py A3.txt Brown_train.txt The perplexity is 13.0759217039 == Part A #4 Performance comparison == With linear interpolation, we got a significantly better fit than the bigram model by itself. However, perplexity wasn't as low as the trigram model by itself. == Part A #5 Sample sentences perplexity == $ python perplexity.py Sample1_scored.txt Sample1.txt The perplexity is 11.6492786046 $ python perplexity.py Sample2_scored.txt Sample2.txt The perplexity is 1611241155.03 Sample1 belongs to the Brown dataset because the perplexity value is much closer to 1. So, the distribution of n-grams in Sample1 is very close to the distribution of n-grams in the training data.