Lecture 1 Introduction and Document Distance 6.006 Spring 2008
Document Distance in Practice
Computing Document Distance: docdist1.py
The python
code and results relevant to this section are available here. This program com-
putes the
distance between 2 documents by performing the following steps:
Read file
•
• Make
word list [“the”,“year”,. . . ]
• Count
frequencies [[“the”,4012],[“year”,55],. . . ]
• Sort into
order [[“a”,3120],[“after”,17],. . . ]
• Compute θ
Ideally,
we would like to run this program to compute document distances between writings
of the following authors:
Jules Verne
- document size 25k •
• Bobsey Twins
- document size 268k
Lewis and
Clark - document size 1M •
• Shakespeare
- document size 5.5M
Churchill
- document size 10M •
Experiment:
Comparing the Bobsey and Lewis documents with docdist1.py gives θ = 0.574.
However, it takes approximately 3 minutes to compute this document distance, and probably
gets slower as the inputs get large.
What is wrong with the efficiency of this program?
Is it a Python vs. C issue? Is it a choice of algorithm issue - θ(n
2
) versus θ(n)?
Profiling: docdist2.py
In order
to figure out why our initial program is so slow, we now “instrument” the program
so that Python will tell us where the running time is going. This can be done simply using
the profile module in Python. The profile module indicates how much time is spent in each
routine.
(See this
link for details on profile).
The profile module is imported into docdist1.py and the end of the docdist1.py file is
modified. The modified docdist1.py file is renamed as docdist2.py
Detailed results
of document comparisons are available here
3
.
5星 · 超过95%的资源 需积分: 13 119 浏览量
2017-03-08
20:54:25
上传
评论 1
收藏 18.19MB PDF 举报 
wangbo_thu2018-07-21很不错，谢谢
