forked from piskvorky/gensim
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathtut3.html
More file actions
306 lines (275 loc) · 21.3 KB
/
Copy pathtut3.html
File metadata and controls
306 lines (275 loc) · 21.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Similarity Queries — gensim</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '0.8.6',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="author" title="About these documents" href="about.html" />
<link rel="top" title="gensim" href="index.html" />
<link rel="up" title="Tutorials" href="tutorial.html" />
<link rel="next" title="Experiments on the English Wikipedia" href="wiki.html" />
<link rel="prev" title="Topics and Transformations" href="tut2.html" />
<!-- twitter search widget
<script type="text/javascript" src="_static/widget.js"></script>
-->
<meta property="og:title" content="#gensim" />
<meta property="og:description" content="Efficient topic modelling in Python" />
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-24066335-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="wiki.html" title="Experiments on the English Wikipedia"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="tut2.html" title="Topics and Transformations"
accesskey="P">previous</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
<li><a href="tutorial.html" accesskey="U">Tutorials</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Similarity Queries</a><ul>
<li><a class="reference internal" href="#similarity-interface">Similarity interface</a><ul>
<li><a class="reference internal" href="#initializing-query-structures">Initializing query structures</a></li>
<li><a class="reference internal" href="#performing-queries">Performing queries</a></li>
</ul>
</li>
<li><a class="reference internal" href="#where-next">Where next?</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="tut2.html"
title="previous chapter">Topics and Transformations</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="wiki.html"
title="next chapter">Experiments on the English Wikipedia</a></p>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="24" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="similarity-queries">
<span id="tut3"></span><h1>Similarity Queries<a class="headerlink" href="#similarity-queries" title="Permalink to this headline">¶</a></h1>
<p>Don’t forget to set</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">logging</span>
<span class="gp">>>> </span><span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(asctime)s</span><span class="s"> : </span><span class="si">%(levelname)s</span><span class="s"> : </span><span class="si">%(message)s</span><span class="s">'</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
</pre></div>
</div>
<p>if you want to see logging events.</p>
<div class="section" id="similarity-interface">
<h2>Similarity interface<a class="headerlink" href="#similarity-interface" title="Permalink to this headline">¶</a></h2>
<p>In the previous tutorials on <a class="reference internal" href="tut1.html"><em>Corpora and Vector Spaces</em></a> and <a class="reference internal" href="tut2.html"><em>Topics and Transformations</em></a>, we covered what it means
to create a corpus in the Vector Space Model and how to transform it between different
vector spaces. A common reason for such a charade is that we want to determine
<strong>similarity between pairs of documents</strong>, or the <strong>similarity between a specific document
and a set of other documents</strong> (such as a user query vs. indexed documents).</p>
<p>To show how this can be done in gensim, let us consider the same corpus as in the
previous examples (which really originally comes from Deerwester et al.’s
<a class="reference external" href="http://www.cs.bham.ac.uk/~pxt/IDA/lsa_ind.pdf">“Indexing by Latent Semantic Analysis”</a>
seminal 1990 article):</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">gensim</span> <span class="kn">import</span> <span class="n">corpora</span><span class="p">,</span> <span class="n">models</span><span class="p">,</span> <span class="n">similarities</span>
<span class="gp">>>> </span><span class="n">dictionary</span> <span class="o">=</span> <span class="n">corpora</span><span class="o">.</span><span class="n">Dictionary</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s">'/tmp/deerwester.dict'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="n">corpora</span><span class="o">.</span><span class="n">MmCorpus</span><span class="p">(</span><span class="s">'/tmp/deerwester.mm'</span><span class="p">)</span> <span class="c"># comes from the first tutorial, "From strings to vectors"</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">corpus</span>
<span class="go">MmCorpus(9 documents, 12 features, 28 non-zero entries)</span>
</pre></div>
</div>
<p>To follow Deerwester’s example, we first use this tiny corpus to define a 2-dimensional
LSI space:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">lsi</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">LsiModel</span><span class="p">(</span><span class="n">corpus</span><span class="p">,</span> <span class="n">id2word</span><span class="o">=</span><span class="n">dictionary</span><span class="p">,</span> <span class="n">num_topics</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
</pre></div>
</div>
<p>Now suppose a user typed in the query <cite>“Human computer interaction”</cite>. We would
like to sort our nine corpus documents in decreasing order of relevance to this query.
Unlike modern search engines, here we only concentrate on a single aspect of possible
similarities—on apparent semantic relatedness of their texts (words). No hyperlinks,
no random-walk static ranks, just a semantic extension over the boolean keyword match:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">doc</span> <span class="o">=</span> <span class="s">"Human computer interaction"</span>
<span class="gp">>>> </span><span class="n">vec_bow</span> <span class="o">=</span> <span class="n">dictionary</span><span class="o">.</span><span class="n">doc2bow</span><span class="p">(</span><span class="n">doc</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">())</span>
<span class="gp">>>> </span><span class="n">vec_lsi</span> <span class="o">=</span> <span class="n">lsi</span><span class="p">[</span><span class="n">vec_bow</span><span class="p">]</span> <span class="c"># convert the query to LSI space</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">vec_lsi</span>
<span class="go">[(0, -0.461821), (1, 0.070028)]</span>
</pre></div>
</div>
<p>In addition, we will be considering <a class="reference external" href="http://en.wikipedia.org/wiki/Cosine_similarity">cosine similarity</a>
to determine the similarity of two vectors. Cosine similarity is a standard measure
in Vector Space Modeling, but wherever the vectors represent probability distributions,
<a class="reference external" href="http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Symmetrised_divergence">different similarity measures</a>
may be more appropriate.</p>
<div class="section" id="initializing-query-structures">
<h3>Initializing query structures<a class="headerlink" href="#initializing-query-structures" title="Permalink to this headline">¶</a></h3>
<p>To prepare for similarity queries, we need to enter all documents which we want
to compare against subsequent queries. In our case, they are the same nine documents
used for training LSI, converted to 2-D LSA space. But that’s only incidental, we
might also be indexing a different corpus altogether.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">index</span> <span class="o">=</span> <span class="n">similarities</span><span class="o">.</span><span class="n">MatrixSimilarity</span><span class="p">(</span><span class="n">lsi</span><span class="p">[</span><span class="n">corpus</span><span class="p">])</span> <span class="c"># transform corpus to LSI space and index it</span>
</pre></div>
</div>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">The class <tt class="xref py py-class docutils literal"><span class="pre">similarities.MatrixSimilarity</span></tt> is only appropriate when the whole
set of vectors fits into memory. For example, a corpus of one million documents
would require 2GB of RAM in a 256-dimensional LSI space, when used with this class.
Without 2GB of free RAM, you would need to use the <tt class="xref py py-class docutils literal"><span class="pre">similarities.Similarity</span></tt> class.
This class operates in fixed memory, by splitting the index across multiple files on disk.
It uses <tt class="xref py py-class docutils literal"><span class="pre">similarities.MatrixSimilarity</span></tt> and <tt class="xref py py-class docutils literal"><span class="pre">similarities.SparseMatrixSimilarity</span></tt> internally,
so it is still fast, although slightly more complex.</p>
</div>
<p>Index persistency is handled via the standard <tt class="xref py py-func docutils literal"><span class="pre">save()</span></tt> and <tt class="xref py py-func docutils literal"><span class="pre">load()</span></tt> functions:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">index</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s">'/tmp/deerwester.index'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">index</span> <span class="o">=</span> <span class="n">similarities</span><span class="o">.</span><span class="n">MatrixSimilarity</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s">'/tmp/deerwester.index'</span><span class="p">)</span>
</pre></div>
</div>
<p>This is true for all similarity indexing classes (<tt class="xref py py-class docutils literal"><span class="pre">similarities.Similarity</span></tt>,
<tt class="xref py py-class docutils literal"><span class="pre">similarities.MatrixSimilarity</span></tt> and <tt class="xref py py-class docutils literal"><span class="pre">similarities.SparseMatrixSimilarity</span></tt>).
Also in the following, <cite>index</cite> can be an object of any of these. When in doubt,
use <tt class="xref py py-class docutils literal"><span class="pre">similarities.Similarity</span></tt>, as it is the most scalable version, and it also
supports adding more documents to the index later.</p>
</div>
<div class="section" id="performing-queries">
<h3>Performing queries<a class="headerlink" href="#performing-queries" title="Permalink to this headline">¶</a></h3>
<p>To obtain similarities of our query document against the nine indexed documents:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">sims</span> <span class="o">=</span> <span class="n">index</span><span class="p">[</span><span class="n">vec_lsi</span><span class="p">]</span> <span class="c"># perform a similarity query against the corpus</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="nb">list</span><span class="p">(</span><span class="nb">enumerate</span><span class="p">(</span><span class="n">sims</span><span class="p">))</span> <span class="c"># print (document_number, document_similarity) 2-tuples</span>
<span class="go">[(0, 0.99809301), (1, 0.93748635), (2, 0.99844527), (3, 0.9865886), (4, 0.90755945),</span>
<span class="go">(5, -0.12416792), (6, -0.1063926), (7, -0.098794639), (8, 0.05004178)]</span>
</pre></div>
</div>
<p>Cosine measure returns similarities in the range <cite><-1, 1></cite> (the greater, the more similar),
so that the first document has a score of 0.99809301 etc.</p>
<p>With some standard Python magic we sort these similarities into descending
order, and obtain the final answer to the query <cite>“Human computer interaction”</cite>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">sims</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">enumerate</span><span class="p">(</span><span class="n">sims</span><span class="p">),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="o">-</span><span class="n">item</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">sims</span> <span class="c"># print sorted (document number, similarity score) 2-tuples</span>
<span class="go">[(2, 0.99844527), # The EPS user interface management system</span>
<span class="go">(0, 0.99809301), # Human machine interface for lab abc computer applications</span>
<span class="go">(3, 0.9865886), # System and human system engineering testing of EPS</span>
<span class="go">(1, 0.93748635), # A survey of user opinion of computer system response time</span>
<span class="go">(4, 0.90755945), # Relation of user perceived response time to error measurement</span>
<span class="go">(8, 0.050041795), # Graph minors A survey</span>
<span class="go">(7, -0.098794639), # Graph minors IV Widths of trees and well quasi ordering</span>
<span class="go">(6, -0.1063926), # The intersection graph of paths in trees</span>
<span class="go">(5, -0.12416792)] # The generation of random binary unordered trees</span>
</pre></div>
</div>
<p>(I added the original documents in their “string form” to the output comments, to
improve clarity.)</p>
<p>The thing to note here is that documents no. 2 (<tt class="docutils literal"><span class="pre">"The</span> <span class="pre">EPS</span> <span class="pre">user</span> <span class="pre">interface</span> <span class="pre">management</span> <span class="pre">system"</span></tt>)
and 4 (<tt class="docutils literal"><span class="pre">"Relation</span> <span class="pre">of</span> <span class="pre">user</span> <span class="pre">perceived</span> <span class="pre">response</span> <span class="pre">time</span> <span class="pre">to</span> <span class="pre">error</span> <span class="pre">measurement"</span></tt>) would never be returned by
a standard boolean fulltext search, because they do not share any common words with <tt class="docutils literal"><span class="pre">"Human</span>
<span class="pre">computer</span> <span class="pre">interaction"</span></tt>. However, after applying LSI, we can observe that both of
them received quite high similarity scores (no. 2 is actually the most similar!),
which corresponds better to our intuition of
them sharing a “computer-human” related topic with the query. In fact, this semantic
generalization is the reason why we apply transformations and do topic modelling
in the first place.</p>
</div>
</div>
<div class="section" id="where-next">
<h2>Where next?<a class="headerlink" href="#where-next" title="Permalink to this headline">¶</a></h2>
<p>Congratulations, you have finished the tutorials – now you know how gensim works :-)
To delve into more details, you can browse through the <a class="reference internal" href="apiref.html"><em>API documentation</em></a>,
see the <a class="reference internal" href="wiki.html"><em>Wikipedia experiments</em></a> or perhaps check out <a class="reference internal" href="distributed.html"><em>distributed computing</em></a> in <cite>gensim</cite>.</p>
<p>Please remember that gensim is an experimental package, aimed at the NLP research community.
This means that:</p>
<ul class="simple">
<li>there certainly are parts that could be implemented more efficiently (in C, for example), and there may also be bugs in the code</li>
<li>your <strong>feedback is most welcome</strong> and appreciated, be it in code and
<a class="reference external" href="https://github.com/piskvorky/gensim/wiki/Ideas-&-Features-proposals">idea contributions</a>,
<a class="reference external" href="https://github.com/piskvorky/gensim/issues">bug reports</a> or just
<a class="reference external" href="http://groups.google.com/group/gensim/topics">user stories and general questions</a>.</li>
</ul>
<p>Gensim has no ambition to become an all-encompassing production level tool, with robust failure handling
and error recoveries. Its main goal is to help NLP newcomers try out popular algorithms
and to facilitate prototyping of new algorithms for NLP researchers.</p>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="wiki.html" title="Experiments on the English Wikipedia"
>next</a> |</li>
<li class="right" >
<a href="tut2.html" title="Topics and Transformations"
>previous</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
<li><a href="tutorial.html" >Tutorials</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>.
Last updated on Sep 16, 2012.
</div>
</body>
</html>