forked from piskvorky/gensim
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathwiki.html
More file actions
368 lines (330 loc) · 30.2 KB
/
Copy pathwiki.html
File metadata and controls
368 lines (330 loc) · 30.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Experiments on the English Wikipedia — gensim</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '0.8.6',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="author" title="About these documents" href="about.html" />
<link rel="top" title="gensim" href="index.html" />
<link rel="next" title="API Reference" href="apiref.html" />
<link rel="prev" title="Distributed Latent Dirichlet Allocation" href="dist_lda.html" />
<!-- twitter search widget
<script type="text/javascript" src="_static/widget.js"></script>
-->
<meta property="og:title" content="#gensim" />
<meta property="og:description" content="Efficient topic modelling in Python" />
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-24066335-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="apiref.html" title="API Reference"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="dist_lda.html" title="Distributed Latent Dirichlet Allocation"
accesskey="P">previous</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Experiments on the English Wikipedia</a><ul>
<li><a class="reference internal" href="#preparing-the-corpus">Preparing the corpus</a></li>
<li><a class="reference internal" href="#latent-sematic-analysis">Latent Sematic Analysis</a></li>
<li><a class="reference internal" href="#latent-dirichlet-allocation">Latent Dirichlet Allocation</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="dist_lda.html"
title="previous chapter">Distributed Latent Dirichlet Allocation</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="apiref.html"
title="next chapter">API Reference</a></p>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="24" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="experiments-on-the-english-wikipedia">
<span id="wiki"></span><h1>Experiments on the English Wikipedia<a class="headerlink" href="#experiments-on-the-english-wikipedia" title="Permalink to this headline">¶</a></h1>
<p>To test <cite>gensim</cite> performance, we run it against the English version of Wikipedia.</p>
<p>This page describes the process of obtaining and processing Wikipedia, so that
anyone can reproduce the results. It is assumed you have <cite>gensim</cite> properly <a class="reference internal" href="install.html"><em>installed</em></a>.</p>
<div class="section" id="preparing-the-corpus">
<h2>Preparing the corpus<a class="headerlink" href="#preparing-the-corpus" title="Permalink to this headline">¶</a></h2>
<ol class="arabic">
<li><p class="first">First, download the dump of all Wikipedia articles from <a class="reference external" href="http://download.wikimedia.org/enwiki/">http://download.wikimedia.org/enwiki/</a>
(you want a file like <cite>enwiki-latest-pages-articles.xml.bz2</cite>). This file is about 8GB in size
and contains (a compressed version of) all articles from the English Wikipedia.</p>
</li>
<li><p class="first">Convert the articles to plain text (process Wiki markup) and store the result as
sparse TF-IDF vectors. In Python, this is easy to do on-the-fly and we don’t
even need to uncompress the whole archive to disk. There is a script included in
<cite>gensim</cite> that does just that, run:</p>
<div class="highlight-python"><pre>$ python -m gensim.scripts.make_wiki</pre>
</div>
</li>
</ol>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>This pre-processing step makes two passes over the 8.2GB compressed wiki dump (one to extract
the dictionary, one to create and store the sparse vectors) and takes about
9 hours on my laptop, so you may want to go have a coffee or two.</p>
<p class="last">Also, you will need about 35GB of free disk space to store the sparse output vectors.
I recommend compressing these files immediately, e.g. with bzip2 (down to ~13GB). Gensim
can work with compressed files directly, so this lets you save disk space.</p>
</div>
</div>
<div class="section" id="latent-sematic-analysis">
<h2>Latent Sematic Analysis<a class="headerlink" href="#latent-sematic-analysis" title="Permalink to this headline">¶</a></h2>
<p>First let’s load the corpus iterator and dictionary, created in the second step above:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">logging</span><span class="o">,</span> <span class="nn">gensim</span><span class="o">,</span> <span class="nn">bz2</span>
<span class="gp">>>> </span><span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(asctime)s</span><span class="s"> : </span><span class="si">%(levelname)s</span><span class="s"> : </span><span class="si">%(message)s</span><span class="s">'</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># load id->word mapping (the dictionary), one of the results of step 2 above</span>
<span class="gp">>>> </span><span class="n">id2word</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">corpora</span><span class="o">.</span><span class="n">Dictionary</span><span class="o">.</span><span class="n">load_from_text</span><span class="p">(</span><span class="s">'wiki_en_wordids.txt'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># load corpus iterator</span>
<span class="gp">>>> </span><span class="n">mm</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">corpora</span><span class="o">.</span><span class="n">MmCorpus</span><span class="p">(</span><span class="s">'wiki_en_tfidf.mm'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output (recommended)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">mm</span>
<span class="go">MmCorpus(3931787 documents, 100000 features, 756379027 non-zero entries)</span>
</pre></div>
</div>
<p>We see that our corpus contains 3.9M documents, 100K features (distinct
tokens) and 0.76G non-zero entries in the sparse TF-IDF matrix. The Wikipedia corpus
contains about 2.24 billion tokens in total.</p>
<p>Now we’re ready to compute LSA of the English Wikipedia:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="c"># extract 400 LSI topics; use the default one-pass algorithm</span>
<span class="gp">>>> </span><span class="n">lsi</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">models</span><span class="o">.</span><span class="n">lsimodel</span><span class="o">.</span><span class="n">LsiModel</span><span class="p">(</span><span class="n">corpus</span><span class="o">=</span><span class="n">mm</span><span class="p">,</span> <span class="n">id2word</span><span class="o">=</span><span class="n">id2word</span><span class="p">,</span> <span class="n">num_topics</span><span class="o">=</span><span class="mi">400</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># print the most contributing words (both positively and negatively) for each of the first ten topics</span>
<span class="gp">>>> </span><span class="n">lsi</span><span class="o">.</span><span class="n">print_topics</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="go">topic #0(332.762): 0.425*"utc" + 0.299*"talk" + 0.293*"page" + 0.226*"article" + 0.224*"delete" + 0.216*"discussion" + 0.205*"deletion" + 0.198*"should" + 0.146*"debate" + 0.132*"be"</span>
<span class="go">topic #1(201.852): 0.282*"link" + 0.209*"he" + 0.145*"com" + 0.139*"his" + -0.137*"page" + -0.118*"delete" + 0.114*"blacklist" + -0.108*"deletion" + -0.105*"discussion" + 0.100*"diff"</span>
<span class="go">topic #2(191.991): -0.565*"link" + -0.241*"com" + -0.238*"blacklist" + -0.202*"diff" + -0.193*"additions" + -0.182*"users" + -0.158*"coibot" + -0.136*"user" + 0.133*"he" + -0.130*"resolves"</span>
<span class="go">topic #3(141.284): -0.476*"image" + -0.255*"copyright" + -0.245*"fair" + -0.225*"use" + -0.173*"album" + -0.163*"cover" + -0.155*"resolution" + -0.141*"licensing" + 0.137*"he" + -0.121*"copies"</span>
<span class="go">topic #4(130.909): 0.264*"population" + 0.246*"age" + 0.243*"median" + 0.213*"income" + 0.195*"census" + -0.189*"he" + 0.184*"households" + 0.175*"were" + 0.167*"females" + 0.166*"males"</span>
<span class="go">topic #5(120.397): 0.304*"diff" + 0.278*"utc" + 0.213*"you" + -0.171*"additions" + 0.165*"talk" + -0.159*"image" + 0.159*"undo" + 0.155*"www" + -0.152*"page" + 0.148*"contribs"</span>
<span class="go">topic #6(115.414): -0.362*"diff" + -0.203*"www" + 0.197*"you" + -0.180*"undo" + -0.180*"kategori" + 0.164*"users" + 0.157*"additions" + -0.150*"contribs" + -0.139*"he" + -0.136*"image"</span>
<span class="go">topic #7(111.440): 0.429*"kategori" + 0.276*"categoria" + 0.251*"category" + 0.207*"kategorija" + 0.198*"kategorie" + -0.188*"diff" + 0.163*"категория" + 0.153*"categoría" + 0.139*"kategoria" + 0.133*"categorie"</span>
<span class="go">topic #8(109.907): 0.385*"album" + 0.224*"song" + 0.209*"chart" + 0.204*"band" + 0.169*"released" + 0.151*"music" + 0.142*"diff" + 0.141*"vocals" + 0.138*"she" + 0.132*"guitar"</span>
<span class="go">topic #9(102.599): -0.237*"league" + -0.214*"he" + -0.180*"season" + -0.174*"football" + -0.166*"team" + 0.159*"station" + -0.137*"played" + -0.131*"cup" + 0.131*"she" + -0.128*"utc"</span>
</pre></div>
</div>
<p>Creating the LSI model of Wikipedia takes about 4 hours and 9 minutes on my laptop <a class="footnote-reference" href="#id6" id="id1">[1]</a>.
That’s about <strong>16,000 documents per minute, including all I/O</strong>.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If you need your results even faster, see the tutorial on <a class="reference internal" href="distributed.html"><em>Distributed Computing</em></a>. Note
that the BLAS libraries inside <cite>gensim</cite> make use of multiple cores transparently, so the same data
will be processed faster on a multicore machine “for free”, without any distributed setup.</p>
</div>
<p>We see that the total processing time is dominated by the preprocessing step of
preparing the TF-IDF corpus from a raw Wikipedia XML dump, which took 9h. <a class="footnote-reference" href="#id7" id="id2">[2]</a></p>
<p>The algorithm used in <cite>gensim</cite> only needs to see each input document once, so it
is suitable for environments where the documents come as a non-repeatable stream,
or where the cost of storing/iterating over the corpus multiple times is too high.</p>
</div>
<div class="section" id="latent-dirichlet-allocation">
<h2>Latent Dirichlet Allocation<a class="headerlink" href="#latent-dirichlet-allocation" title="Permalink to this headline">¶</a></h2>
<p>As with Latent Semantic Analysis above, first load the corpus iterator and dictionary:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">logging</span><span class="o">,</span> <span class="nn">gensim</span><span class="o">,</span> <span class="nn">bz2</span>
<span class="gp">>>> </span><span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(asctime)s</span><span class="s"> : </span><span class="si">%(levelname)s</span><span class="s"> : </span><span class="si">%(message)s</span><span class="s">'</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># load id->word mapping (the dictionary), one of the results of step 2 above</span>
<span class="gp">>>> </span><span class="n">id2word</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">corpora</span><span class="o">.</span><span class="n">Dictionary</span><span class="o">.</span><span class="n">load_from_text</span><span class="p">(</span><span class="s">'wiki_en_wordids.txt'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># load corpus iterator</span>
<span class="gp">>>> </span><span class="n">mm</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">corpora</span><span class="o">.</span><span class="n">MmCorpus</span><span class="p">(</span><span class="s">'wiki_en_tfidf.mm'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c"># mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">mm</span>
<span class="go">MmCorpus(3931787 documents, 100000 features, 756379027 non-zero entries)</span>
</pre></div>
</div>
<p>We will run online LDA (see Hoffman et al. <a class="footnote-reference" href="#id8" id="id3">[3]</a>), which is an algorithm that takes a chunk of documents,
updates the LDA model, takes another chunk, updates the model etc. Online LDA can be contrasted
with batch LDA, which processes the whole corpus (one full pass), then updates
the model, then another pass, another update... The difference is that given a
reasonably stationary document stream (not much topic drift), the online updates
over the smaller chunks (subcorpora) are pretty good in themselves, so that the
model estimation converges faster. As a result, we will perhaps only need a single full
pass over the corpus: if the corpus has 3 million articles, and we update once after
every 10,000 articles, this means we will have done 300 updates in one pass, quite likely
enough to have a very accurate topics estimate:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="c"># extract 100 LDA topics, using 1 pass and updating once every 1 chunk (10,000 documents)</span>
<span class="gp">>>> </span><span class="n">lda</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">models</span><span class="o">.</span><span class="n">ldamodel</span><span class="o">.</span><span class="n">LdaModel</span><span class="p">(</span><span class="n">corpus</span><span class="o">=</span><span class="n">mm</span><span class="p">,</span> <span class="n">id2word</span><span class="o">=</span><span class="n">id2word</span><span class="p">,</span> <span class="n">num_topics</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">update_every</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">chunksize</span><span class="o">=</span><span class="mi">10000</span><span class="p">,</span> <span class="n">passes</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="go">using serial LDA version on this node</span>
<span class="go">running online LDA training, 100 topics, 1 passes over the supplied corpus of 3931787 documents, updating model once every 10000 documents</span>
<span class="gp">...</span>
</pre></div>
</div>
<p>Unlike LSA, the topics coming from LDA are easier to interpret:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="c"># print the most contributing words for 20 randomly selected topics</span>
<span class="gp">>>> </span><span class="n">lda</span><span class="o">.</span><span class="n">print_topics</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="go">topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain + 0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam</span>
<span class="go">topic #1: 0.026*relay + 0.026*athletics + 0.025*metres + 0.023*freestyle + 0.022*hurdles + 0.020*ret + 0.017*divisão + 0.017*athletes + 0.016*bundesliga + 0.014*medals</span>
<span class="go">topic #2: 0.002*were + 0.002*he + 0.002*court + 0.002*his + 0.002*had + 0.002*law + 0.002*government + 0.002*police + 0.002*patrolling + 0.002*their</span>
<span class="go">topic #3: 0.040*courcelles + 0.035*centimeters + 0.023*mattythewhite + 0.021*wine + 0.019*stamps + 0.018*oko + 0.017*perennial + 0.014*stubs + 0.012*ovate + 0.011*greyish</span>
<span class="go">topic #4: 0.039*al + 0.029*sysop + 0.019*iran + 0.015*pakistan + 0.014*ali + 0.013*arab + 0.010*islamic + 0.010*arabic + 0.010*saudi + 0.010*muhammad</span>
<span class="go">topic #5: 0.020*copyrighted + 0.020*northamerica + 0.014*uncopyrighted + 0.007*rihanna + 0.005*cloudz + 0.005*knowles + 0.004*gaga + 0.004*zombie + 0.004*wigan + 0.003*maccabi</span>
<span class="go">topic #6: 0.061*israel + 0.056*israeli + 0.030*sockpuppet + 0.025*jerusalem + 0.025*tel + 0.023*aviv + 0.022*palestinian + 0.019*ifk + 0.016*palestine + 0.014*hebrew</span>
<span class="go">topic #7: 0.015*melbourne + 0.014*rovers + 0.013*vfl + 0.012*australian + 0.012*wanderers + 0.011*afl + 0.008*dinamo + 0.008*queensland + 0.008*tracklist + 0.008*brisbane</span>
<span class="go">topic #8: 0.011*film + 0.007*her + 0.007*she + 0.004*he + 0.004*series + 0.004*his + 0.004*episode + 0.003*films + 0.003*television + 0.003*best</span>
<span class="go">topic #9: 0.019*wrestling + 0.013*château + 0.013*ligue + 0.012*discus + 0.012*estonian + 0.009*uci + 0.008*hockeyarchives + 0.008*wwe + 0.008*estonia + 0.007*reign</span>
<span class="go">topic #10: 0.078*edits + 0.059*notability + 0.035*archived + 0.025*clearer + 0.022*speedy + 0.021*deleted + 0.016*hook + 0.015*checkuser + 0.014*ron + 0.011*nominator</span>
<span class="go">topic #11: 0.013*admins + 0.009*acid + 0.009*molniya + 0.009*chemical + 0.007*ch + 0.007*chemistry + 0.007*compound + 0.007*anemone + 0.006*mg + 0.006*reaction</span>
<span class="go">topic #12: 0.018*india + 0.013*indian + 0.010*tamil + 0.009*singh + 0.008*film + 0.008*temple + 0.006*kumar + 0.006*hindi + 0.006*delhi + 0.005*bengal</span>
<span class="go">topic #13: 0.047*bwebs + 0.024*malta + 0.020*hobart + 0.019*basa + 0.019*columella + 0.019*huon + 0.018*tasmania + 0.016*popups + 0.014*tasmanian + 0.014*modèle</span>
<span class="go">topic #14: 0.014*jewish + 0.011*rabbi + 0.008*bgwhite + 0.008*lebanese + 0.007*lebanon + 0.006*homs + 0.005*beirut + 0.004*jews + 0.004*hebrew + 0.004*caligari</span>
<span class="go">topic #15: 0.025*german + 0.020*der + 0.017*von + 0.015*und + 0.014*berlin + 0.012*germany + 0.012*die + 0.010*des + 0.008*kategorie + 0.007*cross</span>
<span class="go">topic #16: 0.003*can + 0.003*system + 0.003*power + 0.003*are + 0.003*energy + 0.002*data + 0.002*be + 0.002*used + 0.002*or + 0.002*using</span>
<span class="go">topic #17: 0.049*indonesia + 0.042*indonesian + 0.031*malaysia + 0.024*singapore + 0.022*greek + 0.021*jakarta + 0.016*greece + 0.015*dord + 0.014*athens + 0.011*malaysian</span>
<span class="go">topic #18: 0.031*stakes + 0.029*webs + 0.018*futsal + 0.014*whitish + 0.013*hyun + 0.012*thoroughbred + 0.012*dnf + 0.012*jockey + 0.011*medalists + 0.011*racehorse</span>
<span class="go">topic #19: 0.119*oblast + 0.034*uploaded + 0.034*uploads + 0.033*nordland + 0.025*selsoviet + 0.023*raion + 0.022*krai + 0.018*okrug + 0.015*hålogaland + 0.015*russiae + 0.020*manga + 0.017*dragon + 0.012*theme + 0.011*dvd + 0.011*super + 0.011*hunter + 0.009*ash + 0.009*dream + 0.009*angel</span>
</pre></div>
</div>
<p>Creating this LDA model of Wikipedia takes about 6 hours and 20 minutes on my laptop <a class="footnote-reference" href="#id6" id="id4">[1]</a>.
If you need your results faster, consider running <a class="reference internal" href="dist_lda.html"><em>Distributed Latent Dirichlet Allocation</em></a> on a cluster of
computers.</p>
<p>Note two differences between the LDA and LSA runs: we asked LSA
to extract 400 topics, LDA only 100 topics (so the difference in speed is in fact
even greater). Secondly, the LSA implementation in <cite>gensim</cite> is truly online: if the nature of the input
stream changes in time, LSA will re-orient itself to reflect these changes, in a reasonably
small amount of updates. In contrast, LDA is not truly online (the name of the <a class="footnote-reference" href="#id8" id="id5">[3]</a>
article notwithstanding), as the impact of later updates on the model gradually
diminishes. If there is topic drift in the input document stream, LDA will get
confused and be increasingly slower at adjusting itself to the new state of affairs.</p>
<p>In short, be careful if using LDA to incrementally add new documents to the model
over time. <strong>Batch usage of LDA</strong>, where the entire training corpus is either known beforehand or does
not exhibit topic drift, <strong>is ok and not affected</strong>.</p>
<p>To run batch LDA (not online), train <cite>LdaModel</cite> with:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="c"># extract 100 LDA topics, using 20 full passes, no online updates</span>
<span class="gp">>>> </span><span class="n">lda</span> <span class="o">=</span> <span class="n">gensim</span><span class="o">.</span><span class="n">models</span><span class="o">.</span><span class="n">ldamodel</span><span class="o">.</span><span class="n">LdaModel</span><span class="p">(</span><span class="n">corpus</span><span class="o">=</span><span class="n">mm</span><span class="p">,</span> <span class="n">id2word</span><span class="o">=</span><span class="n">id2word</span><span class="p">,</span> <span class="n">num_topics</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">update_every</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">passes</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
</pre></div>
</div>
<p>As usual, a trained model can used be to transform new, unseen documents (plain bag-of-words count vectors)
into LDA topic distributions:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">doc_lda</span> <span class="o">=</span> <span class="n">lda</span><span class="p">[</span><span class="n">doc_bow</span><span class="p">]</span>
</pre></div>
</div>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="id6" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#id1">1</a>, <a class="fn-backref" href="#id4">2</a>)</em> My laptop = MacBook Pro, Intel Core i7 2.3GHz, 16GB DDR3 RAM, OS X with <cite>libVec</cite>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id7" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><p class="first">Here we’re mostly interested in performance, but it is interesting to look at the
retrieved LSA concepts, too. I am no Wikipedia expert and don’t see into Wiki’s bowels,
but Brian Mingus had this to say about the result:</p>
<div class="highlight-python"><pre>There appears to be a lot of noise in your dataset. The first three topics
in your list appear to be meta topics, concerning the administration and
cleanup of Wikipedia. These show up because you didn't exclude templates
such as these, some of which are included in most articles for quality
control: http://en.wikipedia.org/wiki/Wikipedia:Template_messages/Cleanup
The fourth and fifth topics clearly shows the influence of bots that import
massive databases of cities, countries, etc. and their statistics such as
population, capita, etc.
The sixth shows the influence of sports bots, and the seventh of music bots.</pre>
</div>
<p class="last">So the top ten concepts are apparently dominated by Wikipedia robots and expanded
templates; this is a good reminder that LSA is a powerful tool for data analysis,
but no silver bullet. As always, it’s <a class="reference external" href="http://en.wikipedia.org/wiki/Garbage_In,_Garbage_Out">garbage in, garbage out</a>...
By the way, improvements to the Wiki markup parsing code are welcome :-)</p>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id8" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[3]</td><td><em>(<a class="fn-backref" href="#id3">1</a>, <a class="fn-backref" href="#id5">2</a>)</em> Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation
[<a class="reference external" href="http://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf">pdf</a>] [<a class="reference external" href="http://www.cs.princeton.edu/~mdhoffma/">code</a>]</td></tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="apiref.html" title="API Reference"
>next</a> |</li>
<li class="right" >
<a href="dist_lda.html" title="Distributed Latent Dirichlet Allocation"
>previous</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
</ul>
</div>
<div class="footer">
© Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>.
Last updated on Sep 16, 2012.
</div>
</body>
</html>