forked from piskvorky/gensim
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathutils.html
More file actions
458 lines (395 loc) · 28 KB
/
Copy pathutils.html
File metadata and controls
458 lines (395 loc) · 28 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>utils – Various utility functions — gensim</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '0.8.6',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="author" title="About these documents" href="about.html" />
<link rel="top" title="gensim" href="index.html" />
<link rel="up" title="API Reference" href="apiref.html" />
<link rel="next" title="matutils – Math utils" href="matutils.html" />
<link rel="prev" title="interfaces – Core gensim interfaces" href="interfaces.html" />
<!-- twitter search widget
<script type="text/javascript" src="_static/widget.js"></script>
-->
<meta property="og:title" content="#gensim" />
<meta property="og:description" content="Efficient topic modelling in Python" />
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-24066335-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="matutils.html" title="matutils – Math utils"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="interfaces.html" title="interfaces – Core gensim interfaces"
accesskey="P">previous</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
<li><a href="apiref.html" accesskey="U">API Reference</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h4>Previous topic</h4>
<p class="topless"><a href="interfaces.html"
title="previous chapter"><tt class="docutils literal docutils literal docutils literal"><span class="pre">interfaces</span></tt> – Core gensim interfaces</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="matutils.html"
title="next chapter"><tt class="docutils literal docutils literal docutils literal"><span class="pre">matutils</span></tt> – Math utils</a></p>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="24" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="module-gensim.utils">
<span id="utils-various-utility-functions"></span><h1><tt class="xref py py-mod docutils literal"><span class="pre">utils</span></tt> – Various utility functions<a class="headerlink" href="#module-gensim.utils" title="Permalink to this headline">¶</a></h1>
<p>This module contains various general utility functions.</p>
<dl class="class">
<dt id="gensim.utils.FakeDict">
<em class="property">class </em><tt class="descclassname">gensim.utils.</tt><tt class="descname">FakeDict</tt><big>(</big><em>num_terms</em><big>)</big><a class="headerlink" href="#gensim.utils.FakeDict" title="Permalink to this definition">¶</a></dt>
<dd><p>Objects of this class act as dictionaries that map integer->str(integer), for
a specified range of integers <0, num_terms).</p>
<p>This is meant to avoid allocating real dictionaries when <cite>num_terms</cite> is huge, which
is a waste of memory.</p>
<dl class="method">
<dt id="gensim.utils.FakeDict.keys">
<tt class="descname">keys</tt><big>(</big><big>)</big><a class="headerlink" href="#gensim.utils.FakeDict.keys" title="Permalink to this definition">¶</a></dt>
<dd><p>Override the dict.keys() function, which is used to determine the maximum
internal id of a corpus = the vocabulary dimensionality.</p>
<p>HACK: To avoid materializing the whole <cite>range(0, self.num_terms)</cite>, this returns
<cite>[self.num_terms - 1]</cite> only.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="gensim.utils.RepeatCorpus">
<em class="property">class </em><tt class="descclassname">gensim.utils.</tt><tt class="descname">RepeatCorpus</tt><big>(</big><em>corpus</em>, <em>reps</em><big>)</big><a class="headerlink" href="#gensim.utils.RepeatCorpus" title="Permalink to this definition">¶</a></dt>
<dd><p>Used in the tutorial on distributed computing and likely not useful anywhere else.</p>
<p>Wrap a <cite>corpus</cite> as another corpus of length <cite>reps</cite>. This is achieved by
repeating documents from <cite>corpus</cite> over and over again, until the requested
length <cite>len(result)==reps</cite> is reached. Repetition is done
on-the-fly=efficiently, via <cite>itertools</cite>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="p">[[(</span><span class="mi">1</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">)],</span> <span class="p">[]]</span> <span class="c"># 2 documents</span>
<span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="n">RepeatCorpus</span><span class="p">(</span><span class="n">corpus</span><span class="p">,</span> <span class="mi">5</span><span class="p">))</span> <span class="c"># repeat 2.5 times to get 5 documents</span>
<span class="go">[[(1, 0.5)], [], [(1, 0.5)], [], [(1, 0.5)]]</span>
</pre></div>
</div>
<dl class="classmethod">
<dt id="gensim.utils.RepeatCorpus.load">
<em class="property">classmethod </em><tt class="descname">load</tt><big>(</big><em>fname</em><big>)</big><a class="headerlink" href="#gensim.utils.RepeatCorpus.load" title="Permalink to this definition">¶</a></dt>
<dd><p>Load a previously saved object from file (also see <cite>save</cite>).</p>
</dd></dl>
<dl class="method">
<dt id="gensim.utils.RepeatCorpus.save">
<tt class="descname">save</tt><big>(</big><em>fname</em><big>)</big><a class="headerlink" href="#gensim.utils.RepeatCorpus.save" title="Permalink to this definition">¶</a></dt>
<dd><p>Save the object to file via pickling (also see <cite>load</cite>).</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="gensim.utils.SaveLoad">
<em class="property">class </em><tt class="descclassname">gensim.utils.</tt><tt class="descname">SaveLoad</tt><a class="headerlink" href="#gensim.utils.SaveLoad" title="Permalink to this definition">¶</a></dt>
<dd><p>Objects which inherit from this class have save/load functions, which un/pickle
them to disk.</p>
<p>This uses cPickle for de/serializing, so objects must not contains unpicklable
attributes, such as lambda functions etc.</p>
<dl class="classmethod">
<dt id="gensim.utils.SaveLoad.load">
<em class="property">classmethod </em><tt class="descname">load</tt><big>(</big><em>fname</em><big>)</big><a class="headerlink" href="#gensim.utils.SaveLoad.load" title="Permalink to this definition">¶</a></dt>
<dd><p>Load a previously saved object from file (also see <cite>save</cite>).</p>
</dd></dl>
<dl class="method">
<dt id="gensim.utils.SaveLoad.save">
<tt class="descname">save</tt><big>(</big><em>fname</em><big>)</big><a class="headerlink" href="#gensim.utils.SaveLoad.save" title="Permalink to this definition">¶</a></dt>
<dd><p>Save the object to file via pickling (also see <cite>load</cite>).</p>
</dd></dl>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.any2unicode">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">any2unicode</tt><big>(</big><em>text</em>, <em>encoding='utf8'</em>, <em>errors='strict'</em><big>)</big><a class="headerlink" href="#gensim.utils.any2unicode" title="Permalink to this definition">¶</a></dt>
<dd><p>Convert a string (bytestring in <cite>encoding</cite> or unicode), to unicode.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.any2utf8">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">any2utf8</tt><big>(</big><em>text</em>, <em>errors='strict'</em>, <em>encoding='utf8'</em><big>)</big><a class="headerlink" href="#gensim.utils.any2utf8" title="Permalink to this definition">¶</a></dt>
<dd><p>Convert a string (unicode or bytestring in <cite>encoding</cite>), to bytestring in utf8.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.chunkize">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">chunkize</tt><big>(</big><em>corpus</em>, <em>chunksize</em>, <em>maxsize=0</em>, <em>as_numpy=False</em><big>)</big><a class="headerlink" href="#gensim.utils.chunkize" title="Permalink to this definition">¶</a></dt>
<dd><p>Split a stream of values into smaller chunks.
Each chunk is of length <cite>chunksize</cite>, except the last one which may be smaller.
A once-only input stream (<cite>corpus</cite> from a generator) is ok, chunking is done
efficiently via itertools.</p>
<p>If <cite>maxsize > 1</cite>, don’t wait idly in between successive chunk <cite>yields</cite>, but
rather keep filling a short queue (of size at most <cite>maxsize</cite>) with forthcoming
chunks in advance. This is realized by starting a separate process, and is
meant to reduce I/O delays, which can be significant when <cite>corpus</cite> comes
from a slow medium (like harddisk).</p>
<p>If <cite>maxsize==0</cite>, don’t fool around with parallelism and simply yield the chunksize
via <cite>chunkize_serial()</cite> (no I/O optimizations).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">chunkize</span><span class="p">(</span><span class="nb">xrange</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span> <span class="mi">4</span><span class="p">):</span> <span class="k">print</span> <span class="n">chunk</span>
<span class="go">[0, 1, 2, 3]</span>
<span class="go">[4, 5, 6, 7]</span>
<span class="go">[8, 9]</span>
</pre></div>
</div>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.chunkize_serial">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">chunkize_serial</tt><big>(</big><em>iterable</em>, <em>chunksize</em>, <em>as_numpy=False</em><big>)</big><a class="headerlink" href="#gensim.utils.chunkize_serial" title="Permalink to this definition">¶</a></dt>
<dd><p>Return elements from the iterable in <cite>chunksize</cite>-ed lists. The last returned
element may be smaller (if length of collection is not divisible by <cite>chunksize</cite>).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="k">print</span> <span class="nb">list</span><span class="p">(</span><span class="n">grouper</span><span class="p">(</span><span class="nb">xrange</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span> <span class="mi">3</span><span class="p">))</span>
<span class="go">[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]</span>
</pre></div>
</div>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.copytree_hardlink">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">copytree_hardlink</tt><big>(</big><em>source</em>, <em>dest</em><big>)</big><a class="headerlink" href="#gensim.utils.copytree_hardlink" title="Permalink to this definition">¶</a></dt>
<dd><p>Recursively copy a directory ala shutils.copytree, but hardlink files
instead of copying. Available on UNIX systems only.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.deaccent">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">deaccent</tt><big>(</big><em>text</em><big>)</big><a class="headerlink" href="#gensim.utils.deaccent" title="Permalink to this definition">¶</a></dt>
<dd><p>Remove accentuation from the given string. Input text is either a unicode string or utf8 encoded bytestring.</p>
<p>Return input string with accents removed, as unicode.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">deaccent</span><span class="p">(</span><span class="s">"Šéf chomutovských komunistů dostal poštou bílý prášek"</span><span class="p">)</span>
<span class="go">u'Sef chomutovskych komunistu dostal postou bily prasek'</span>
</pre></div>
</div>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.decode_htmlentities">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">decode_htmlentities</tt><big>(</big><em>text</em><big>)</big><a class="headerlink" href="#gensim.utils.decode_htmlentities" title="Permalink to this definition">¶</a></dt>
<dd><p>Decode HTML entities in text, coded as hex, decimal or named.</p>
<p>Adapted from <a class="reference external" href="http://github.com/sku/python-twitter-ircbot/blob/321d94e0e40d0acc92f5bf57d126b57369da70de/html_decode.py">http://github.com/sku/python-twitter-ircbot/blob/321d94e0e40d0acc92f5bf57d126b57369da70de/html_decode.py</a></p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">u</span> <span class="o">=</span> <span class="s">u'E tu vivrai nel terrore - L&#x27;aldil&#xE0; (1981)'</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">decode_htmlentities</span><span class="p">(</span><span class="n">u</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'UTF-8'</span><span class="p">)</span>
<span class="go">E tu vivrai nel terrore - L'aldilà (1981)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">decode_htmlentities</span><span class="p">(</span><span class="s">"l&#39;eau"</span><span class="p">)</span>
<span class="go">l'eau</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">decode_htmlentities</span><span class="p">(</span><span class="s">"foo &lt; bar"</span><span class="p">)</span>
<span class="go">foo < bar</span>
</pre></div>
</div>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.dict_from_corpus">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">dict_from_corpus</tt><big>(</big><em>corpus</em><big>)</big><a class="headerlink" href="#gensim.utils.dict_from_corpus" title="Permalink to this definition">¶</a></dt>
<dd><p>Scan corpus for all word ids that appear in it, then construct and return a mapping
which maps each <tt class="docutils literal"><span class="pre">wordId</span> <span class="pre">-></span> <span class="pre">str(wordId)</span></tt>.</p>
<p>This function is used whenever <em>words</em> need to be displayed (as opposed to just
their ids) but no wordId->word mapping was provided. The resulting mapping
only covers words actually used in the corpus, up to the highest wordId found.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.getNS">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">getNS</tt><big>(</big><big>)</big><a class="headerlink" href="#gensim.utils.getNS" title="Permalink to this definition">¶</a></dt>
<dd><p>Return a Pyro name server proxy. If there is no name server running,
start one on 0.0.0.0 (all interfaces), as a background process.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.get_max_id">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">get_max_id</tt><big>(</big><em>corpus</em><big>)</big><a class="headerlink" href="#gensim.utils.get_max_id" title="Permalink to this definition">¶</a></dt>
<dd><p>Return highest feature id that appears in the corpus.</p>
<p>For empty corpora (no features at all), return -1.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.get_my_ip">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">get_my_ip</tt><big>(</big><big>)</big><a class="headerlink" href="#gensim.utils.get_my_ip" title="Permalink to this definition">¶</a></dt>
<dd><p>Try to obtain our external ip (from the pyro nameserver’s point of view)</p>
<p>This tries to sidestep the issue of bogus <cite>/etc/hosts</cite> entries and other
local misconfigurations, which often mess up hostname resolution.</p>
<p>If all else fails, fall back to simple <cite>socket.gethostbyname()</cite> lookup.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.grouper">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">grouper</tt><big>(</big><em>iterable</em>, <em>chunksize</em>, <em>as_numpy=False</em><big>)</big><a class="headerlink" href="#gensim.utils.grouper" title="Permalink to this definition">¶</a></dt>
<dd><p>Return elements from the iterable in <cite>chunksize</cite>-ed lists. The last returned
element may be smaller (if length of collection is not divisible by <cite>chunksize</cite>).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="k">print</span> <span class="nb">list</span><span class="p">(</span><span class="n">grouper</span><span class="p">(</span><span class="nb">xrange</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span> <span class="mi">3</span><span class="p">))</span>
<span class="go">[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]</span>
</pre></div>
</div>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.is_corpus">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">is_corpus</tt><big>(</big><em>obj</em><big>)</big><a class="headerlink" href="#gensim.utils.is_corpus" title="Permalink to this definition">¶</a></dt>
<dd><p>Check whether <cite>obj</cite> is a corpus. Return (is_corpus, new) 2-tuple, where
<cite>new is obj</cite> if <cite>obj</cite> was an iterable, or <cite>new</cite> yields the same sequence as
<cite>obj</cite> if it was an iterator.</p>
<p><cite>obj</cite> is a corpus if it supports iteration over documents, where a document
is in turn anything that acts as a sequence of 2-tuples (int, float).</p>
<p>Note: An “empty” corpus (empty input sequence) is ambiguous, so in this case the
result is forcefully defined as <cite>is_corpus=False</cite>.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.lemmatize">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">lemmatize</tt><big>(</big><em>content</em><big>)</big><a class="headerlink" href="#gensim.utils.lemmatize" title="Permalink to this definition">¶</a></dt>
<dd><p>Use the English lemmatizer from the <cite>pattern</cite> package to extract tokens in
their base form (lemmas: “are, is, being”->”be” etc.).
This is a smarter version of stemming.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.pickle">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">pickle</tt><big>(</big><em>obj</em>, <em>fname</em>, <em>protocol=-1</em><big>)</big><a class="headerlink" href="#gensim.utils.pickle" title="Permalink to this definition">¶</a></dt>
<dd><p>Pickle object <cite>obj</cite> to file <cite>fname</cite>.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.pyro_daemon">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">pyro_daemon</tt><big>(</big><em>name</em>, <em>obj</em>, <em>random_suffix=False</em>, <em>ip=None</em>, <em>port=None</em><big>)</big><a class="headerlink" href="#gensim.utils.pyro_daemon" title="Permalink to this definition">¶</a></dt>
<dd><p>Register object with name server (starting the name server if not running
yet) and block until the daemon is terminated. The object is registered under
<cite>name</cite>, or <cite>name`+ some random suffix if `random_suffix</cite> is set.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.revdict">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">revdict</tt><big>(</big><em>d</em><big>)</big><a class="headerlink" href="#gensim.utils.revdict" title="Permalink to this definition">¶</a></dt>
<dd><p>Reverse a dictionary mapping.</p>
<p>When two keys map to the same value, only one of them will be kept in the
result (which one is kept is arbitrary).</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.simple_preprocess">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">simple_preprocess</tt><big>(</big><em>doc</em><big>)</big><a class="headerlink" href="#gensim.utils.simple_preprocess" title="Permalink to this definition">¶</a></dt>
<dd><p>Convert a document into a list of tokens.</p>
<p>This lowercases, tokenizes, stems, normalizes etc. – the output are final,
utf8 encoded strings that won’t be processed any further.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.synchronous">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">synchronous</tt><big>(</big><em>tlockname</em><big>)</big><a class="headerlink" href="#gensim.utils.synchronous" title="Permalink to this definition">¶</a></dt>
<dd><p>A decorator to place an instance-based lock around a method.</p>
<p>Adapted from <a class="reference external" href="http://code.activestate.com/recipes/577105-synchronization-decorator-for-class-methods/">http://code.activestate.com/recipes/577105-synchronization-decorator-for-class-methods/</a></p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.to_unicode">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">to_unicode</tt><big>(</big><em>text</em>, <em>encoding='utf8'</em>, <em>errors='strict'</em><big>)</big><a class="headerlink" href="#gensim.utils.to_unicode" title="Permalink to this definition">¶</a></dt>
<dd><p>Convert a string (bytestring in <cite>encoding</cite> or unicode), to unicode.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.to_utf8">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">to_utf8</tt><big>(</big><em>text</em>, <em>errors='strict'</em>, <em>encoding='utf8'</em><big>)</big><a class="headerlink" href="#gensim.utils.to_utf8" title="Permalink to this definition">¶</a></dt>
<dd><p>Convert a string (unicode or bytestring in <cite>encoding</cite>), to bytestring in utf8.</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.tokenize">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">tokenize</tt><big>(</big><em>text</em>, <em>lowercase=False</em>, <em>deacc=False</em>, <em>errors='strict'</em>, <em>to_lower=False</em>, <em>lower=False</em><big>)</big><a class="headerlink" href="#gensim.utils.tokenize" title="Permalink to this definition">¶</a></dt>
<dd><p>Iteratively yield tokens as unicode strings, optionally also lowercasing them
and removing accent marks.</p>
<p>Input text may be either unicode or utf8-encoded byte string.</p>
<p>The tokens on output are maximal contiguous sequences of alphabetic
characters (no digits!).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="nb">list</span><span class="p">(</span><span class="n">tokenize</span><span class="p">(</span><span class="s">'Nic nemůže letět rychlostí vyšší, než 300 tisíc kilometrů za sekundu!'</span><span class="p">,</span> <span class="n">deacc</span> <span class="o">=</span> <span class="bp">True</span><span class="p">))</span>
<span class="go">[u'Nic', u'nemuze', u'letet', u'rychlosti', u'vyssi', u'nez', u'tisic', u'kilometru', u'za', u'sekundu']</span>
</pre></div>
</div>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.toptexts">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">toptexts</tt><big>(</big><em>query</em>, <em>texts</em>, <em>index</em>, <em>n=10</em><big>)</big><a class="headerlink" href="#gensim.utils.toptexts" title="Permalink to this definition">¶</a></dt>
<dd><p>Debug fnc to help inspect the top <cite>n</cite> most similar documents (according to a
similarity index <cite>index</cite>), to see if they are actually related to the query.</p>
<p><cite>texts</cite> is any object that can return something insightful for each document
via <cite>texts[docid]</cite>, such as its fulltext or snippet.</p>
<p>Return a list of 3-tuples (docid, doc’s similarity to the query, texts[docid]).</p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.unpickle">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">unpickle</tt><big>(</big><em>fname</em><big>)</big><a class="headerlink" href="#gensim.utils.unpickle" title="Permalink to this definition">¶</a></dt>
<dd><p>Load pickled object from <cite>fname</cite></p>
</dd></dl>
<dl class="function">
<dt id="gensim.utils.upload_chunked">
<tt class="descclassname">gensim.utils.</tt><tt class="descname">upload_chunked</tt><big>(</big><em>server</em>, <em>docs</em>, <em>chunksize=1000</em>, <em>preprocess=None</em><big>)</big><a class="headerlink" href="#gensim.utils.upload_chunked" title="Permalink to this definition">¶</a></dt>
<dd><p>Memory-friendly upload of documents to a SimServer (or Pyro SimServer proxy).</p>
<p>Use this function to train or index large collections – avoid sending the
entire corpus over the wire as a single Pyro in-memory object. The documents
will be sent in smaller chunks, of <cite>chunksize</cite> documents each.</p>
</dd></dl>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="matutils.html" title="matutils – Math utils"
>next</a> |</li>
<li class="right" >
<a href="interfaces.html" title="interfaces – Core gensim interfaces"
>previous</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
<li><a href="apiref.html" >API Reference</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>.
Last updated on Sep 16, 2012.
</div>
</body>
</html>