A machine learning–based package for transforming text into instrumental variables (IV).

Visit Stata Example File and Python Example File.

There is a step-by-step example.

• Support multiple Chinese word segmentation and embedding methods
• Customizable stopwords
• Support keyword relevance filtering and two-stage filtering
• Output includes frequency, total count, and ratio statistics

• Python 3.11+
• Recommended to use virtual environment (e.g., venv or conda)

pip install texiv

from typing import List

from texiv import TexIV

texiv = TexIV()
content: str = "This is a test text..."
keywords: List[str] = ["keyword1", "keyword2", "keyword3"]
texiv.texiv_it(content, keywords)

Async usage:

import asyncio
from typing import List

from texiv import AsyncTexIV, set_parallel_count

# Set the default concurrency for new AsyncTexIV instances.
set_parallel_count(10)

texiv = AsyncTexIV()
content: str = "This is a test text..."
keywords: List[str] = ["keyword1", "keyword2", "keyword3"]
result = asyncio.run(texiv.texiv_it(content, keywords))

For DataFrame processing, call the async API from an event loop:

import asyncio
import pandas as pd

from texiv import AsyncTexIV, set_parallel_count

async def main():
    set_parallel_count(10)
    texiv = AsyncTexIV()
    df = pd.DataFrame({"text": ["First document", "Second document"]})
    return await texiv.texiv_df(df, "text", ["document"])

result_df = asyncio.run(main())

set_parallel_count() only affects TexIV or AsyncTexIV instances created after it is called. You can also pass max_concurrency= directly to an instance when you need per-instance control.

Output example:

{'freq': 7, 'count': 34, 'rate': 0.20588235294117646}

The project also provides a command-line interface that can be used directly after installation:

texiv --help

All models and parameters can be adjusted through configuration files in ～/.texiv/config.toml.

This project is licensed under the GNU Affero General Public License v3.0. See LICENSE for details.

Note: Commercial use requires compliance with AGPL-3.0 terms, including source code disclosure for network services.