BTR-TOOLS

Generic command-line toolkit for Btrieve database file analysis and export.

BTR-TOOLS is a comprehensive toolkit for analyzing, repairing, and exporting data from Btrieve database files. This toolkit provides generic tools that can work with any Btrieve v5 format database files, automatically detecting record structures, extracting data, and exporting to multiple formats.

• Directory Scanning: Automatically detect Btrieve files in directories
• File Analysis: Comprehensive analysis of file structure and content patterns
• Data Export: Export data to CSV, JSON Lines, SQLite, and Excel formats
• Schema Detection: Automatically detect field boundaries and data types
• Integrity Checking: Verify file integrity and detect corruption
• Record Size Detection: Automatically determine optimal record sizes
• Rich CLI Output: Enhanced terminal interface with progress bars and colored output
• Comprehensive Testing: Full test suite with unit, integration, and performance tests
• CI/CD Pipeline: Automated testing, linting, and quality assurance
• Batch Processing: Process multiple Btrieve files simultaneously
• Data Repair: Validate and repair corrupted Btrieve files
• Advanced Search: Search and filter records with regex support
• Data Visualization: Generate HTML reports with statistics and analysis
• Performance Profiling: Monitor memory usage and processing speed

git clone <repository-url>
cd btrtools
pip install -e .

For development with all testing and quality tools:

git clone <repository-url>
cd btrtools
pip install -e ".[dev]"

You can also run the tools directly without installation:

python scripts/btrtools --help

All commands support rich output with progress bars using the --progress or -P flag.

Scan a directory for Btrieve files:

# Basic scan
btrtools scan /path/to/directory --recursive

# With rich output and progress bars
btrtools --progress scan /path/to/directory --recursive

Analyze a specific Btrieve file:

# Basic analysis
btrtools analyze file.btr --max-records 1000

# With rich output
btrtools --progress analyze file.btr --max-records 1000

Export Btrieve data to various formats:

# Export to CSV
btrtools export file.btr --format csv

# Export to JSON Lines with progress
btrtools --progress export file.btr --format jsonl

# Export to SQLite database
btrtools export file.btr --format sqlite

# Export to Excel spreadsheet
btrtools export file.btr --format excel

# Specify record size and limit records
btrtools export file.btr --format csv --record-size 128 --max-records 1000

Automatically detect the schema/structure of a Btrieve file:

btrtools schema file.btr --max-records 1000

# With rich table output
btrtools --progress schema file.btr --max-records 1000

Verify file integrity and check for corruption:

btrtools check file.btr --verbose

# With progress indicators
btrtools --progress check file.btr --verbose

Compare two Btrieve files to identify differences and similarities:

btrtools compare file1.btr file2.btr

# With rich comparison output
btrtools --progress compare file1.btr file2.btr --max-records 1000

# Save comparison results to JSON
btrtools compare file1.btr file2.btr --output comparison.json

Process multiple Btrieve files simultaneously with various operations:

# Batch analyze multiple files
btrtools batch *.btr --operation analyze

# Batch export all files to CSV
btrtools batch *.btr --operation export --format csv --output-dir ./exports

# Parallel processing with 4 workers
btrtools batch *.btr --operation schema --parallel 4

Validate and repair corrupted Btrieve files:

# Validate file integrity without repairing
btrtools repair file.btr --validate-only

# Attempt to repair corrupted file
btrtools repair file.btr --fix-corruption --backup

# Repair with custom output location
btrtools repair file.btr --fix-corruption --output repaired_file.btr

Search for specific content within Btrieve records:

# Simple text search
btrtools search file.btr --query "JOHN DOE"

# Case-insensitive regex search
btrtools search file.btr --query "customer.*\\d+" --regex

# Invert match (show non-matching records)
btrtools search file.btr --query "test" --invert-match

# Export search results to CSV
btrtools search file.btr --query "error" --format csv --output search_results.csv

Create comprehensive analysis reports with visualizations:

# Generate HTML report
btrtools report file.btr --format html --output-dir ./reports

# Generate JSON report for programmatic use
btrtools report file.btr --format json

# Text report with limited records
btrtools report file.btr --format text --max-records 500

Monitor processing performance and memory usage:

# Basic performance statistics
btrtools stats file.btr

# Run performance benchmarks
btrtools stats file.btr --benchmark

# Include memory profiling
btrtools stats file.btr --memory-profile --benchmark

# Save statistics to JSON file
btrtools stats file.btr --output performance.json

BTR-TOOLS supports enhanced terminal output with progress bars, colored text, and structured tables. Enable rich output with the --progress or -P flag:

# Enable rich output for any command
btrtools --progress analyze file.btr
btrtools -P scan /data --recursive

• Progress Bars: Visual progress indicators for long-running operations
• Colored Output: Success (green), warnings (yellow), errors (red)
• Structured Tables: Formatted data display for analysis results
• Icons: Visual indicators for different types of messages
• Spinners: Activity indicators during processing

BTR-TOOLS - Btrieve File Analysis Toolkit
Version 2.3.0 | Rich output enabled
⠋ Reading file structure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

    Btrieve File Analysis Results    
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Property        ┃ Value           ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Filename        │ example.btr     │
│ File Size       │ 4,096 bytes     │
│ ASCII Content   │ 85.2%           │
│ Quality Score   │ 7.8             │
└─────────────────┴─────────────────┘

✅ Operation completed successfully

The toolkit uses several analysis techniques to understand Btrieve file content:

• ASCII content percentage
• Digit sequence detection
• Date pattern recognition
• Content type classification

• Tests common record sizes (32, 64, 128, 256, 512, 1024 bytes)
• Quality scoring based on text content, digit patterns, and printable characters
• Automatic selection of optimal record size

• Regex-based pattern matching for common data patterns:
  • Alphanumeric codes (uppercase letter sequences)
  • Addresses and location patterns
  • Numeric identifiers and sequences
  • Date and time formats
  • Monetary amounts and quantities

• Position-by-position analysis across multiple records
• Field boundary detection based on content patterns
• Automatic field type inference
• Quality metrics for each detected field

Standard CSV format with headers. Includes both raw data and extracted fields:

record_num,record_size,decoded_text,printable_chars,has_digits,has_alpha,code_field,text_field,numeric_field
1,64,"ABC COMPANY     123 MAIN ST   ANYTOWN  ST 12345",45,true,true,ABC,"MAIN ST",12345
2,64,"XYZ CORP        800-555-0123         CODE123 125.00",42,true,true,XYZ,,8005550123,CODE123,125.00

One JSON object per line, suitable for streaming processing:

{"record_num": 1, "record_size": 64, "raw_bytes": "4142432...", "decoded_text": "ABC COMPANY...", "printable_chars": 45, "has_digits": true, "has_alpha": true, "extracted_fields": {"code_field": "ABC", "text_field": "MAIN ST", "numeric_field": "12345"}}

Creates a SQLite database with a btrieve_records table containing all data and extracted fields.

Creates an Excel spreadsheet (.xlsx) with formatted data, auto-adjusted column widths, and bold headers. Includes all record data and extracted fields in a user-friendly spreadsheet format.

# Scan a directory for Btrieve files
btrtools scan /data/archive --recursive --output scan_results.txt

# Analyze a discovered file
btrtools analyze inventory.btr

# Check file integrity
btrtools check inventory.btr --verbose

# Detect the data schema
btrtools schema inventory.btr --output schema.txt

# Compare with backup file
btrtools compare inventory.btr inventory_backup.btr

# Export data to CSV format
btrtools export inventory.btr --format csv --output inventory.csv

# Export data to Excel format
btrtools export inventory.btr --format excel --output inventory.xlsx

# Analyze with more records for better schema detection
btrtools analyze large_file.btr --max-records 5000

# Export specific number of records
btrtools export data.btr --format jsonl --max-records 1000 --output data.jsonl

# Force specific record size for problematic files
btrtools export data.btr --format sqlite --record-size 256 --output data.db

# Compare files with detailed record analysis
btrtools compare production.btr staging.btr --max-records 1000 --output diff.json

# Batch process multiple files
for file in *.btr; do
    btrtools export "$file" --format csv
done

# Step 1: Verify file integrity
btrtools check legacy_data.btr

# Step 2: Analyze file structure
btrtools analyze legacy_data.btr --max-records 1000

# Step 3: Detect schema automatically
btrtools schema legacy_data.btr

# Step 4: Export to modern format
btrtools export legacy_data.btr --format sqlite --output modern_data.db

The toolkit uses a quality scoring system to evaluate extraction success:

• Text Content (30 points): Percentage of records with readable text
• Digit Patterns (20 points): Records containing digit sequences
• Alpha Patterns (20 points): Records containing alphabetic characters
• Printable Density (30 points): Average printable characters per record

Scores range from 0-100, with higher scores indicating better data quality and more reliable extractions.

1. "Could not detect record size"

   • Try specifying the record size manually: --record-size 128
   • Check file integrity: btrtools check file.btr

2. "No records found to export"

   • Verify the file is a valid Btrieve file
   • Try different record sizes
   • Check if the file has data pages beyond the FCR

3. Low quality scores

   • File may be corrupted or not a Btrieve file
   • Try smaller record sizes for better granularity
   • Check ASCII content percentage

4. Memory issues with large files

   • Use --max-records to limit processing
   • Export in smaller batches

• Minimum file size: 8KB (2 FCR pages + 1 data page)
• Files should be multiples of 4KB (page size) for best results
• Very large files (>100MB) may require batch processing

BTR-TOOLS is useful for:

• Legacy System Migration: Extract data from old Btrieve databases
• Data Recovery: Recover data from corrupted or damaged files
• Database Archaeology: Analyze unknown legacy file formats
• Data Conversion: Transform Btrieve data to modern formats
• Forensic Analysis: Examine file contents for auditing purposes
• Research: Study historical data storage formats

BTR-TOOLS includes a comprehensive test suite covering unit tests, integration tests, and performance benchmarks:

# Run all tests
pytest btrtools/tests/

# Run with coverage report
pytest btrtools/tests/ --cov=btrtools --cov-report=html

# Run specific test categories
pytest btrtools/tests/test_basic.py     # Unit tests
pytest btrtools/tests/test_cli.py       # CLI integration tests
pytest btrtools/tests/test_performance.py  # Performance tests

The project uses multiple tools for code quality assurance:

# Format code
black btrtools/
isort btrtools/

# Lint code
flake8 btrtools/
mypy btrtools/

# Security scanning
bandit -r btrtools/
safety check

BTR-TOOLS uses GitHub Actions for automated testing and quality assurance:

• Multi-platform testing: Windows, macOS, Linux
• Python version matrix: 3.8, 3.9, 3.10, 3.11
• Automated linting: black, isort, flake8, mypy
• Security scanning: bandit, safety
• Documentation building: Sphinx
• Test coverage reporting: pytest-cov

btrtools/
├── __init__.py          # Package initialization
├── core/
│   ├── __init__.py
│   └── btrieve.py       # Core Btrieve file handling
├── cli/
│   ├── __init__.py      # Main CLI entry point
│   ├── scan.py          # Directory scanning
│   ├── analyze.py       # File analysis
│   ├── export.py        # Data export
│   ├── schema.py        # Schema detection
│   └── check.py         # Integrity checking
├── utils/
│   ├── __init__.py
│   └── logging.py       # Logging and error handling
└── tests/
    ├── __init__.py
    ├── test_basic.py    # Unit tests
    ├── test_cli.py      # CLI integration tests
    └── test_performance.py  # Performance benchmarks

1. Create a new module in btrtools/cli/
2. Implement the command function
3. Add the command to the main CLI parser in __init__.py
4. Update this README

The field extraction patterns in btrieve.py can be extended by adding new regex patterns to the _extract_basic_fields method for domain-specific data patterns.

For detailed technical documentation and code organization:

• Code Outline Reference Document - Complete codebase overview, function indices, and development guidelines
• Sphinx Documentation - Comprehensive API documentation and usage guides
• Contributing Guide - Development setup and contribution guidelines

MIT License - see LICENSE file for details.

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

• Btrieve file format documentation
• Database migration tools
• Legacy system analysis utilities