RCT-Reviewer is a modernized, standalone version of RobotReviewer, designed as a third-party reference tool for Risk of Bias assessment. It builds upon RobotReviewer’s original machine learning models trained on 12,808 randomized controlled trials (RCTs).

RCT-Reviewer is designed as a Third-Party Tiebreaker Reference for systematic reviews. Standard guidelines require two independent human reviewers; when they disagree, this tool provides an instant, objective, and data-driven third opinion to resolve ties.

• Near-Human Accuracy: The system achieves 71.0% accuracy for Risk of Bias judgments, performing within <8% of human expert consensus (which stands at 78.3%) [1].

• Highly Precise Extraction: In a randomized Cochrane user trial, the models demonstrated 87% Precision and 90% Recall for identifying the exact text snippets supporting the bias judgment [2].

• Validated Acceptance: Real-world feasibility studies show that human reviewers accept the tool's judgments at a rate equal to that of their human peers (Risk Ratio 1.02) [3].

• Rigorous Methodology: Developed by Marshall, Kuiper, and Wallace, the models were trained on 12,808 clinical trial PDFs using "distant supervision" to ensure high-quality classification without prohibitive manual labeling costs [1,4].

RCT-Reviewer has been independently validated against the original 2017 RobotReviewer implementation using a dedicated five-tier validation harness.

• Predictive validity: On 751 human-labelled Clinical Hedges records, RCT-Reviewer achieved 91.5% accuracy, 94.1% sensitivity, 88.1% specificity, 0.925 F1 and 0.966 ROC AUC.
• RobotReviewer fidelity: The refactored RCT classification pipeline reproduces the executed original bit-identically — 751/751 records, with max |Δ| = 0.0.
• Risk-of-Bias fidelity: 100% agreement with the original across 6,018 document × domain comparisons (κ = 1.0), including identical sentence scores and vectorizer outputs.
• SVM/CNN ablation: The SVM-only pipeline achieves 92.5% decision agreement with the original full SVM+CNN+publication-type ensemble; most of the difference is attributable to the legacy publication-type features.
• PDF robustness: 1,000/1,000 PDFs parsed successfully, with a median processing time of 1.57 seconds per PDF.
• External human validation (Tier E, Tian 2024, n=313 open-access trials): concordance 60–76% across the four RoB domains (κ 0.26/0.20/0.48/0.12) — within the range published for the original tool (κ 0.25–0.59, concordance 63–83%).
• Human-reference control: on identical PMC text, the original 2017 code agrees with RCT-Reviewer on 100.0% of domain judgements; external fidelity vs the original's deposited publisher-PDF labels: 78.9%.
• Predictive validity detail: PPV 91.1, NPV 92.1, Cohen's κ 0.826, Brier score 0.067.
• Parser robustness detail: 12,060 pages parsed, success 95% CI 99.6–100.0, max 7.0 s per PDF, parse time scales linearly (Pearson r = 0.93).

The complete methodology, reproducibility data, statistical results and validation outputs are available in the RCT-Reviewer Validation repository.

This project preserves the machine learning core of the original RobotReviewer while modernizing the infrastructure.

1. PDF Parsing: Uses PyMuPDF (fitz) to extract text and spaCy to segment sentences. No Java/GROBID dependency required.
2. Feature Extraction: Uses HashingVectorizer (scikit-learn) to convert text into high-dimensional sparse matrices.
3. Classification:
   • MiniClassifier: A lightweight Linear SVM wrapper that loads pre-trained .npz weights.
   • SVM-Only Pipeline: CNN models have been removed due to dead TensorFlow 1.x dependencies. Their removal results in a measured 4.4% F1 drop (validated in Tier B) while preserving the core predictive signal.

This repository contains joblib-converted model artifacts originally developed in RobotReviewer. The old pickle model files were compressed and redistributed into .joblib format using the convert_models.py script and respective directories to ensure better compatibility and smaller file sizes for modern Python environments.

The users can access the RCT-Reviewer Hugging Face Repository at: huggingface.co/Aurumz/RCT-Reviewer

PROJECT ARCHITECTURE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
│
├── 🚀 Entry point detected
│
├── 📁 File Structure
├── 📄 .dockerignore
├── 📄 .gitattributes
├── 📄 .gitignore
├── 📄 .zenodo.json
├── 📄 INFO.md
├── 📄 LFS push:pull guide.md
├── 📄 LICENSE.txt
├── 📄 README.md
├── 📄 convert_models.py
│
├── 📁 data/
│   ├── 📁 bias/
│   │   ├── 📄 bias_doc_level.npz
│   │   ├── 📄 bias_sent_level.npz
│   │   ├── 📄 bias_ab.npz
│   │   ├── 📄 bias_prob_clf.pck
│   │   ├── 📄 domain_clf.pck
│   │   ├── 📄 overall_clf.pck
│   │   ├── 📄 drugbank.pck
│   │   └── 📁 humans/
│   │       ├── 📄 AC.hdf5 / AC.json / AC.pickle
│   │       ├── 📄 BOA.hdf5 / BOA.json / BOA.pickle
│   │       ├── 📄 BPP.hdf5 / BPP.json / BPP.pickle
│   │       └── 📄 RSG.hdf5 / RSG.json / RSG.pickle
│   ├── 📁 pico/
│   │   ├── 📄 I_idf.npz / I_model.npz
│   │   ├── 📄 O_idf.npz / O_model.npz
│   │   ├── 📄 P_idf.npz / P_model.npz
│   ├── 📁 rct/
│   │   ├── 📄 rct_svm_weights.npz
│   │   └── ... (CNN artifacts retained but unused)
│   └── 📁 vocab/
│       └── 📄 embeddings.200d.trimmed.npz
│
├── 📁 rct_reviewer/
│   ├── 📄 __init__.py
│   ├── 📄 app.py (Joblib Mode)
│   ├── 📄 app1.py (Pickle Mode)
│   ├── 📄 app2.py (HF Hub Mode)
│   ├── 📄 config.py
│   ├── 📁 processors/ (bias_robot, pico_robot, rct_robot)
│   └── 📁 ui/ (streamlit components)
│
└── 📄 requirements.txt

RCT-Reviewer uses a Linear SVM-only pipeline because the original CNN depends on obsolete TensorFlow/Keras infrastructure. Validation shows 92.5% agreement with the original SVM+CNN+publication-type ensemble, with the majority of the difference attributable to the legacy publication-type features. The SVM-only pipeline preserves the original classifier's scores exactly while providing a modern, deterministic and reproducible implementation.

This repository allows users to run RCT-Reviewer locally by downloading the required model files (.joblib, .pickle, .pck) using Git Large File Storage (LFS) or running directly via Hugging Face Hub integration.

Online Deployment: The default hosted version at rct-reviewer.streamlit.app uses app2.py (Hugging Face Hub). The repository for the online deployment is RCT-Reviewer/RCT-Reviewer-Online.

Before you begin, ensure you have the following installed:

1. Python 3.12: This project is optimized for the latest Python version.
2. Git: For cloning the repository.
3. Git LFS (Large File Storage): Required if you plan to run the offline modes (app.py or app1.py) to download pre-trained ML model weights.

⚠️ Note: This project strictly requires Python 3.12. It has been tested and verified to work perfectly on this version.

This repository contains the main model files along with the code required to run RCT-Reviewer. There are three ways to run this application locally:

1. app.py: Uses .joblib and .npz files (Modernized local storage).
2. app1.py: Uses .pickle, .pck, and .npz files (Legacy local storage).
3. app2.py: Connects to the Hugging Face Hub repository [Aurumz/RCT-Reviewer] (No large local files needed).

[Recommended to run app2.py over other .py to run latest version.]

Clone the repository and set up the virtual environment:

git clone https://github.com/aurumz-rgb/RCT-Reviewer.git
cd RCT-Reviewer

# Create virtual environment
python3.12 -m venv .venv

# Activate it
# Linux / macOS:
source .venv/bin/activate
# Windows:
.venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Download NLP Model (Required for all modes)
python -m spacy download en_core_web_sm

Choose one of the following methods to run the app. [Recommended to run app2.py over other .py to run latest version.]

git lfs install
git lfs pull

python -m streamlit run rct_reviewer/app.py

git lfs install
git lfs pull

python -m streamlit run rct_reviewer/app1.py

python -m streamlit run rct_reviewer/app2.py

For independent verification of predictive validity, model fidelity, SVM/CNN ablation and PDF parsing robustness, see the RCT-Reviewer Validation repository.

Q: I get FileNotFoundError: data/pico/P_model.npz A: You likely skipped the Git LFS step. Run git lfs pull to download the actual model weights (only required for app.py and app1.py).

Q: I get ModuleNotFoundError: No module named 'rct_reviewer' A: Ensure you are running the command from the root directory, or set your PYTHONPATH:

export PYTHONPATH=$PYTHONPATH:$(pwd)

Q: Can't find model 'en_core_web_sm' A: You forgot to download the spaCy model. Run:

python -m spacy download en_core_web_sm

1. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association. 2016;23(1):193-201. doi

2. Soboczenski F, et al. Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study. BMC Medical Informatics and Decision Making. 2019;19(1):96. doi

3. Nussbaumer-Streit B, et al. Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Medical Research Methodology. 2022;22:160. doi

4. Marshall I, Kuiper J, Wallace B. Automating Risk of Bias Assessment for Clinical Trials. IEEE Journal of Biomedical and Health Informatics. 2015;19(4):1406-1412. doi

Contributions are welcome! Please see CONTRIBUTING.md for details on how to suggest additions or changes.

RCT-Reviewer is a modernized version of the original RobotReviewer. I extend my sincere gratitude to the original authors: Iain J. Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace for their foundational work in biomedical NLP and for releasing the project as open-source.

I would also like to thank all contributors and collaborators involved in the RobotReviewer ecosystem, including the Cochrane Crowd and the research teams at UPenn, Northeastern, and UCL, whose efforts in data collection and model development made this tool possible.

Additionally, I would like to acknowledge the use of RikaiCode (Code Repository Context Generator) and GLM-4.7, which were invaluable in analyzing and understanding the complex logic of the original RobotReviewer codebase, as well as assisting in the development and modernization of RobotReviewer.

If you use this software in your research, please cite both RCT-Reviewer and the original RobotReviewer paper.

RCT-Reviewer:

Sahu, V. (2026). RCT-Reviewer: A Modernized, Standalone Tool for Automated Analysis of Clinical Trials (RCTs). Zenodo. https://doi.org/10.5281/zenodo.20618338

@software{RCT-Reviewer,
  author    = {Sahu, V.},
  title     = {RCT-Reviewer: A Modernized, Standalone Tool for Automated Analysis of Clinical Trials (RCTs)},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.20618338},
  url       = {https://doi.org/10.5281/zenodo.20618338}
}

Original RobotReviewer:

Marshall IJ, Kuiper J, Banner E, Wallace BC. “Automating Biomedical Evidence Synthesis: RobotReviewer.” Proceedings of the Conference of the Association for Computational Linguistics (ACL). 2017 (July): 7–12.

@article{RobotReviewer2017,
  title    = "Automating Biomedical Evidence Synthesis: {RobotReviewer}",
  author   = "Marshall, Iain J and Kuiper, Jo{\"e}l and Banner, Edward and Wallace, Byron C",
  journal  = "Proceedings of the Conference of the Association for Computational Linguistics (ACL)",
  volume   = 2017,
  pages    = "7--12",
  month    = jul,
  year     = 2017,
}

This project is a derivative work of RobotReviewer and is distributed under the GNU GENERAL PUBLIC LICENSE v3.0.