Department of Artificial Intelligence and Data Science
Phishing URL Detection using the DRL
algorithms with knowledge infusion
GUIDED BY : PRESENTED BY :
Mrs.S.Priya ARKAT SREEKANTH (621721243005)
Assistant Professor (CSE)
KAMINENI LAKSHMIDHAR (621721243025)
KOSURI BHARGAV KUMAR (621721243028)
PULI SAI CHARAN (621721243042)
PROBLEM STATEMENT
• Over 3 million phishing URLs are created every month, with attackers constantly evolving their strategies to
bypass traditional detection systems (Google Transparency Report, 2024).
• Existing static supervised models lack adaptability and struggle to generalize across newly crafted or low-
reputation URLs.
• Less than 15% of detection systems integrate real-time user feedback or external threat intelligence (e.g.,
Virus Total) into their classification loop.
• This highlights a critical gap in interactive, reinforcement-driven models that can learn continuously and
refine predictions on-the-fly.
ABSTRACT
• This research proposes a Reinforcement Learning (RL) approach using Proximal Policy Optimization
(PPO) for multi-class URL classification, targeting Harmless (0), Suspicious (1), and Malicious (2)
categories.
• The model leverages rich feature engineering, incorporating VirusTotal analytics, WHOIS data, Shannon
entropy, and URL structure metrics, followed by feature normalization to enhance learning efficiency.
• A custom Gym environment is developed where each observation (URL feature vector) is associated with a
discrete classification action, and the agent receives a reward signal (+1/-1) based on the correctness of the
prediction.
• Unlike static Supervised Learning (SL) methods such as Random Forest, SVC, and XGBoost, the PPO
model supports real-time inference with online feedback integration, enabling it to adapt dynamically in
production environments.
• Experimental results show that the RL model achieves 98% accuracy with competitive F1-scores across all
classes and significantly outperforms SL models in terms of interactivity, robustness, and adaptability to
unseen threats.
EXISTING SYSTEM
Traditional URL classification systems heavily rely on supervised learning models such as Random Forest, SVM,
or XGBoost, which perform well on static, labeled datasets but lack adaptability in evolving threat landscapes.
Recent deep learning (DL) approaches primarily use Convolutional Neural Networks (CNNs) to learn URL text
patterns, but these models often treat URLs as sequences without incorporating semantic or contextual threat
intelligence (like WHOIS or VirusTotal data).
Most existing solutions are offline, requiring retraining with labeled data and do not adapt to live user feedback
or environmental changes post-deployment.
Additionally, current models fail to dynamically adjust to newly emerging, low-reputation domains, especially
when feature distribution shifts in real-time scenaris.
Our project directly addresses these limitations by implementing a Reinforcement Learning-based system
(PPO) that uses reward shaping, API-derived intelligence, and an interactive feedback.
DISADVANTAGE
Existing models rely heavily on static supervised learning, making them ineffective when faced with new or
evolving URL threats that were not present in the training data.
Most systems lack real-time adaptability, meaning they cannot adjust predictions based on user feedback or
integrate updated threat intelligence dynamically.
Deep learning models, particularly those based on CNNs, often treat URLs as mere character sequences and
fail to leverage semantic context or external threat sources like VirusTotal or WHOIS data.
There is a significant absence of feedback loops in traditional approaches, preventing them from learning
from false positives or misclassifications post-deployment.
Many models are optimized solely for offline accuracy metrics, resulting in poor robustness and
generalization to zero-day attacks or domain spoofing scenarios in real environments.
PROPOSED SYSTEM
Reinforcement Learning-Based Detection
Real-Time Feedback Integration
Multi-Class Threat Classification
Threat Intelligence Augmentation
Adaptive Learning Environment
ADVANTAGES
Continuously Learns from Feedback
Adapts to New Threats
Real-Time URL Analysis
Integrates External Intelligence
Minimizes False Predictions
BLOCK DIAGRAM:
HARDWARE REQUIREMENTS
Computer or Laptop – For model development and implementation.
Processor – Intel i5/i7 or AMD Ryzen 5/7 for efficient computation.
RAM – Minimum 8GB (16GB recommended) for handling large datasets.
Storage – At least 256GB SSD for faster data processing.
Internet Connection – Required for data collection, API integration, and model deployment.
SOFTWARE
REQUIREMENTS
Python 3.8+ : Primary language for implementation and scripting.
Stable-Baselines3: Library for Proximal Policy Optimization (PPO) reinforcement learning.
Gymnasium: For building custom RL environments.
Scikit-learn: Used for preprocessing, feature scaling, and supervised model benchmarking.Requests and
WHOIS: To fetch real-time threat intelligence from VirusTotal and domain registrars.
MODULES
Data Preprocessing Module
– Handles cleaning, feature extraction (length, entropy, WHOIS), and normalization.
Threat Intelligence Module
– Integrates VirusTotal API and WHOIS data to enrich each URL with contextual metadata.
RL Environment Module
– Defines custom Gym environment, actions (0, 1, 2), reward logic, and state transitions.
PPO Agent Module
– Implements and trains the Proximal Policy Optimization model to classify URLs.
Feedback and Evaluation Module
– Allows user feedback, updates reward logic, and provides real-time performance evaluation.
OUTPUT
FUTURE ENHANCEMENT
Real-Time Deployment at Scale: Integrate the PPO model into a live browser extension or network
gateway for proactive threat detection.
Active Learning Loop: Incorporate an automated feedback retraining mechanism where high-confidence
user input updates the policy continuously.
Multi-Modal Feature Expansion: Combine URL features with HTML, DNS, and SSL certificate analysis
to further strengthen detection accuracy.
Federated Threat Intelligence: Enable secure, privacy-preserving collaboration across organizations to
share real-time malicious indicators.
Explainable RL Outputs: Add interpretability layers to help cybersecurity analysts understand why a URL
was marked suspicious or malicious.
CONCLUSION
Developed a pure PPO-based reinforcement learning model for multi-class URL classification: Harmless,
Suspicious, Malicious.
Integrated real-time feedback and external intelligence from VirusTotal and WHOIS to enhance decision
accuracy.
Outperformed static supervised models in terms of adaptability, resilience, and real-time threat handling.
Demonstrated robust training performance with structured reward signals and policy optimization
techniques.
Enabled live URL prediction and user interaction, paving the way for self-improving cybersecurity systems.
REFERENCES
M. Bhattacharya et al., "Random Forest for Phishing Detection," IEEE Access, 2020.
Y. Li et al., "Deep Reinforcement Learning for Cyber Anomaly Detection," IEEE TNNLS, 2022.
S. Khanzadeh, E.C.P. Neto, S. Iqbal, M. Alalfi, S. Buffett, "An Exploratory Study on Domain Knowledge
Infusion in Deep Learning for Automated Threat Defense," Published Online: 28 Jan 2025.
© Crown 2025.C. Zhang, P. Sun, Y. Fu, "RL4Security: Reinforcement Learning for Cybersecurity," arXiv
preprint arXiv:2103.06665, 2021.
L. Li, D. Wu, "Malicious URL Detection via Deep Learning and Feature Fusion," Computers &
Security, Vol. 114, 2022.
THANK
YOU
