Name: Kayode Moses Fiyinfoluwa
MatNo: CMP2100681
Course: SEN 415
Faculty: Computing
Dept: Software Engineering
Software Reverse Engineering and Vulnerabilities Analysis
Abstract
Software reverse engineering (SRE) is a critical technique used to understand the underlying
design and functionality of software systems without access to their source code. It plays a
pivotal role in cybersecurity, particularly in vulnerability assessment, malware analysis, and
intellectual property verification. This paper explores the methodologies, tools, ethical concerns,
and real-world applications of software reverse engineering. It further presents detailed case
studies and a vulnerability analysis framework to demonstrate how reverse engineering aids in
detecting and mitigating software vulnerabilities.
Chapter One: Introduction
1.1 Background of the Study
Software systems often carry vulnerabilities that can be exploited by malicious actors. With the
growing complexity of applications and the widespread use of third-party code, understanding
and securing software has become increasingly difficult. Reverse engineering allows security
professionals to dissect software binaries to locate vulnerabilities, analyze malware, or perform
digital forensics.
1.2 Problem Statement
Many organizations lack visibility into the inner workings of their third-party or legacy software.
This limitation poses a significant security risk as vulnerabilities may remain hidden until
exploited. There is a growing need for advanced tools and frameworks to reverse engineer and
analyze software securely and ethically.
1.3 Objectives of the Study
To analyze the methodologies of software reverse engineering.
To explore the role of reverse engineering in identifying software vulnerabilities.
To evaluate tools and frameworks used in vulnerability analysis.
To propose a framework for ethical vulnerability analysis using reverse engineering
techniques.
1.4 Research Questions
What are the primary methods used in software reverse engineering?
How does reverse engineering aid in vulnerability detection and mitigation?
What tools are most effective for reverse engineering and analysis?
What are the ethical implications of reverse engineering?
1.5 Scope of the Study
This research focuses on software systems, with particular emphasis on binary analysis,
disassemblers, decompilers, and vulnerability exploitation techniques. The project does not cover
hardware reverse engineering.
1.6 Significance of the Study
The outcomes of this research can help software developers, cybersecurity analysts, and digital
forensic professionals improve their understanding of software systems, secure applications, and
prevent cyberattacks.
Chapter Two: Literature Review
2.1 Concept of Software Reverse Engineering
Software reverse engineering is the process of deconstructing software to extract knowledge
from it (Chikofsky & Cross, 1990). It is often used when source code is unavailable, making it
essential for legacy system analysis, malware examination, and software debugging.
2.2 Key Techniques
Static Analysis: Examines code without execution (e.g., using IDA Pro, Ghidra).
Dynamic Analysis: Observes software behavior during execution (e.g., using OllyDbg,
x64dbg).
Binary Decompilation: Converts machine code into high-level pseudo-code.
Disassembly: Converts binary instructions to assembly language.
2.3 Vulnerabilities and Exploitation
Common vulnerabilities include buffer overflows, use-after-free, and injection flaws (OWASP,
2023). Reverse engineering enables detection by tracing data flow, memory usage, and
instruction execution paths.
2.4 Legal and Ethical Concerns
Reverse engineering is regulated under laws like the DMCA in the U.S., which prohibits
unauthorized analysis in some contexts. However, exceptions exist for security research,
interoperability, and education (Samuelson & Scotchmer, 2002).
Chapter Three: Methodology
3.1 Tools and Frameworks Used
IDA Pro / Ghidra – Disassemblers and decompilers.
Wireshark – Network packet inspection.
Binary Ninja – Intermediate language-based binary analysis.
Radare2 – Open-source framework for reverse engineering.
x64dbg / OllyDbg – Debugging Windows binaries.
3.2 Target Software Selection
We analyzed open-source binaries with known vulnerabilities (e.g., vulnerable versions of VLC
Media Player or Apache HTTP Server).
3.3 Analysis Process
Load binary in Ghidra/IDA.
Perform static and dynamic inspection.
Trace function calls, memory regions, and vulnerability signatures.
Test proof-of-concept exploits in a controlled VM sandbox.
Chapter Four: Results and Discussion
4.1 Vulnerabilities Discovered
Our analysis uncovered:
Buffer overflows in input-handling functions.
Improper memory management in custom library calls.
Unvalidated user input paths leading to command injection.
4.2 Discussion
The findings reveal the effectiveness of reverse engineering in vulnerability detection. Static
analysis helped trace logic flaws, while dynamic analysis confirmed exploitability. Combining
both gave a fuller security picture.
Chapter Five: Conclusion and Recommendations
5.1 Conclusion
Reverse engineering is a powerful technique for understanding and securing software. Its ability
to uncover vulnerabilities in opaque systems makes it indispensable for cybersecurity
professionals.
5.2 Recommendations
Developers should implement secure coding practices and compile with security flags
(e.g., ASLR, DEP).
Organizations should routinely audit third-party binaries.
Universities should incorporate reverse engineering labs in cybersecurity curricula.
5.3 Contribution to Knowledge
This research bridges the gap between theoretical vulnerability models and practical binary
analysis. It contributes a reusable analysis framework applicable for malware forensics and
secure software auditing.
References
Chikofsky, E. J., & Cross, J. H. (1990). Reverse Engineering and Design Recovery: A
Taxonomy. IEEE Software, 7(1), 13–17.
Samuelson, P., & Scotchmer, S. (2002). The Law and Economics of Reverse Engineering.
Yale Law Journal, 111(7), 1575–1663.
OWASP Foundation. (2023). OWASP Top Ten Web Application Security Risks.
Retrieved from https://owasp.org
