INFORMATION AND TECHNOLOGY AND CYBER

SECURITY

LAB - 2

LOG MANAGMENT
(22/01/25)

Submitted to : Dr. Keshav Sinha

Submitted By: Prasha Singh

B-4

500125215
 LOGS

Logs serve as invaluable records of past events, providing essential insights to address these
questions. By preserving an archive of historical activities, we can bolster our security posture and
protect our digital assets more effectively.

Log analysis tools and methods empower individuals to interpret historical events and establish a
reliable source of historical evidence, streamlining the processing and scrutiny of log data. This
efficiency facilitates prompt detection and response to potential incidents or significant events.

WORKING WITH LOGS

- Connecting to TryHackMe Machine:
 Intro to Logs
Task-1

Logs serve as invaluable records of past events, providing essential insights to address these
questions. By preserving an archive of historical activities, we can bolster our security posture and
protect our digital assets more effectively.
Logs are a record of events within a system. These records provide a detailed account of what a
system has been doing, capturing a wide range of events such as user logins, file accesses, system
errors, network connections, and changes to data or system configurations.
While the specific details may differ based on the type of log, a log entry usually includes the
following information:
• A timestamp of when an event was logged

• The name of the system or application that generated the log entry

• The type of event that occurred

• Additional details about the event, such as the user who initiated the event or the device's IP
address that generated the event

This information is typically stored in a log file, which contains aggregated entries of what occurred
at any given time on a system.

Log Types
Specific log types can offer a unique perspective on a system's operation, performance, and security.
While there are various log types, we will focus on the most common ones that cover approximately
80% of the typical use cases.

• Application Logs: Messages about specific applications, including status, errors, warnings,
etc.

• Audit Logs: Activities related to operational procedures crucial for regulatory compliance.

• Security Logs: Security events such as logins, permissions changes, firewall activity, etc.

• Server Logs: Various logs a server generates, including system, event, error, and access logs.

• System Logs: Kernel activities, system errors, boot sequences, and hardware status.
• Network Logs: Network traffic, connections, and other network-related events.

• Database Logs: Activities within a database system, such as queries and updates.

• Web Server Logs: Requests processed by a web server, including URLs, response codes, etc.

Log Formats
A log format defines the structure and organisation of data within a log file. It specifies how the data
is encoded, how each entry is delimited, and what fields are included in each row. These formats can
vary widely and may fall into three main categories: Semi-structured, Structured, and Unstructured.
We'll explore these categories and illustrate their usage with examples.

Semi-structured Logs: These logs may contain structured and unstructured data, with predictable
components accommodating free-form text. Examples include: Syslog Message Format: A widely
adopted logging protocol for system and network logs.

Windows Event Log (EVTX) Format: Proprietary Microsoft log for Windows systems.

Structured Logs: Following a strict and standardised format, these logs are conducive to parsing and
analysis. Typical structured log formats include:
Field Delimited Formats: Comma-Separated Values (CSV) and Tab-Separated Values (TSV) are
formats often used for tabular data.
JavaScript Object Notation (JSON): Known for its readability and compatibility with modern
programming languages.
W3C Extended Log Format (ELF): Defined by the World Wide Web Consortium (W3C), customizable
for web server logging. It is typically used by Microsoft Internet Information Services (IIS) Web
Server.
eXtensible Markup Language (XML): Flexible and customizable for creating standardized logging
formats.

• Unstructured Logs: Comprising free-form text, these logs can be rich in context but may
pose challenges in systematic parsing. Examples include:

• NCSA Common Log Format (CLF): A standardized web server log format for client requests.
It is typically used by the Apache HTTP Server by default.

• NCSA Combined Log Format (Combined): An extension of CLF, adding fields like referrer and
user agent. It is typically used by Nginx HTTP Server by default.

Log Collection

• Log collection is an essential component of log analysis, involving the aggregation of logs
from diverse sources such as servers, network devices, software, and databases.
• For logs to effectively represent a chronological sequence of events, it's crucial to maintain
the system's time accuracy during logging. Utilising the Network Time Protocol (NTP) is a
method to achieve this synchronisation and ensure the integrity of the timeline stored in the
logs.
 • Time synchronisation can be performed automatically on Linux-based systems or manually
initiated by executing. ntpdate pool.ntp.org.

Practical Activity: Log Collection with rsyslog

This activity aims to introduce rsyslog and demonstrate how it can enhance the centralisation and
management of logs. As part of the collection process, we will configure rsyslog to log all sshd
messages to a specific file, such as /var/log/websrv-02/rsyslog_sshd.log. The steps below can be
followed to achieve this:

1. Open a Terminal.

2. Ensure rsyslog is Installed: You can check if rsyslog is installed by running the command: sudo
systemctl status rsyslog

3. Create a Configuration File: Use a text editor to create the following configuration file: gedit
/etc/rsyslog.d/98-websrv-02-sshd.conf, nano /etc/rsyslog.d/98-websrv-02-sshd.conf, vi
/etc/rsyslog.d/98-websrv-02-sshd.conf, or vim /etc/rsyslog.d/98-websrv-02-sshd.conf

4. Add the Configuration: Add the following lines in /etc/rsyslog.d/98-websrv-02-sshd.conf to

direct the sshd messages to the specific log file.

5. Save and Close the Configuration File.

6. Restart rsyslog: Apply the changes by restarting rsyslog with the command: sudo systemctl
restart rsyslog

7. Verify the Configuration: You can verify the configuration works by initiating an SSH connection
to localhost via ssh localhost or by checking the log file after a minute or two.
 Log Storage

Logs can be stored in various locations, such as the local system that generates them, a centralised
repository, or cloud-based storage.

Log Deletion

Log deletion must be performed carefully to avoid removing logs that could still be of value. The
backup of log files, especially crucial ones, is necessary before deletion.
 Practical Activity: Log Management with logrotate

This activity aims to introduce logrotate, a tool that automates log file rotation, compression, and
management, ensuring that log files are handled systematically. It allows automatic rotation,
compression, and removal of log files. As an example, here's how we can set it up for
/var/log/websrv-02/rsyslog_sshd.log:
1. Create a Configuration File:

2. Define Log Settings:

3. Save and Close the file.

4. Manual Execution: sudo logrotate -f /etc/logrotate.d/98-websrv-02_sshd.conf

Log Analysis Tools

Security Information and Event Management (SIEM) tools such as Splunk or Elastic Search can
be used for complex log analysis tasks.
• Use awk and sed to normalize the log entries to the desired format. For this example, we will
sort by date and time:
• Optional: Use grep to filter specific entries:
• Use sort to sort all the log entries by date and time:
• Use uniq to remove duplicate entries:

TASK-2
TASK-3

TASK-4

TASK-5

TASK-6