Stock Price and News Sentiment Analysis
By Anuj, Archita and Dimple (Mandelbulb Technologies)
Project Overview
Our project is a real-time analytics platform that tracks US news events and their impact on
stocks. Using advanced NLP algorithms, it analyzes news articles and social media posts to
gauge market sentiment, empowering investors to make informed trading decisions.
This project aims to analyze the sentiment of US news related to 10 US stocks and potentially
correlate it with the stock prices. It is divided into two phases:
Phase 1: Stock Price Data Ingestion
1.1.Data Source: Twelve Data API (https://twelvedata.com)
• Twelve Data offers a programmatic interface (API) for accessing various
financial data, including real-time and historical stock prices.
• You'll need to sign up for a Twelve Data account and obtain an API key to
access their service.
• The API allows you to specify the stocks you're interested in and the desired
data points (e.g., opening price, closing price, volume).
1.2.Data Fetching: Azure Logic Apps
• Azure Logic Apps is a cloud-based service for automating workflows by
connecting to various Azure services and third-party APIs.
• In this phase, you'll create a Logic App that:
o Triggers at regular intervals (e.g., every hour) using a built-in scheduler.
o Uses an HTTP connector action to call the Twelve Data API with the
desired stock information.
o Parses the response from the API to extract the relevant stock price data.
1.3.Data Streaming: Event Hub (Azure)
• Event Hub is a highly scalable real-time data ingestion service in Azure.
• It acts as a buffer, decoupling data producers (Logic Apps) from data
consumers (KQL database) and handling high volumes of data streams.
• The Logic App will send the fetched stock price data as events to the
configured Event Hub in your Azure subscription.
1.4. Data Streaming with Eventstream
• Data Source: Fetching Data from twelvedata API using logic apps
• Event Hub: This score is likely published to the Azure Event Hub using libraries like
azure-eventhub-py. The Event Hub acts as a buffer, decoupling the data producer
(Python notebook) from the data consumer (Eventstream).
• Eventstream: An Eventstream within Microsoft Fabric is configured to consume data
from the Azure Event Hub.
• Event Transformation: (Optional) As mentioned earlier, you can utilize Eventstream's
transformation capabilities to process the data further before sending it to KQL.
• Event Destination: The Eventstream is configured with a destination linked to the
KQL database. This specifies the target KQL table where the combined sentiment
scores will be stored.
1.5. Data Storage: KQL Database (Microsoft Fabric)
• KQL Database, powered by Kusto Query Language (KQL), is a service
optimized for storing, analyzing, and visualizing large volumes of time-series
data.
• The streamed stock price data from the Event Hub is ingested into the KQL
database.
• You can use KQL to efficiently query, analyze, and visualize the historical
stock price data for further insights.
Phase 2: News Sentiment Analysis and Integration
2.1. Data Source: MarketAux API ([https://marketaux.com])
• MarketAux provides an API for accessing real-time and historical market news.
• Similar to Twelve Data, you'll need a MarketAux API key to retrieve relevant news
articles.
• You can configure the API to filter news based on keywords related to the 10 US
stocks you're tracking.
2.2. Data Processing: Python notebook in Microsoft Fabric
• Microsoft Fabric is a platform for developing and deploying dataflows.
• In this project, you'll create a Python notebook that runs within the Fabric
environment.
• The notebook will:
o Use Python libraries to interact with the MarketAux API.
o Fetch news articles relevant to the tracked stocks based on the configured
keywords.
2.3. Sentiment Analysis Tools:
• Sentiment analysis involves determining the emotional tone (positive, negative, or
neutral) of a text piece.
• This project utilizes three different tools for a more comprehensive analysis:
o finBERT: This pre-trained model is specifically designed for understanding
financial text, potentially offering better accuracy for news related to stocks.
o RoBERT: A robust general-purpose model trained on a massive dataset of text and
code, offering a broader understanding of language.
o OpenAI (ChatGPT): A large language model known for its ability to interpret
complex language nuances, potentially providing additional insights.2.4. Score
Combination:
o Each sentiment analysis tool assigns a score (positive, negative, or
neutral) to each news article.
o This project employs a weighted average to combine the scores from
different tools:
▪ finBERT: 20% weight - Capitalizes on its financial text expertise.
▪ RoBERT: 20% weight - Leverages its general understanding of
language.
▪ OpenAI: 60% weight - Places greater emphasis on its ability to
capture subtle nuances.
2.5. Data Sending: Custom App in Microsoft Fabric
• A custom app developed within Microsoft Fabric is responsible for sending the
combined sentiment score for each news article.
• This custom app likely utilizes Python libraries to format and send the data efficiently.
2.6. Data Streaming with Eventstream:
Eventstream is a component within Microsoft Fabric specifically designed for
real-time data streaming and transformation.
In this project, an Eventstream is used to stream the combined sentiment score
data.
• Event Source: The Python notebook acting as the data source generates
the combined sentiment score for each news article.
• Event Transformation: (Optional) You can utilize Eventstream's built-in
transformation capabilities within the event processor node to further
refine the data before sending it to KQL. For example, you could add
timestamps or format the data in a specific way.
• Event Destination: The final step involves configuring an Eventstream
destination linked to the KQL database. This destination specifies the
target KQL table where the data will be stored.
2.7. Data Storage: KQL Database (Microsoft Fabric)
• The combined sentiment scores, along with timestamps, are streamed into the KQL
database alongside the corresponding stock price data.
• This allows you to analyze and potentially identify correlations between news
sentiment and stock price movements.
Further Analysis and Applications
• The combined sentiment data and stock prices can be used for various analyses,
including:
• Exploring potential correlations between news sentiment and stock price movements.
• Identifying news events that may have a significant impact on specific stocks.
• Building machine learning models to predict stock price trends based on sentiment
analysis.
Tools and Technologies
• Twelve Data API
• Azure Logic Apps
• Azure Event Hub
• KQL Database
• Microsoft Fabric
• Python
• finBERT
• RoBERT
• OpenAI (ChatGPT)
