18AIC402J & DEEP LEARNING MODELS AND AI ANALYST

UNIT 1 -PREFACE TO ML-DL MODELS & IBM CLOUD

OVERVIEW OF AI:

➢Artificial intelligence (AI) refers to the simulation of human intelligence in machines

that are programmed to think and act like humans. The goal of AI is to develop systems

that can perform tasks that typically require human intelligence, such as visual perception,

speech recognition, decision-making, and language understanding.

Why AI:

❖ To create such software or devices which can solve real-world problems such as health

issues, marketing, traffic issues, etc.

❖ To can create your personal virtual Assistant, such as Cortana, Google Assistant, Siri, etc.

❖ Reduces Man power.

1. Efficiency: AI systems can automate repetitive tasks, saving time and resources for
businesses and individuals.

2. Decision Making: AI can analyze vast amounts of data to make informed decisions
quickly and accurately, leading to better outcomes.

3. Personalization: AI enables personalized experiences by understanding and adapting to

individual preferences and behavior.

4. Innovation: AI drives innovation by enabling the development of new products, services,

and solutions that were previously impossible or impractical.

5. Problem Solving: AI can tackle complex problems in various domains, from healthcare to

finance, by leveraging advanced algorithms and computational power.

 INTRODUCTION OF MACHINE LEARNING:

 Machine learning (ML) is a subset of artificial intelligence (AI) that focuses on developing
 algorithms and statistical models that enable computers to learn from and make
predictions or
 decisions based on data, without being explicitly programmed to perform specific tasks.
 In other words, ML algorithms allow computers to learn patterns and relationships from
data and use that knowledge to make informed decisions or predictions.

MACHINE LEARNING WORKING PROCESS

• Data Collection: Gathering relevant data for your problem is crucial. This data could be

structured, like databases, or unstructured, like text or images. The quality and quantity

of your data greatly influence the success of your model.

• Data Preprocessing: Once you have your data, you need to clean and preprocess it.

This involves handling missing values, dealing with outliers, normalizing or

standardizing the data, and often converting it into a format suitable for the chosen

machine learning algorithm.

• Model Selection: Choosing the right machine learning algorithm or model architecture

depends on the nature of your problem and data. It could be a decision tree, neural

network, support vector machine, or another algorithm.

• Training the Model: This is where you use your prepared data to train the selected

model. During training, the model learns to make predictions by adjusting its internal

parameters based on the input data.

• Model Evaluation: Once the model is trained, you need to evaluate its performance

using validation data or techniques like cross-validation. This step helps you

understand how well your model generalizes to unseen data and whether it's

performing as expected.

• Hyperparameter Tuning: Many machine learning algorithms have parameters that are

not learned during training but are set beforehand. Tuning these hyperparameters can

significantly impact the performance of the model.

• Model Deployment: After you're satisfied with the model's performance, you deploy it

to make predictions on new, unseen data. Deployment could involve integrating the

model into a software application, a web service, or any other relevant system.

• Monitoring and Maintenance: Once deployed, it's essential to monitor the model's

performance over time and update it as needed. Data distributions may change,

requiring retraining or recalibration of the model to ensure it continues to make

accurate predictions.

Introduction to Scikit-learn:

Scikit-learn is a popular open-source machine learning library for Python. It provides

simple and efficient tools for data mining and data analysis, built on top of other Python
libraries like NumPy, SciPy, and Matplotlib. Scikit-learn is designed to be user-friendly and

accessible, making it a valuable resource for both beginners and experienced practitioners in

the field of machine learning.

Installation Steps:

To install scikit-learn using pip, you can follow these steps:

1. Open Jupyter Notebook Environment

2. Create a New Notebook (Optional):

3. Install scikit-learn:

In a code cell within the notebook, type the following command:

pip install scikit-learn

pip is the Python package installer.

üscikit-learn is the name of the package you want to install.

4. Run the cell by pressing Shift + Enter.

Importing scikit-learn

To import scikit-learn, you typically use the following import statement:

import sklearn

Purpose of Import Statement:

➢ The import statement in Python is used to load and make available external code (modules
or packages) to your current Python script or Jupyter Notebook.

➢ It allows you to access classes, functions, and other objects defined in the imported
module.

Machine Learning Tasks That Scikit-learn Supports

➢ Classification - Examples: Spam detection, image recognition, sentiment analysis.

➢ Regression - Examples: House price prediction, stock price forecasting.

➢ Clustering - Examples: Customer segmentation, anomaly detection.

➢ Preprocessing - Examples: Scaling features, handling missing values, encoding categorical

variables.

➢ Image Processing - Examples: Image classification, object detection.

➢ Recommendation Systems - Examples: Movie recommendations, product

recommendations.

MACHINE LEARNING:

➢ Data-driven: Machine learning algorithms learn patterns and relationships from

data rather than relying on predefined rules. They automatically learn from

examples and adjust their internal parameters to improve performance.

➢ Probabilistic: Machine learning models make predictions based on probabilities

rather than deterministic rules. They provide a degree of confidence or

uncertainty in their predictions.

➢ Less Reliant on Explicit Instructions: While developers still play a crucial role

in designing and training machine learning models, they don't need to explicitly

program every rule. Instead, they focus on selecting and preparing data, choosing

appropriate algorithms, and fine-tuning model parameters.

➢ Generalization: Machine learning models can generalize well to new or unseen

data if they are trained on representative samples and designed properly. They

can learn complex patterns and make predictions on data that they haven't

encountered during training.

Training And Testing Data

Training Data:

➢ Used to train the machine learning model to learn patterns and relationships in the data.

➢ Consists of a set of input-output pairs, where the model learns from the input features to
predict the corresponding output.

Testing Data:

➢ Used to evaluate the performance of the trained model on unseen data.

➢ Assesses how well the model generalizes to new, unseen examples.

Code to Split a Dataset into Training And Testing Sets Using Scikit-learn

import numpy as np

from sklearn.model_selection import train_test_split

# Sample dataset

df = pd.read_csv('path_to_your_dataset.csv')

X = df.loc(columns=[‘Input columns’])

y = df.loc(['target_column’])

# Split the dataset into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

Explanation

➢ X: Contains the feature data (input variables).

➢ y: Contains the target data (output variable to predict).

➢ train_test_split: This function from sklearn.model_selection module splits arrays or

matrices into random train and test subsets.

➢ test_size: Specifies the proportion of the dataset to include in the testing split. Here,

test_size=0.2 means 20% of the data will be used for testing, and the remaining 80% for

training.

➢ X_train, X_test: The resulting training and testing sets for features.

➢ y_train, y_test: The corresponding target (labels) for the training and testing sets.
Fit() Method

The fit() method is used to train (fit) the machine learning model on the training data.

✓ It learns the patterns and relationships present in the training data to make predictions.

Syntax: model.fit(X_train, y_train)

X_train: The features (input variables) of the training dataset.

y_train: The target variable (output) of the training dataset.

predict() Method:

✓ The predict() method is used to make predictions with the trained model on new, unseen
data.

✓ It takes the features of the new data as input and returns the predicted target values.

Syntax: y_pred = model.predict(X_test)

X_test: The features (input variables) of the testing dataset or new data.

y_pred: The predicted target values corresponding to the input features.

TYPES OF ML
 SUPERVISED LEARNING

➢ In which an algorithm learns from labeled training data to make predictions or take actions

based on input features.

➢ In supervised learning, the goal is to train a model that can accurately map input features
to the

desired output.

Supervised learning can be grouped further in two categories of algorithms:

1. Classification

2. Regression

Classification(Defined Label)

➢ The model is trained in such a way that the output data is separated into different labels (or

categories) according to the given input data.

➢ Output variable will be assigned to a category is Discrete Value.

For Example:

1.To Predict the customer is eligible for getting loan?

o/p: yes or no

2.Predict team India will win or loss?

o/p: win or loss / yes or no

REGRESSION

➢ Regression goal is to predict a continuous or numerical output value based on input

features or variables.

➢ In regression, the output variable is not limited to a set of predefined classes

Examples:

1.To predict the whether for next 24 hours?

o/p: Continues value depends on temperature

2.Predict share price?

o/p: Continues numeric range not exacted.

 OVERFITTING AND UNDER FITTING

Overfitting:

➢ This occurs when a model learns the training data too well, including noise and random

fluctuations, to the extent that it performs poorly on new, unseen data.

➢ Overfitting typically happens when the model is too complex relative to the amount of

training data available.

Underfitting:

➢ This happens when a model is too simple to capture the underlying structure of the data.

➢ It fails to learn the patterns in the training data and thus performs poorly on both the
training and new data.

➢ Signs of underfitting include low accuracy on both the training and validation or test data.
 SUPERVISED ML – ALGORITHMS

1. Logistics Regression Algorithm

2. K- Nearest Neighbour Algorithm

3. Support Vector Machine (SVM) Algorithm

4. Naïve Bayes Classifier

5. Decision Tree

6. Random forest

UNSUPERVISED LEARNING

Unsupervised learning is a type of machine learning where the algorithm is trained on

unlabeled data. The goal is to find hidden patterns, groupings, or features in the data without
any predefined labels or categories.

Key Concepts:

1. Clustering:

o K-Means Clustering: Divides the data into K clusters by minimizing the

variance within each cluster.

o Hierarchical Clustering: Builds a tree of clusters by either merging or

splitting existing clusters.
 o DBSCAN (Density-Based Spatial Clustering of Applications with Noise):
Groups together points that are closely packed and marks points in low-density
regions as outliers.

2. Dimensionality Reduction:

o Principal Component Analysis (PCA): Reduces the number of dimensions

by transforming data into a new coordinate system with axes that maximize
variance.

o t-Distributed Stochastic Neighbor Embedding (t-SNE): Reduces

dimensions while preserving the structure and distances between points, often
used for visualization.

3. Anomaly Detection:

o Identifies outliers or unusual data points that do not fit the expected patterns.

Applications:

 Customer segmentation for marketing.

 Anomaly detection in network security.

 Gene expression analysis in bioinformatics.

 Document clustering for topic modeling.

REINFORCEMENT LEARNING

Reinforcement learning (RL) is a type of machine learning where an agent learns to make
decisions by performing actions in an environment to maximize cumulative rewards. The
agent interacts with the environment, receives feedback in the form of rewards or penalties,
and adjusts its actions accordingly.
Key Concepts:

1. Agent: The learner or decision-maker.

2. Environment: The external system with which the agent interacts.

3. State: A representation of the current situation or configuration of the environment.

4. Action: A decision or move made by the agent that affects the environment.

5. Reward: Feedback from the environment indicating the success or failure of an

action.

6. Policy: A strategy used by the agent to determine actions based on the current state.

7. Value Function: A function that estimates the expected cumulative reward of being in
a state and following a policy.

Types of Reinforcement Learning:

1. Model-Free RL: The agent learns a policy directly without modeling the
environment.

o Q-Learning: A value-based method where the agent learns a Q-value for each
state-action pair.

o Deep Q-Networks (DQN): Uses neural networks to approximate Q-values for

high-dimensional state spaces.

2. Model-Based RL: The agent builds a model of the environment and uses it to plan
actions.

Applications:

 Game playing (e.g., AlphaGo, Chess).

 Robotics for learning complex tasks.

 Autonomous driving.

 Financial trading.
Difference between ML -DL

Machine Learning
Aspect Deep Learning (DL)
(ML)

Broad range of
Scope and Specialized in artificial neural
algorithms to learn
Approach networks (ANNs)
from data

Training Data Works well with Requires large amounts of data for
Size smaller datasets effective training

May require domain

Can achieve state-of-the-art
Performance expertise for optimal
performance in many tasks
model selection

Models are often

Models are complex and less
Interpretability interpretable and
interpretable
explainable

Widely used across

Dominates in fields like image
Use Cases various industries and
recognition, NLP, etc.
applications

Regression, SVMs, Convolutional Neural Networks

Examples
Decision Trees (CNNs), Recurrent NNs
Parametric vs Non-Parametric Models

Parametric Models

Definition: Parametric models are those that summarize data with a fixed number of
parameters. They assume a specific form for the function that generates the data, and once
these parameters are determined, the model can predict outcomes for new data points.

Characteristics:

1. Fixed Number of Parameters: The model has a predetermined number of

parameters.

2. Assumption of Distribution: Assumes that the underlying data follows a specific

distribution (e.g., normal distribution).

3. Simpler to Interpret: Generally easier to interpret and understand due to fewer

parameters.
 4. Efficiency: Requires less data to train, and training is typically faster.

Examples:

1. Linear Regression: Assumes a linear relationship between the input variables and the
output variable.

2. Logistic Regression: Used for binary classification, assuming a logistic function.

3. Naive Bayes: Assumes the features are independent given the class.

Advantages:

 Simplicity and ease of interpretation.

 Fewer parameters make them computationally efficient.

 Effective when the assumptions about the data distribution are correct.

Disadvantages:

 Limited flexibility due to strong assumptions about the data.

 Poor performance if the assumptions are incorrect or the model is too simple for the
complexity of the data.

Non-Parametric Models

Definition: Non-parametric models do not assume a specific form for the function that
generates the data. They can adapt to the complexity of the data without predefined
parameters, making them more flexible.

Characteristics:

1. Variable Number of Parameters: The number of parameters can grow with the size
of the dataset.
 2. No Assumption of Distribution: Do not assume any specific distribution for the data.

3. Flexibility: Can model more complex relationships in the data.

4. Data-Driven: Typically require more data to effectively capture patterns and trends.

Examples:

1. K-Nearest Neighbors (KNN): Classifies a data point based on the majority class of
its nearest neighbors.

2. Decision Trees: Splits the data into subsets based on feature values to make
predictions.

3. Random Forest: An ensemble of decision trees to improve prediction accuracy.

4. Support Vector Machines (SVM): Finds the optimal boundary that separates
different classes in the data.

Advantages:

 Flexibility to model complex and unknown data distributions.

 Often better performance when the data does not meet the assumptions of parametric
models.

Disadvantages:

 More computationally intensive and require more memory.

 Prone to overfitting if not properly regularized.

 Harder to interpret compared to parametric models.

Comparison: Parametric vs Non-Parametric Models

Aspect Parametric Models Non-Parametric Models

Strong assumptions about data No assumptions about data

Assumptions
distribution distribution

Parameters Fixed number of parameters Variable number of parameters

Less flexible, limited by More flexible, can adapt to data

Flexibility
assumptions complexity

Interpretability Easier to interpret Harder to interpret

Data
Requires less data Requires more data
Requirements

Computational Generally lower computational

Higher computational cost
Cost cost

Risk of Lower risk of overfitting if Higher risk of overfitting without

Overfitting assumptions are correct proper regularization

Real-Time Example Comparison

Example Scenario: Predicting House Prices

 Parametric Model: Linear Regression

o Assumes a linear relationship between features (size, location, number of

bedrooms) and the house price.

o Simple to understand and interpret.

o Efficient with less data.

 Non-Parametric Model: Decision Tree

o Does not assume a specific form of the relationship between features and the
house price.

o Can capture more complex patterns, such as interactions between features.

o Requires more data and computational resources.

What is a Linear Model?

A linear model is a statistical model that assumes a linear relationship between the input
variables (features) and the single output variable (response). It is one of the simplest and
most commonly used types of models in machine learning and statistics.
Types of Linear Models

1. Simple Linear Regression

2. Multiple Linear Regression

1. Simple Linear Regression

Concept: Simple Linear Regression is used to predict the value of a dependent variable
based on a single independent variable. The relationship between the variables is modeled
using a straight line:

y=β0+β1x+ϵy = \beta_0 + \beta_1 x + \epsilony=β0 +β1 x+ϵ

where:

 yyy is the dependent variable.

 xxx is the independent variable.

 β0\beta_0β0 is the y-intercept.

 β1\beta_1β1 is the slope of the line.

 ϵ\epsilonϵ is the error term.

Real-Time Example: Predicting a person's weight based on their height.

2. Multiple Linear Regression

Concept: Multiple Linear Regression is an extension of Simple Linear Regression where
multiple independent variables are used to predict the dependent variable:

y=β0+β1x1+β2x2+…+βnxn+ϵy = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_n x_n

+ \epsilony=β0 +β1 x1​+β2 x2​+…+βn​xn +ϵ

where:

 yyy is the dependent variable.

 x1,x2,…,xnx_1, x_2, \ldots, x_nx1​,x2​,…,xn​ are independent variables.

 β0\beta_0β0 is the y-intercept.

 β1,β2,…,βn\beta_1, \beta_2, \ldots, \beta_nβ1 ,β2 ,…,βn​ are the coefficients.

 ϵ\epsilonϵ is the error term.

Real-Time Example: Predicting house prices based on features such as size, location, and
number of bedrooms.

Applications of Linear Models

1. Healthcare:

o Predicting patient outcomes based on historical data.

o Estimating medical costs based on patient demographics and medical history.

2. Finance:

o Predicting stock prices based on historical data and market indicators.

 o Credit scoring based on customer financial behavior.

3. Marketing:

o Analyzing the impact of marketing campaigns on sales.

o Customer segmentation and targeting.

4. Real Estate:

o Estimating property prices based on location, size, and other features.

Advantages of Linear Models

 Simplicity: Easy to understand and interpret.

 Efficiency: Computationally efficient and quick to train.

 Applicability: Suitable for a wide range of problems.

Limitations of Linear Models

 Linearity Assumption: Assumes a linear relationship between the dependent and

independent variables, which may not always hold true.

 Sensitivity to Outliers: Outliers can significantly affect the model.

 Multicollinearity: High correlation between independent variables can affect the

stability of coefficient estimates.

INTRODUCTION OF IBM CLOUD

Introduction to IBM Cloud

IBM Cloud is a comprehensive suite of cloud computing services offered by IBM, designed
to support a wide range of applications, including AI, data analytics, IoT, and blockchain. It
provides both public and private cloud environments, enabling businesses to leverage the
power of cloud computing while maintaining control over their data and workloads.

Resources in IBM Cloud

1. Compute Resources:

o Virtual Servers: Scalable and customizable virtual machines.

o Bare Metal Servers: High-performance physical servers for dedicated use.

o Kubernetes Service: Managed Kubernetes clusters for containerized

applications.

o Cloud Functions: Serverless computing for running code in response to

events.

2. Storage Resources:

o Block Storage: High-performance storage for virtual and bare metal servers.

o File Storage: Scalable, shared storage for applications.

o Object Storage: Highly durable storage for unstructured data, accessible via
API.

o Cloud Databases: Managed database services including SQL and NoSQL

databases.

3. Networking Resources:

o Virtual Private Cloud (VPC): Isolated network environments for secure

cloud computing.

o Load Balancers: Distribute traffic across multiple servers for high availability.

o Content Delivery Network (CDN): Accelerate delivery of content to users

globally.

4. AI and Machine Learning:

o Watson AI: AI services including natural language processing, visual

recognition, and more.
 o Machine Learning: Tools for building, training, and deploying machine
learning models.

IBM Cloud Infrastructure

IBM Cloud Infrastructure provides the foundational elements for building and managing
cloud environments. Key components include:

 Data Centers: Globally distributed data centers ensuring low latency and high
availability.

 Bare Metal Servers: Dedicated servers providing high performance and security.

 Virtual Servers: Scalable VMs for various workloads.

 Networking: Robust networking solutions, including VPC, VPN, and Direct Link for
secure connections.

Security in IBM Cloud

IBM Cloud places a strong emphasis on security, offering a range of services and features to
protect data and applications:

 Identity and Access Management (IAM): Secure user access with multi-factor
authentication and role-based access control.
  Data Encryption: End-to-end encryption for data at rest and in transit.

 Security and Compliance: Services to ensure compliance with industry standards

and regulations, such as GDPR, HIPAA, and SOC 2.

 Security Information and Event Management (SIEM): Tools for real-time

monitoring and threat detection.

IBM Cloud Foundry

IBM Cloud Foundry is a platform-as-a-service (PaaS) that provides a runtime environment

for developing and deploying cloud-native applications. Features include:

 Multi-Language Support: Supports various programming languages, including Java,

Node.js, Python, and Ruby.

 Service Integration: Easy integration with IBM Cloud services like databases, AI,
and IoT.

 Scalability: Automatically scales applications based on demand.

IBM Cloud Pak for Data

IBM Cloud Pak for Data is an integrated data and AI platform that helps organizations
collect, organize, and analyze data:

 Data Virtualization: Access and query data across multiple sources without moving
it.
  Data Governance: Ensure data quality, lineage, and security.

 AI and Machine Learning: Build, deploy, and manage AI models using integrated
tools.

 Hybrid Cloud Support: Deploy on any cloud or on-premises environment.

IBM Cloud vs. Amazon Cloud (AWS)

Feature IBM Cloud Amazon Cloud (AWS)

Compute Options Virtual servers, bare metal, Kubernetes, EC2 instances, ECS, Lambda,
serverless Fargate
AI and ML SageMaker, Comprehend,
Watson AI, Machine Learning
Services Rekognition

Storage Solutions Block, file, object storage EBS, EFS, S3

VPC, Direct Connect,

Networking VPC, Direct Link, CDN
CloudFront

IAM, data encryption,

Security IAM, data encryption, SIEM
GuardDuty

Hybrid services available with

Hybrid Cloud Strong focus on hybrid cloud solutions
Outposts

PaaS IBM Cloud Foundry Elastic Beanstalk

Data Platform Cloud Pak for Data Redshift, Aurora, RDS

Cloud Native Storage and Data Services

Cloud native storage and data services are designed to support modern applications that are
deployed in cloud environments, offering scalability, high availability, and seamless
integration.

1. Object Storage:

o Scalable storage for unstructured data.

o Accessible via APIs.

o High durability and availability.

2. Block Storage:

o High-performance storage for virtual machines and databases.

o Snapshot and replication capabilities for data protection.

3. File Storage:

o Shared storage for applications requiring file-level access.

o Scalable and high-performance.

 4. Database Services:

o Managed SQL and NoSQL databases.

o Automated backups, scaling, and maintenance.

o Examples include IBM Db2, Cloudant, and MongoDB.

5. Data Integration:

o Tools for data migration, transformation, and integration.

o Support for ETL processes and data pipelines.

6. Data Governance:

o Ensure data quality, lineage, and security.

o Tools for cataloging and managing data assets.

7. Analytics and AI:

o Platforms for data analytics and AI model development.

o Integration with data storage and processing services.

IBM Cloud Data Service

IBM Cloud Data Services encompass a wide range of cloud-based data solutions designed to
help organizations store, manage, analyze, and secure their data.
 IBM Db2 on Cloud-A fully managed SQL database that offers high availability and
scalability for mission-critical applications.
 IBM Db2 Warehouse- An on-cloud data warehouse service for analytics and big data
workloads.
 IBM Watson Studio-A data science platform that provides tools for data preparation,
modeling, and deployment of machine learning models.