School of Computing Science and Engineering
Course Code : Course Name: Data mining and web Algo
Unit – 1
Data Mining
Faculty Name: Mr. Soumalya Ghosh Program Name: B.Tech CSE
What is Data Mining?
• Data mining is the process of
– extracting knowledge or insights from large amounts of data
• using various statistical and computational techniques.
• The primary goal of data mining
– is to discover hidden patterns and relationships in the data that can be used
to make informed decisions or predictions.
What is Data Mining?
• This involves exploring the data using various techniques such as
– Clustering
– Classification
– regression analysis,
– association rule mining
– anomaly detection.
Data Mining: Applications
• Data mining has a wide range of applications across various industries,
including marketing, finance, healthcare, and telecommunications.
• For example,
– in marketing,
• data mining can be used to identify customer segments and target marketing
campaigns
– in healthcare
• it can be used to identify risk factors for diseases and develop personalized
treatment plans.
Evolution of Database Technology
• The Explosive Growth of Data: from terabytes to petabytes
– Data collection and data availability
• Automated data collection tools, database systems, Web, computerized society
– Major sources of abundant data
• Business: Web, e-commerce, transactions, stocks, …
• Science: Remote sensing, bioinformatics, scientific simulation, …
• Society and everyone: news, digital cameras, YouTube
• We are drowning in data, but starving for knowledge!
• “Necessity is the mother of invention”—Data mining—Automated analysis of massive data
sets
Why it is called Data Mining?
• Simply stated, data mining refers to extracting or “mining” knowledge from
large amounts of data.
• The term is actually a misnomer.
– Remember that the mining of gold from rocks or sand is referred to as gold
mining rather than rock or sand mining.
– Thus, data mining should have been more appropriately named “knowledge
mining from data,” which is unfortunately somewhat long.
– “Knowledge mining,” a shorter term, may not reflect the emphasis on mining
from large amounts of data.
• Thus, such a misnomer that carries both “data” and “mining” became a
popular choice.
Why it is called Data Mining?
• Many other terms carry a similar or slightly different meaning to data
mining, such as
– knowledge mining from data,
– knowledge extraction,
– data/pattern analysis,
– data archaeology
– data dredging
• Many people treat data mining as a synonym for another popularly used
term, Knowledge Discovery from Data, or KDD
• Alternatively, others view data mining as simply an essential step in the
process of knowledge discovery
Data mining as a step in the process of knowledge discovery
• 1. Data cleaning (to remove noise and inconsistent data)
• 2. Data integration (where multiple data sources may be combined)
• 3. Data selection (where data relevant to the analysis task are retrieved from the database)
• 4. Data transformation (where data are transformed or consolidated into forms
appropriate for mining by performing summary or aggregation operations, for instance)
• 5. Data mining (an essential process where intelligent methods are applied in order to
• extract data patterns)
• 6. Pattern evaluation (to identify the truly interesting patterns representing knowledge
• based on some interestingness measures)
• 7. Knowledge presentation (where visualization and knowledge representation techniques
are used to present the mined knowledge to the user)
Knowledge Discovery (KDD) Process
– Data mining—core of
Pattern Evaluation
knowledge discovery
process
Data Mining
Task-relevant Data
Data Warehouse Selection
Data Cleaning
Data Integration
Databases
Difference between KDD and Data Mining
• Although the two terms KDD and Data Mining are heavily used interchangeably,
they refer to two related yet slightly different concepts.
• KDD is the overall process of extracting knowledge from data, while Data Mining
is a step inside the KDD process, which deals with identifying patterns in data.
• And Data Mining is only the application of a specific algorithm based on the
overall goal of the KDD process.
• KDD is an iterative process where evaluation measures can be enhanced, mining
can be refined, and new data can be integrated and transformed to get different
and more appropriate results.
Data Mining and Business Intelligence
Increasing potential
to support
business decisions End User
Decision
Making
Data Presentation Business
Analyst
Visualization Techniques
Data Mining Data
Information Discovery Analyst
Data Exploration
Statistical Summary, Querying, and Reporting
Data Preprocessing/Integration, Data Warehouses
DBA
Data Sources
Paper, Files, Web documents, Scientific experiments, Database Systems
Architecture: Typical Data Mining System
Graphical User Interface
Pattern Evaluation
Knowl
Data Mining Engine edge-
Base
Database or Data Warehouse
Server
data cleaning, integration, and selection
Data World-Wide Other Info
Database Repositories
Warehouse Web
Data Mining: Confluence of Multiple Disciplines
Database
Technology Statistics
Machine Visualization
Data Mining
Learning
Pattern
Recognition Other
Algorithm Disciplines
Data Mining: On What Kinds of Data?
• Database-oriented data sets and applications
– Relational database, data warehouse, transactional database
• Advanced data sets and advanced applications
– Data streams and sensor data
– Time-series data, temporal data, sequence data (incl. bio-sequences)
– Structure data, graphs, social networks and multi-linked data
– Object-relational databases
– Heterogeneous databases and legacy databases
– Spatial data and spatiotemporal data
– Multimedia database
– Text databases
– The World-Wide Web
Data Mining Functionalities
• Multidimensional concept description: Characterization and discrimination
– Generalize, summarize, and contrast data characteristics, e.g., dry vs. wet regions
• Frequent patterns, association, correlation vs. causality
– Diaper Beer [0.5%, 75%] (Correlation or causality?)
• Classification and prediction
– Construct models (functions) that describe and distinguish classes or concepts for
future prediction
• E.g., classify countries based on (climate), or classify cars based on (gas mileage)
– Predict some unknown or missing numerical values
Data Mining Functionalities
• Cluster analysis
– Class label is unknown: Group data to form new classes, e.g., cluster houses to find
distribution patterns
– Maximizing intra-class similarity & minimizing interclass similarity
• Outlier analysis
– Outlier: Data object that does not comply with the general behavior of the data
– Noise or exception? Useful in fraud detection, rare events analysis
• Trend and evolution analysis
– Trend and deviation: e.g., regression analysis
– Sequential pattern mining: e.g., digital camera large SD memory
– Periodicity analysis
– Similarity-based analysis
• Other pattern-directed or statistical analyses
Data Mining - Issues
Data Mining - Issues
• Mining methodology
– Mining different kinds of knowledge from diverse data types, e.g., bio, stream, Web
– Performance: efficiency, effectiveness, and scalability
– Pattern evaluation: the interestingness problem
– Incorporation of background knowledge
– Handling noise and incomplete data
– Parallel, distributed and incremental mining methods
– Integration of the discovered knowledge with existing one: knowledge fusion
• User interaction
– Data mining query languages and ad-hoc mining
– Expression and visualization of data mining results
– Interactive mining of knowledge at multiple levels of abstraction
• Applications and social impacts
– Domain-specific data mining & invisible data mining
– Protection of data security, integrity, and privacy
Data Mining Applications
