Introduction to Hadoop
Certified Big Data & Hadoop Training – DataFlair
Topics
Introduction to Hadoop
Hadoop nodes & daemons
Hadoop Architecture
Characteristics Hadoop
Certified Big Data & Hadoop Training – DataFlair
What is Hadoop?
An Open Source framework that
allows distributed processing of
large data-sets across the cluster
of commodity hardware
Certified Big Data & Hadoop Training – DataFlair
What is Hadoop?
An Open Source framework that Open Source
allows distributed processing of
large data-sets across the cluster Source code is freely available
of commodity hardware It may be redistributed and
modified
Certified Big Data & Hadoop Training – DataFlair
What is Hadoop?
An open source framework that Distributed Processing
allows Distributed Processing of
large data-sets across the cluster Data is processed distributedly
of commodity hardware on multiple nodes / servers
Multiple machines processes
the data independently
Certified Big Data & Hadoop Training – DataFlair
What is Hadoop?
An open source framework that Cluster
allows distributed processing of
large data-sets across the Cluster Multiple machines connected
of commodity hardware together
Nodes are connected via LAN
Certified Big Data & Hadoop Training – DataFlair
What is Hadoop?
An open source framework that Commodity Hardware
allows distributed processing of
large data-sets across the cluster Economic / affordable
of Commodity Hardware machines
Typically low performance
hardware
Certified Big Data & Hadoop Training – DataFlair
What is Hadoop?
• Open source framework written in Java
• Inspired by Google's Map-Reduce programming model
Certified Big Data & Hadoop Training – DataFlair
Hadoop History
Doug Cutting added Hadoop defeated
DFS & MapReduce Super computer
in
converted 4TB of
Doug Cutting started Doug Cutting
image archives over
working on joined Cloudera
100 EC2 instances
2002 2003 2004 2005 2006 2007 2008 2009
published GFS & Hadoop became
Development of
MapReduce papers top-level project
started as Lucene sub-project
launched Hive,
SQL Support for Hadoop
Certified Big Data & Hadoop Training – DataFlair
Hadoop Components
Hadoop consists of three key parts
Certified Big Data & Hadoop Training – DataFlair
Hadoop Nodes
Nodes
Master Node Slave Node
Certified Big Data & Hadoop Training – DataFlair
Hadoop Daemons
Nodes
Master Node Slave Node
Resource Node
Manager Manager
NameNode DataNode
Certified Big Data & Hadoop Training – DataFlair
Basic Hadoop Architecture
Sub Work Sub Work Sub Work Sub Work
Sub Work Sub Work Sub Work Sub Work
Sub Work Sub Work Sub Work Sub Work
Sub Work Sub Work Sub Work Sub Work
Sub Work Sub Work Sub Work Sub Work
Work Sub Work Sub Work Sub Work Sub Work
Sub Work Sub Work Sub Work Sub Work
Sub Work Sub Work Sub Work Sub Work
Certified Big Data & Hadoop Training – DataFlair
Hadoop Characteristics
Certified Big Data & Hadoop Training – DataFlair
Open Source
• Source code is freely
available Free Transparent
• Can be redistributed
• Can be modified Inter- Open Affordable
operable
Source
No vendor
Community
lock
Certified Big Data & Hadoop Training – DataFlair
Distributed Processing
• Data is processed distributedly
on cluster
• Multiple nodes in the cluster
process data independently
Centralized Processing
Distributed Processing
Certified Big Data & Hadoop Training – DataFlair
Fault Tolerance
• Failure of nodes are recovered
automatically
• Framework takes care of failure
of hardware as well tasks
Certified Big Data & Hadoop Training – DataFlair
Reliability
• Data is reliably stored on the
cluster of machines despite
machine failures
• Failure of nodes doesn’t
cause data loss
Certified Big Data & Hadoop Training – DataFlair
High Availability
• Data is highly available and
accessible despite hardware
failure
• There will be no downtime for
end user application due to
data
Certified Big Data & Hadoop Training – DataFlair
Scalability
• Vertical Scalability – New
hardware can be added to the
nodes
• Horizontal Scalability – New
nodes can be added on the fly
Certified Big Data & Hadoop Training – DataFlair
Economic
• No need to purchase costly license
• No need to purchase costly hardware
Commodity
Open Source + Hardware = Economic
Certified Big Data & Hadoop Training – DataFlair
Easy to Use
• Distributed computing challenges
are handled by framework
• Client just need to concentrate on
business logic
Certified Big Data & Hadoop Training – DataFlair
Data Locality
•
Data Data
Move computation to data
instead of data to computation
•
Data Data
Data is processed on the nodes
Storage Servers App Servers
where it is stored
Algo Algo
Data Data
Algorithm
Algo Algo
Data Data
Servers
Certified Big Data & Hadoop Training – DataFlair
Summary
• Everyday we generate 2.3 trillion GBs of data
• Hadoop handles huge volumes of data efficiently
• Hadoop uses the power of distributed computing
• HDFS & Yarn are two main components of Hadoop
• It is highly fault tolerant, reliable & available
Certified Big Data & Hadoop Training – DataFlair
