L

Smart Grid – Part I

E
Transforming the Traditional Electrical Grid

PT
Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
N IIT KHARAGPUR
Email: smisra@sit.iitkgp.ernet.in
Website: http://www.cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

Introduction
 Advancement of traditional electrical grid
 Traditional electrical grid

L
 Energy generation is done in centralized power plants

E
 Energy distribution is one directional – from the power plant to the homes or industries.
 Monitoring and restoration of grid is done manually

PT
 Uni‐directional communication
 Smart Grid –
 Achieve high reliability in power systems
N
 A cyber‐physical system equipped with sustainable models of energy production,
distribution, and usage

Introduction to Internet of Things 2

What is Smart Grid
 Smart grid is conceptualized as a planned nationwide network that uses information
technology to deliver electricity efficiently, reliably, and securely.

L
 Smart grid is also named as –
 Electricity with a brain

E
 The energy internet

PT
 The electronet
 According to the definition given by NIST, smart grid is – “a modernized grid that
enables bidirectional flows of energy and uses two‐way communication and control
capabilities that will lead to an array of new functionalities and applications.”
N
Source: https://www.nist.gov/engineering‐laboratory/smart‐grid/about‐smart‐grid/smart‐grid‐beginners‐guide

Introduction to Internet of Things 3

Benefits of Smart Grid
 Benefits associated with the Smart Grid include:
 More efficient transmission of electricity

L
 Quicker restoration of electricity after power disturbances
 Reduced operations and management costs for utilities, and ultimately lower power

E
costs for consumers
 Reduced peak demand, which will also help lower electricity rates

PT
 Increased integration of large‐scale renewable energy systems
 Better integration of customer‐owner power generation systems, including renewable
energy systems
 Improved security
N
 Using smart grid, both the consumers and the energy service providers or
stakeholders get benefited.

Introduction to Internet of Things 4

Benefits of Customers
 For consumers, the benefit of using smart grid are as follows:
 Updated information on their energy usage in real‐time

L
 Enabling electric cars, smart appliances, and other smart devices to be
charged

E
 Program the smart devices to run during off‐peak hours to lower energy bills
 Different pricing options

PT
N
Introduction to Internet of Things 5
Benefits to Stakeholders
 For stakeholders, the benefit of using smart grid are as follows:
 Increase grid reliability

L
 Reduce the frequency of power blackouts and brownouts
 Provide infrastructure for monitoring, analysis, and decision‐making

E
 Increase grid resiliency by providing detailed information

PT
 Reduce inefficiencies in energy delivery
 Integrate the sustainable resources of wind and solar alongside the main grid
 Improve management of distributed energy resources, including micro‐grid
operations and storage management.
N
Introduction to Internet of Things 6
Properties of Smart Grid
 Consumer Participation
 Real‐time monitoring of consumption

L
 Control of smart appliances

E
 Building Automation
 Real‐time Pricing

PT
 Distributed Generation
 Integration of renewable energy resources
 Integration of micro‐grid
N
Introduction to Internet of Things 7
Properties of Smart Grid (Contd.)
 Power System Efficiency
 Power Monitoring

L
 Asset Management and optimal utilizations

E
 Distribution Automation and Protection

PT
 Power Quality
 Self‐Healing
 Frequency Monitoring and Control
 Load Forecasting
N
 Anticipation of Disturbances

Introduction to Internet of Things 8

Smart Grid Architecture

E L
PT
N
Fig 1: Basic architecture of smart grid [D. Niyato and P. Wang, IEEE CM, 2012]

Introduction to Internet of Things 9

Smart Grid Domains

E L
PT
N
Source: NIST Framework and Roadmap for Smart Grid Interoperability Standards, Release 3.0

Introduction to Internet of Things 10

 Components of Smart Grid

E L
Smart Home Renewable Energy Consumer Engagement

PT
N
Operation Center Distribution Intelligence Plug‐in Electric Vehicle
Source: https://www.smartgrid.gov/the_smart_grid/

Introduction to Internet of Things 11

Smart Home
 Smart home uses emerging smart grid technologies to save energy, seek out the
lowest rates, and contribute to the smooth and efficient functioning of our

L
electric grid

E
 The interactive relationship between the grid operators, utilities, and

PT
consumers helps in proper functioning of smart grid technologies

 Computerized controls in smart homes helps to minimize energy use at times

when the power grid is under stress from high demand, or even to shift some of
N
their power use to times when power is available at a lower cost, i.e., from on‐
peak hours to off‐peak hours

Introduction to Internet of Things 12

Smart Home (Contd.)
 Smart home depends on –
 Smart meters and home energy management systems

L
 Smart appliances
 Home power generation

E
PT
N
Introduction to Internet of Things 13
Smart Home (Contd.)
 Smart Meters
 Provide the Smart Grid interface between consumer and the energy service

L
provider
 Operate digitally

E
 Allow for automated and complex transfers of information between consumer‐end
and the energy service provider

PT
 Help to reduce the energy costs of the consumers
 Provides information about usage of electricity in different service areas to the
energy service providers
N
Introduction to Internet of Things 14
Smart Home (Contd.)
 Home energy management systems
 Allows consumers to track energy usage in detail to better save energy

L
 Allows consumers to monitor real‐time information and price signals from the
energy service provider

E
 Allows to create settings to automatically use power when prices are lowest
 Avoids peak demand rates

PT
 Helps to balance the energy load in different area
 Prevents blackouts
 In return, the service provider also may choose to provide financial incentives
N
Introduction to Internet of Things 15
Smart Home (Contd.)
 Smart Appliances
 Automated and robust in nature

L
 Response to signals from the energy service provider to avoid using energy
during times of peak demand

E
 Include consumer controls to override the automated controls
 By overriding, the consumer can consume energy as per their requirement,

PT
while paying minimum is not ensured

N
Introduction to Internet of Things 16
Smart Home (Contd.)
 Home Power Generation
 Power generation system at consumers‐end

L
 Rooftop solar electric systems
 Small wind turbines

E
 Small hydropower System

PT
 Home fuel cell systems – produce heat and power from natural gas
 Surplus energy generated by the home power generation systems can be fed
back into the grid
 In case of “Islanding”, a home can have power from distributed resources, i.e.,
N
home power generation systems

Introduction to Internet of Things 17

Renewable Energy
 According to the International Energy Agency –
 “Renewable energy is derived from natural processes that are replenished constantly.

L
In its various forms, it derives directly from the sun, or from heat generated deep
within the earth. Included in the definition is electricity and heat generated from

E
solar, wind, ocean, hydropower, biomass, geothermal resources, and biofuels and
hydrogen derived from renewable resources.”

PT
 Reduced environmental pollution
 Consumers capable of generating energy from renewable energy resources are
less dependent on the micro‐grid or main grid
N
 In addition to that, they can supply surplus amount of energy from the
renewable resources and can make profit out of it

Introduction to Internet of Things 18

Consumer Engagement
 Consumers can –
 Save energy with proper scheduling of smart home appliances

L
 Pay less for consuming energy in off‐peak hours

E
 Energy service provider gives incentives based on the energy consumption of the

PT
consumer and they can save money

 Consumers’ involvement in following ways:

 Time‐of‐Use pricing
 Net metering
N
 Financial incentives

Introduction to Internet of Things 19

Consumer Engagement (Contd.)
 In Time‐of‐Use pricing
 The consumers are encouraged to consume energy in off‐peak hours when the energy

L
load is less
 Throughout the day, the energy load on the grids are dynamic

E
 In on‐peak hours, if the requested amount of energy is higher, it leads to –
 Less‐efficient energy distribution

PT
 More pollution – it depends on the non‐renewable energy resource to meet the peak
requirement
 Home energy management system tries to schedule the smart appliances in off‐
N
peak hours
 To ensure efficient service
 To pay less

Introduction to Internet of Things 20

Consumer Engagement (Contd.)
 Net metering
 It is feasible with the installation of smart meters

L
 Consumers are paid high, if they are supplying excess amount of generated energy
to the grid in on‐peak hours

E
 The price is less in case of off‐peak hours

PT
 Final bills to be paid by the consumers depends on
 The in‐flow of energy (from the grid to the consumers‐end)
 The out‐flow of energy (from the consumers‐end to the grid)
N
 The consumer may get incentives from the energy service provider at the end of
the year based on the net metering value

Introduction to Internet of Things 21

Consumer Engagement (Contd.)
 Financial Incentives
 Energy service provider offers some financial incentives for the consumers’

L
participation
 Incentives for shifting operation of appliances to the off‐peak hours

E
 Incentives for using stored energy at the battery installed at the consumers‐end or

PT
at the plug‐in hybrid electric vehicles (PHEVs)
 Smart grid enables consumers engagement to a large extend
 Consumers get financial incentives by different means from the energy service
providers N
 Energy service providers maintain efficient and load balancing energy
distribution

Introduction to Internet of Things 22

 EL
PT
N
Introduction to Internet of Things 23
 L
Smart Grid – Part II

E
Transforming the Traditional Electrical Grid

PT
Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
N IIT KHARAGPUR
Email: smisra@sit.iitkgp.ernet.in
Website: http://www.cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

Operation Centers
 Drawbacks of traditional operation centers
 Tries to make sure the amount of generated energy is getting used

L
 The grid is unstable, if the grid voltage drops due to excess energy generation

E
 Limited control capabilities
 No means to detect oscillation which leads to blackout

PT
 Limited information about the energy flow through the grid
 Smart grid
 Provides information and control on the transmission system
 Makes the energy grid more reliable
N
 Minimize the possibility of widespread blackouts

Introduction to Internet of Things 2

Operation Centers (Contd.)
 For monitoring and controlling the transmission System in smart grid, phasor
measurement unit (PMU) is used

L
 PMU samples voltage and current with a fixed sample rate at the installed
location

E
 It provides a snapshot of the active power system at that location
 By increasing the sampling rate, PMU provides the dynamic scenario of the

PT
energy distribution system
 PMU helps to identify the possibility of blackout in advance
 Multiple PMUs form a phasor network
N
 Collected information by the phasor network is analyzed at centralized system,
i.e., Supervisory Control And Data Acquisition (SCADA) system

Introduction to Internet of Things 3

Operation Centers (Contd.)
 Self‐healing of grid
 Dampen unwanted power oscillations

L
 Avoid unwanted flows of current through the grid
 Reroute power flows in order to avoid overloading in a transmission line

E
 This is part of distribution intelligence
 Demand side energy distribution

PT
 Energy supply is done based on the requirement of the consumers
 The consumers pay according the consumed energy and price decide by the energy
service provider at that time
N
 In smart grid, the energy distributors can form coalition and serve the energy
requirement in a specific geographic location

Introduction to Internet of Things 4

Distribution Intelligence
 Distribution intelligence means the energy distribution systems equipped with smart
IoT devices
 Along with smart meters, distribution intelligence can –

L
 Identify the source of a power outage

E
 Ensure power flow automatically by combining automated switching
 Optimize the balance between real and reactive power

PT
 Reactive power:
 Devices that store and release energy
 Cause increased electrical currents without consuming real power
 Intelligent distribution System
N
 Maintains the proper level of reactive power in the System
 Protect and control the feeder lines

Introduction to Internet of Things 5

Plug-In Electric Vehicles
 Smart Grids have the infrastructure needed to enable the efficient use of plug‐in
electric vehicle (PEVs)

L
 Using PEVs –
 Reduce dependency on oil

E
 No pollution when running on electricity
 PEVs rely on power plants to charge their batteries

PT
 Energy service provider encourages the consumers to charge batteries of PEVs in
off‐peak hours
 PEVs also can be used as an energy source in on‐peak hours
N
 PEVs get incentives from energy service provider for providing energy to the grid
through discharging

Introduction to Internet of Things 6

Smart Grid Communication

E L
PT
N
Fig 2: Smart Grid Communication[D. Niyato and P. Wang, IEEE CM, 2012]

Introduction to Internet of Things 7

Smart Grid Communication (Contd.)
 Components for smart grid communication are as follows:
 Smart Home Appliances

L
 Smart Meters
 Gateways

E
 Data Aggregator Units (DAUs)
 Meter Data Management Systems (MDMSs)

PT
 Different networks associated with smart grid communication
 Home Area Networks (HANs)
 Neighborhood Area Networks (NANs)
 Wide Area Networks (WANs)
 IP Networks
N
 Sensors and Actuators Networks (SANETs)

8
Smart Grid Communication (Contd.)
 For Smart Home Appliances, the available protocol are as follows:
 C‐Bus:
 Data Rate: 3500 bits/sec

L
 Able to handle cable lengths upto 1000 m

E
 DECT
 Data rate: 64000 bits/sec

PT
 Operates in 1880 – 1930 MHz
 EnOcean
 Data rate: 9600 bits/sec
 Operates in 902 MHz in North America
 Universal Power line Bus
N
 Data rate: 480 bits/sec
 Enable two‐way communication protocol

Introduction to Internet of Things 9

Smart Grid Communication (Contd.)
 Thread
 Data Rate: 20‐250 Kbits/sec
 IPv6 addressing based 6LowPAN networking protocol

L
 Zigbee

E
 Data Rate: 20‐250 Kbits/sec
 Operates in 2.4 GHz band

PT
 IEEE 802.15.4 protocol
 Communication range ~100 m
 Simplified Cable Solution (SCS)
 Data rate: 9.6 Kbits/sec
N
 Works on twisted pair
 Developed based on OpenWebNet

Introduction to Internet of Things 10

Smart Grid Communication (Contd.)
 Smart Meters and Gateways
 Each gateway connects few closely located smart meters

L
 Gateways communicate mostly based on WiFi, i.e., IEEE 802.11
 Gateways helps in two‐way communication

E
 Smart meters

PT
 Forward the energy consumption information fro the home appliances to the
gateways
 Forward the billing amount and the control information from the gateways to the
home appliances
N
 Gateway acts as link between the smart meters and the data aggregator units
(DAUs)

Introduction to Internet of Things 11

Smart Grid Communication (Contd.)
 Data Aggregator Units (DAUs)
 Aggregate the energy consumption or energy request of certain geographical

L
area
 Forward the energy consumption information to the centralized coordinator –

E
meter data management system (MDMS)
 Maintains a buffer to queue the energy consumption information of the

PT
consumers

N
Introduction to Internet of Things 12
Smart Grid Communication (Contd.)
 Meter Data Management Systems (MDMSs)
 Act as the centralized coordinator for smart grid communication

L
 Handled by the energy service providers
 Part of operation center

E
 Decide the price per unit energy to be paid by the consumers

PT
N
Introduction to Internet of Things 13
Smart Grid Security
 Smart grid is a cyber physical system
 Following vulnerabilities are there in smart grid

L
 Integrity – credibility of the data collected and transferred over the grid
 Availability – accessibility to every grid component as well as to the information

E
transmitted and collected

PT
 Dynamic system attacks – based on the previous information same type of request
can be replicated by the attacker
 Physical threats – physical attack to the smart grid components
 Coordinated attacks – cascading failure of systems in smart grid
N
Introduction to Internet of Things 14
Smart Grid Security (Contd.)
 Integrity
 Data injection attacks (DIAs)

L
 Manipulation of exchanged data such as sensor readings, feedback control signals, and
electricity price signals

E
 Performed by compromising the hardware components (as in the case of Stuxnet), or
intercepting the communication links

PT
 System Damage
 An attacker can manipulate system measurements so that a congested transmission line
falsely seems to not have reached its thermal transmission limit
N
 Induce large fluctuations in system dynamics that can lead to tripping additional lines,
disconnecting generators, load shedding, or even a system blackout

Introduction to Internet of Things 15

Smart Grid Security (Contd.)
 Integrity
 Financial benefit

L
 Manipulating the electricity prices
 Doing this one can buy energy with lesser price from a service provider and make high

E
profit
 Time synchronization attacks

PT
 An adversary can manipulate the time reference of the time stamped measured phasors to
create a false visualization of the actual system conditions thus yielding inaccurate control
and protection actions
 Attacks that target PMU time synchronization are known as time synchronization attacks
N
(TSAs)

Introduction to Internet of Things 16

Smart Grid Security (Contd.)
 Availability
 Accessibility unavailable to every grid component as well as to the information

L
transmitted and collected, whenever needed
 Attacks compromising this availability are known as denial of service (DoS) attacks

E
 Block key signals to compromise the stability of the grid and observability of its states

PT
 Manipulating generation‐load balance

N
Introduction to Internet of Things 17
Smart Grid Security (Contd.)
 Dynamic System Attacks
 Replay attacks (RAs)

L
 Injects input data in the system without causing changes to the measurable outputs
 In RAs –

E
 Compromises sensors, monitors their outputs
 Learns the outputs and repeats them while injecting its attack signal

PT
 Dynamic data injection attacks (D‐DIA)
 Uses knowledge of the grid’s dynamic model to inject data that causes unobservability of
unstable poles
 Can lead to a system collapse
 Covert attack
N
 Closed loop version of replay attacks

Introduction to Internet of Things 18

Smart Grid Security (Contd.)
 Physical Threats
 Attacks a physical component such as a generator, substation, or transmission line is

L
prominent
 Physical manipulation of smart meters for energy theft purposes

E
 Coordinated Attacks

PT
 Power system typically incorporates robustness measures
 An attack leading to the failure of one or few components
 Exploit the dense interconnections between grid components to launch
N
simultaneous attacks of different types targeting various components

Introduction to Internet of Things 19

 Smart Grid and Cloud Applications
 In smart grid, cloud applications
take a lead in several aspects

L
 Energy management
 Information management

E
 Security

PT
S. Bera, S. Misra, and J. J. P. C. Rodrigues, “Cloud Computing Applications for
Smart Grid: A Survey,” IEEE Transactions on Parallel and Distributed Systems, vol.
26, no. 5, pp. 1477–1494, May 2015.
N
Introduction to Internet of Things 20
Energy Management and Cloud
Application
 The energy management in smart grid can be more efficient by using cloud
applications

L
 Cloud‐Based Demand Response for fast response times in large scale deployment

E
 Two cloud‐based demand response models are proposed as follows:
 Data‐centric communication and

PT
 Topic‐based group communication
 With the integration of cloud, requests from customers are scheduled which are
to be executed depending on the available resources, priority, and other
applicable constraints
N
 Incoming jobs from users are scheduled according to their priority, available
resources, and applicable constraints

Introduction to Internet of Things 21

Energy Management and Cloud
Application (Contd.)
 Integrating cloud computing applications for micro‐grid management in the form
of different modules such as infrastructure, power management, and service

L
 The number of supported customers increases

E
 With cloud application, integrate and analyze information streaming from

PT
multiple smart meters simultaneously can be done, in order to balance the real‐
time demand and supply curves
 Real‐time energy usage and pricing information can be shared
N
 Mobile agent can be used to monitor power system using cloud computing
platform due to the smart grid’s heterogeneous architecture

Introduction to Internet of Things 22

 Information Management and Cloud
Application
 Information processing in smart grid fit well
with the computing and storage mechanisms

L
available for cloud applications

E
 Information from different components, and
the supply and demand state conditions can

PT
be shared with the help of cloud computing
 Real‐time distributed data management and
parallel processing of information can be
utilized using smart grid data cloud
application
N
Introduction to Internet of Things 23
Information Management and Cloud
Application (Contd.)
 With the flexibility of cloud computing, information is retrieved from the data
cloud more conveniently in smart grid

L
 Dynamic pricing mechanism in smart grid is feasible with the use of cloud

E
application
 Cloud computing services are used as a dynamic data centers to store the real‐

PT
time information from the smart meters
 Use of multi‐mobile agent combined with cloud computing for profitable smart
grid operation N
 Interactive cooperation using cloud services to support multiple customers and
multiple energy sources for large‐scale development of smart grid for energy
management

Introduction to Internet of Things 24

Security in Smart Grid and Cloud
Application
 An electric power information security and
protection system can be developed using

L
based on cloud security

E
 Private cloud platforms are suitable for
scaling out and processing millions of data

PT
from users
 Using the cloud computing platform, the
electrical utilities can quickly and
effectively deal with malicious software
N
Introduction to Internet of Things 25
Security in Smart Grid and Cloud
Application (Contd.)
 Security and protection system for electrical power

L
 Servers act as cloud and take decision according to the clients’ data
 Privacy issue in smart grid

E
 Quickly and effectively deal with malicious software with the implementation of
cloud computing applications

PT
 Data storage security for distributed verification in smart grid using cloud
application
 Real‐time data can be analyzed and estimated using cloud in smart grid
N
 Cloud‐based information privacy scheme can be used for smart grid data privacy

Introduction to Internet of Things 26

References
 G. Strbac, “Demand side management: Benefits and challenges,” Energy Policy, vol. 36, no. 12, pp. 4419–4426, 2008.
 M. Such and C. Hill, “Battery energy storage and wind energy integrated into the Smart Grid,” in Proceedings of IEEE PES on
Innovative Smart Grid Technologies, Washington, Jan 2012, pp. 1–4.

L
 S. Misra, P. V. Krishna, V. Saritha, and M. S. Obaidat, “Learning Automata as a Utility for Power Management in Smart Grids,”
IEEE Communications Magazine, vol. 51, no. 1, pp. 98–104, 2013.
 V. Bakker, M. G. C. Bosman, A. Molderink, J. L. Hurink, and G. J. M. Smit, “Demand Side Load Management Using a Three Step

E
Optimization Methodology,” in Proceedings of the 1st IEEE International Conference on Smart Grid Communications,
Gaithersburg, Oct 2010, pp. 431–436.

PT
 S. Misra, S. Bera, and T. Ojha, “D2P: Distributed Dynamic Pricing Policy in Smart Grid for PHEVs Management,” IEEE
Transactions on Parallel and Distributed Systems, vol. 26, no. 3, pp. 702–712, Mar 2015.
 S. Bera, S. Misra, and J. J. P. C. Rodrigues, “Cloud Computing Applications for Smart Grid: A Survey,” IEEE Transactions on
Parallel and Distributed Systems, vol. 26, no. 5, pp. 1477–1494, May 2015.
 S. Misra, A. Mondal, S. Banik, M. Khatua, S. Bera, and M. S. Obaidat, “Residential Energy Management in Smart Grid: A
Markov Decision Process‐Based Approach,” in IEEE International Conference on Internet of Things, Beijing, Chaina, Aug 2013,
pp. 1152–1157.
N
 A. Mondal and S. Misra, “Game‐Theoretic Green Electric Vehicle Energy Networks Management in Smart Grid,” in IEEE
International Conference on Advanced Networks and Telecommunications Systems,Dec 2015, pp. 1–6.

CSE, IIT Kharagpur Introduction to Internet of Things 27

References (Contd.)
 A. Molderink, V. Bakker, M. G. C. Bosman, J. L. Hurink, and G. J. M. Smit, “Management and Control of Domestic Smart Grid
Technology,” IEEE Transactions on Smart Grid, vol. 1, no. 2, pp. 109–119, Aug 2010.
 M. Erol‐Kantarci and H. T. Mouftah, “TOU‐Aware Energy Management and Wireless Sensor Networks for Reducing Peak Load

L
in Smart Grids,” in the 72nd IEEE Vehicular Technology Conference Fall, Ottawa, ON, Sept 2010, pp. 1 – 5.
 A. Mondal and S. Misra, “Dynamic Coalition Formation in a Smart Grid: A Game Theoretic Approach,” in Proceedings of IEEE
International Workshop on Smart Communication Protocols and Algorithms in conjunction with IEEE ICC, Budapest, Hungary,

E
Jun 2013, pp. 1067 – 1071.
 F. Farzan, F. Farzan, M. A. Jafari, and J. Gong, “Integration of Demand Dynamics and Investment Decisions on Distributed

PT
Energy Resources,” IEEE Transactions on Smart Grid, vol. 7, no. 4, pp. 1886–1895, Jul 2016.
 A. Mondal and S. Misra, “Game‐Theoretic Energy Trading Network Topology Control for Electric Vehicles in Mobile Smart
Grid,” IET Networks, vol. 4, no. 4, pp. 220–228, 2015.
 F. Kamyab, M. Amini, S. Sheykhha, M. Hasanpour, and M. M. Jalali, “Demand Response Program in Smart Grid Using Supply
Function Bidding Mechanism,” IEEE Transactions on Smart Grid, vol. 7, no. 3, pp. 1277–1284, May 2016.
 A. Sanjab, W. Saad, I. Guvenc, A. Sarwat, and S. Biswas, "Smart Grid Security: Threats, Challenges, and Solutions," arXiv
N
preprint arXiv:1606.06992 (2016).
 A. Mondal and S. Misra, “Dynamic Data Aggregator Unit Selection in Smart Grid: An Evolutionary Game Theoretic Approach,”
in IEEE India Conference, Dec 2014, pp. 1–6.

CSE, IIT Kharagpur Introduction to Internet of Things 28

References (Contd.)
 P. Samadi, V. W. S. Wong, and R. Schober, “Load Scheduling and Power Trading in Systems With High Penetration of
Renewable Energy Resources,” IEEE Transactions on Smart Grid, vol. 7, no. 4, pp. 1802–1812, Jul 2016.
 A. Mondal and S. Misra, “Game‐Theoretic Distributed Virtual Energy Cloud Topology Control for Mobile Smart Grid,” in IEEE

L
6th International Conference on Cloud Computing Technology and Science (CloudCom), Dec 2014, pp. 54–61.
 A. Mondal, S. Misra, and M. S. Obaidat, “Distributed Home Energy Management System With Storage in Smart Grid Using

E
Game Theory,” IEEE Systems Journal, pp. 1–10, 2015.
 C. P. Mediwaththe, E. R. Stephens, D. B. Smith, and A. Mahanti, “A Dynamic Game for Electricity Load Management in
Neighborhood Area Networks,” IEEE Transactions on Smart Grid, vol. 7, no. 3, pp. 1329–1336, May 2016.

PT
 X. Liang, X. Li, R. Lu, X. Lin, and X. Shen, “UDP: Usage‐Based Dynamic Pricing With Privacy Preservation for Smart Grid,” IEEE
Transactions on Smart Grid, vol. 4, no. 1, pp. 141–150, Mar 2013.
 S. Shivshankar and A. Jamalipour, “An Evolutionary Game Theory‐Based Approach to Cooperation in VANETs Under Different
Network Conditions,” IEEE Transactions on Vehicular Technology, vol. 64, no. 5, pp. 2015–2022, May 2015.
 P. Samadi, H. Mohsenian‐Rad, R. Schober, and V. W. S. Wong, “Advanced Demand Side Management for the Future Smart
N
Grid UsingMechanism Design,” IEEE Transactions on Smart Grid, vol. 3, no. 3, pp. 1170–1180, Sept 2012.

CSE, IIT Kharagpur Introduction to Internet of Things 29

 EL
PT
N
Introduction to Internet of Things 30
 E L
IIoT: Industrial Internet of Things – Part I

PT
Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
IIT Kharagpur
N Email: smisra@sit.iitkgp.ernet.in
Website: http://cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

 “IoT as a concept has crossed the chasm from slideware to

L
reality with many industries implementing IoT solutions.”

E
‐ Paul Howarth, Senior Manager, Corporate Development, CISCO

PT
N Source : http://www.mcrockcapital.com

Introduction to Internet of Things 2

Introduction
 The main aim of Internet of Things (IoT) is

L
 to globally connect smart ‘things’ or ‘objects’ .

E
 objects are uniquely identified.
 interoperability among the objects.

PT
 The Industrial Internet of Things (IIoT) is an application of IoT in industries
to modify the various existing industrial systems. IIoT links the automation
system with enterprise, planning and product lifecycle.
N
Introduction to Internet of Things 3
Introduction (contd.)

L
‐ Automation and data
exchange in manufacturing

E
technologies
Internet of IIoT ‐ Cyber‐physical systems, the
Industry 4.0 Internet of things and cloud
Things

PT
computing
‐ Smart factory

N
Fig 1(a) : IIoT as an intersection of industries and IoT

Introduction to Internet of Things 4

Introduction (contd.) Enterprise IoT

Consumer IoT

L
Industrial

E
Internet of
Things Internet
Internet of IIoT Industries of Things
Things 4.0

PT
N
Fig 1(a) : IIoT as an intersection of industries and IoT

Fig 1 : IIoT Platform

Fig 1(b) : IIoT ≠ IoT

Introduction to Internet of Things 5

Introduction (contd.)
 IIoT includes –

L
 machine learning

E
 big data technology
 machine ‐ to ‐ machine interaction (M‐2‐M)

PT
 automation.
 IIoT is supported by huge amount of data collected from sensors. It is
based on “wrap & re‐use” approach, rather than “rip & replace” approach.
N
(Source : http://www.mhi.org)

Introduction to Internet of Things 6

Introduction (contd.)

L
 1st Industrial Revolution : Mechanized

E
production Smart
 2nd Industrial Revolution : Mass Electronic
Automation
(today)

PT
production Automation
Industrialization (1969)
 3rd Industrial Revolution : Internet (1870)
Power Generation
evolution and automation & Mechanical
 4th Industrial Revolution : IIoT
N Automation (1782)

Fig 2: Industry 4.0

Source: http://www.industry40wood.com

Introduction to Internet of Things 7

Introduction (contd.)

L
Cloud
computing

E
PT
Source: https://www.artika.info Source: http://www.rehm‐group.com

IIoT : 2nd generation of Internet evolution and 4th Industrial Automation

N
Introduction to Internet of Things 8
Introduction (contd.)
 IIoT is a network of

L
 physical objects
 systems

E
 platforms
 applications

PT
 These networks can communicate with each other, external environment
and other people.
 The acquisition of IIoT has led to availability and affordability of sensors,
processors, and other technologies which facilitates capture and access to
N
real‐time information

Introduction to Internet of Things 9

IIoT Requirements

L
Hardware

E
and Cloud
Software platform
connectivity

PT
Big Data Application
analytics Development

N
Introduction to Internet of Things 10
IIoT Requirements (contd.)
Access

L
(anything,
anytime,
anywhere)
Cloud for

E
End‐to‐end
efficiency
security
and agility

PT
Industrial
Internet
of Things User
Big data
experience

N Asset
management
Transition
to smart
machines

Introduction to Internet of Things 11

IIoT Requirements (contd.)

L
Physical Plant

E
Sensor
readings

PT
Virtual Plant
Machine
N
instructions

Introduction to Internet of Things 12

Design Considerations
 To use an IoT device for industrial applications, the following design

L
objectives are to be considered –
 Energy : Time for which the IoT device can operate with limited power

E
supply.

PT
 Latency : Time required to transmit the data.
 Throughput : Maximum data transmitted across the network.
 Scalability : Number of devices supported.
N
 Topology: Communication among the devices, i.e. interoperability.
 Safety and Security: Degree of safety and security of the application.

Introduction to Internet of Things 13

Difference between IoT and IIoT

E L
PT
N
Introduction to Internet of Things 14
Difference between IoT and IIoT
The main differences between IoT and IIoT are :

L
IoT IIoT

E
• Focused on convenience • Focused on efficiency,
of individuals safety and security of

PT
• M‐2‐M communication: the operation.
Limited • M‐2‐M communication:
• Applications areas are Extensively.
at consumer‐level • Application areas are at
N industries.

Introduction to Internet of Things 15

Difference between IoT and IIoT (contd.)

E L
PT
Network Service Application and System
Devices
(connectivity) enablement data integration

M‐2‐M focus
N IoT focus

Introduction to Internet of Things 16

Service Management in IIoT
 “Service management refers to the implementation and management of

L
the quality of services which meets the end‐users demand”
 “Service is a collection of data and associated behaviors to accomplish a

E
particular function or feature of a device or portions of a device”.

PT
N
Source: Ning Lu, Nan Cheng, Ning Zhang, Xuemin Shen, Jon W. Mark, Connected Vehicles : Solutions and Challenges, IEEE
Internet of Things Journal, Vol. 1, No. 4, August 2014.

Introduction to Internet of Things 17

Service Management in IIoT
 Service can be of two types, which are ‐

L
 Primary service ‐ The basic services which are responsible for the

E
primary node functions are termed as primary service.
 Secondary service ‐ The auxiliary functions which provide services to

PT
the primary service or secondary services are termed as secondary
service.

N
Introduction to Internet of Things 18
 EL
PT
N
Introduction to Internet of Things 19
 E L
IIoT: Industrial Internet of Things – Part II

PT
Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
IIT Kharagpur
N Email: smisra@sit.iitkgp.ernet.in
Website: http://cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

Applications of IIoT
 The key application areas of IIoT are ‐

L
 Manufacturing industry

E
 Healthcare Service industry
 Transportation & logistics

PT
 Mining
 Firefighting
N
Introduction to Internet of Things 2
Manufacturing Industry
 The devices, equipment, workforce, supply chain, work platform are

L
integrated and connected to achieve smart production. This will led to –
 reduction in operational costs

E
 improvement in the productivity of the worker

PT
 reduction in the injuries at the workplace
 resource optimization and waste reduction
 end‐to‐end automation.
N
Introduction to Internet of Things 3
Healthcare Service Industry
 Patients can be continuously monitored due to the implanted on‐body

L
sensors. This has led to –
 improved treatment outcome

E
 costs has reduced

PT
 improved disease detection
 improved accuracy in the collection of data
 improved drugs management.
N
Introduction to Internet of Things 4
Transportation & logistics
 To improve safety, efficiency of transportation, Intelligent Transportation

L
system (ITS) is developed which consists of connected vehicles. ITS
provides –

E
 Vehicle – to – sensor connectivity
 Vehicle – to – vehicle connectivity

PT
 Vehicle – to – internet connectivity
 Vehicle – to – road infrastructure
 Dedicated short‐range communications (DSRC) is the key enabling
N
technology for V2V and V2R communications.

Introduction to Internet of Things 5

Transportation & logistics
 In IIoT scenario the physical objects are provided with

L
 bar codes

E
 RFID tags
hence, real‐time monitoring of the status and location of the physical

PT
objects from destination to the origin, across the supply chain is possible.
 Security and privacy of the data should be maintained.
N
Introduction to Internet of Things 6
Mining
 To prevent accidents inside the mines ‐ RFID, Wi‐Fi and other wireless

L
technologies are used, which
 provides early warning of any disaster

E
 monitors air‐quality

PT
 detects the presence of poisonous gases inside the mines
 oxygen level inside the mines.
N
Introduction to Internet of Things 7
Firefighting
 Sensor networks, RFID tags are used to perform

L
 automatic diagnosis
 early warning of disaster

E
 emergency rescue

PT
 provides real‐time monitoring
Hence, improves public security.
N
Introduction to Internet of Things 8
Examples of IIoT
 Examples of IIoT are ‐

L
 unmanned aerial vehicles (UAVs) to inspect oil pipelines.

E
 monitoring food safety using sensors.
 minimizing workers’ exposure to noise, chemicals and other hazardous

PT
gases.
 unmanned marine vehicle which can collect data up to a year without
fuel or crew.
N
Introduction to Internet of Things 9
Connected Ecosystems in IIoT scenario
 Traditional supply chains in industries are linear in nature.

L
 To shift the business focus from products to outcomes, new ecosystem

E
should be followed.
 Digital ecosystems progress at a much faster rate than physical industries.

PT
Hence, it can quickly adapt to the changes in the external environments.

N
Introduction to Internet of Things 10
Integration of Digital and Human Workforce
 In IIoT, machines become more intelligent. Hence, the automated tasks

L
can be done in the industries at lower costs and higher quality level.
 Humans will work with machines, the outcome will be higher overall

E
productivity.

PT
 IIoT will reform and redefine the skills of the workers.

N
Introduction to Internet of Things 11
Creation of New Jobs
 The creation of new composite industries, such as precision agriculture,

L
digital healthcare system, digital mines etc., will lead to development of
new job opportunities.

E
 Highly automated machines will require lesser number of unskilled

PT
workers, but will require skilled experts with digital and analytical skills.

N
Introduction to Internet of Things 12
Reformation of Robots
 In IIoT environment, robots are featured with three capabilities : sensing,

L
thinking and acting. They will be reformed with the ability to carry out
repetitive tasks.

E
 Robots will be more intelligent but will work under the supervision of

PT
human beings. Their availability will increase.
 Robots will be reprogrammable to perform new tasks. They have the
capability to ‘learn’ faster.
N
Introduction to Internet of Things 13
Challenges in IIoT
 Primary challenges

L
Identification of objects or
things

E
Manage huge amount of data

PT
Integrate existing
infrastructures into new IIoT
N infrastructure
Enabling data storage

Introduction to Internet of Things 14

Challenges in IIoT(contd.)
 Safety Challenges

L
Worker health and safety

E
Regulatory compliance

PT
Environmental protection
N Optimized operations

Introduction to Internet of Things 15

Challenges in IIoT(contd.)
 Hazards (related)

L
Handling, storing or using hazardous substances

E
Oxygen deficiency

PT
Particulates

Radiation
N Physiological stress

Introduction to Internet of Things 16

Challenges in IIoT(contd.)
Standardization

L
 Standardization plays an important role in the development of the system.

E
 Goal: To improve the interoperability of the different systems/ applications
and allow the products/services to perform better.

PT
N
Introduction to Internet of Things 17
Challenges in IIoT(contd.)

Standardization

L
 The problems related to standardization are:

E
 Interoperability
 Semantic interoperability (data sematics)

PT
 Security and privacy
 Radio access level issues.
N
Introduction to Internet of Things 18
Challenges in IIoT(contd.)
Privacy and security issues

L
 The two most important concerns related with IIoT are ‐
 information security

E
 data privacy protection

PT
 The devices/things can be tracked, monitored and connected. So there are
chances of attack on the personal and private data.

N
Introduction to Internet of Things 19
Challenges in IIoT(contd.)
Privacy and security issues
 Examples –

L
 Healthcare industry – the medical data of a patient must not be

E
tampered, or altered by any person in the middle.
 Food industry – the deterioration of any food item being sent to the

PT
company must be kept confidential as it will affect the reputation of
the company.
N
Introduction to Internet of Things 20
Risks associated with IIoT in Manufacturing
 Though IIoT provides new opportunities, but few factors may cause

L
hindrance in the path to success, which are :
 lack of vision and leadership

E
 lack of understanding of values among management employees

PT
 costly sensors
 inadequate infrastructure.
N
Introduction to Internet of Things 21
Meet the challenges: Sensor improvement
 Improvement in sensor technologies –

L
 miniaturization

E
 performance
 cost and energy consumption.

PT
N
Introduction to Internet of Things 22
Meet the challenges : Manufacturing
 Manufacturers use software capabilities to improve operational efficiency

L
through –
 predictive maintenance

E
 savings on scheduled repairs

PT
 reduced maintenance costs
 reduced number of breakdowns.
N
Introduction to Internet of Things 23
Case study : Rt Tech Software
 Rt Tech particularizes in software which –

L
 improves industrial facilities’ efficiency

E
 improves productivity.
 Energy management solution, which leads to reduction in the plant’s

PT
highest variable cost.
 Rt Tech automates the process of mapping and managing energy
consumption.N Source : http://www.mcrockcapital.com

Introduction to Internet of Things 24

PRODUCTS DEVELOPED
 M‐2‐M communication : Intelligent Radio Modem (IRM)

L
 IRM 1500 & ACE 1000 ‐ IRM
 simple

E
 M‐2‐M connectivity

PT
 data transmission
 These devices provide easy maintenance and installation. They can be
connected to IP and non‐IP serial devices to extend the capability to
monitor and communicate with other technologies.
N
Source : https://www.motorolasolutions.com

Introduction to Internet of Things 25

PRODUCTS DEVELOPED (contd.)
 Comtrol – IO Link Master Gateway

L
 It can be easily integrated into the industrial

E
network with existing and new installations.
 It supports Ethernet/IP, PROFINET (PNIO)

PT
and Modbus TCP.

N Source :
http://pdfserv.maximintegrated.com
http://www.comtrol.com

Introduction to Internet of Things 26

Benefits of IIoT
 The benefits of IIoT are
Improved connectivity

L
among devices

E
Improved efficiency

Upgraded scalability

PT
Reduces operation time

Remote diagnosis

Cost effective
N
Introduction to Internet of Things 27
Recent Research trends in IIoT
 Recent research challenges in IIoT are ‐

L
 To improve the communications among the different things or objects.

E
 To develop energy‐efficient techniques so as to reduce power
consumption by sensors.

PT
 To develop context‐aware IoT middleware for better understanding of
the sensor data.
 To create smart objects with larger memory, processing and reasoning
capabilities.
N
Introduction to Internet of Things 28
Conclusion
 IIoT system requires the following :

L
 Smaller, less expensive sensors which makes them easily accessible.

E
 Distributed control of assembly line, automated monitoring, control
and maintenance.

PT
N
Introduction to Internet of Things 29
References
 Daniele Miorandi, Sabrina Sicari, Francesco De Pellegrini, Imrich Chlamtac, Internet of things: Vision,

L
applications and research challenges, Ad Hoc Networks, Volume 10, Issue 7, September 2012.
 http://internetofthingsagenda.techtarget.com/definition/Industrial‐Internet‐of‐Things‐IIoT.
 Ning Lu, Nan Cheng, Ning Zhang, Xuemin Shen, Jon W. Mark, Connected Vehicles : Solutions and

E
Challenges, IEEE Internet of Things Journal, Vol. 1, No. 4, August 2014.
 Zhibo Pang, Qiang Chen, Junzhe Tian, Lirong Zheng and E. Dubrova, Ecosystem analysis in the design of

PT
open platform‐based in‐home healthcare terminals towards the internet‐of‐things, 2013, 15th
International Conference on Advanced Communications Technology (ICACT), PyeongChang, 2013.
 Wei Qiuping, Zhu Shunbing, Du Chunquan, Study On Key Technologies Of Internet Of Things Perceiving
Mine, Procedia Engineering, Volume 26, 2011.
 Bill Karakostas, A DNS Architecture for the Internet of Things: A Case Study in Transport Logistics, Procedia
Computer Science, Volume 19, 2013.
N
 Ying‐cong Zhang, Jing Yu, A Study on the Fire IOT Development Strategy, Procedia Engineering, Volume 52,
2013.

Introduction to Internet of Things 30

References
 J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, Internet of things(IoT): A vision, architectural

L
elements, and future directions, Future Gen. Comput. Syst., vol. 29, no. 7, 2013 .
 D. Bandyopadhyay and Jaydip Sen, Internet of things: Applications and challenges in technology and

E
standardization, Wireless Personal Communications 58.1 (2011).
 Industry 4.0, The Industrial Internet of Things, by Alasdair Gilchrist
 http://pdfserv.maximintegrated.com

PT
 http://www.comtrol.com
 http://www.mcrockcapital.com
 http://web.stanford.edu
 http://www.accenture.com
N
Introduction to Internet of Things 31
 EL
PT
N
Introduction to Internet of Things 32
 L
Data Handling and Analytics – Part I

E
Data is Precious

PT
Dr. Sudip Misra
Associate Professor
Department of Computer Science and Engineering
N IIT KHARAGPUR
Email: smisra@sit.iitkgp.ernet.in
Website: http://cse.iitkgp.ac.in/~smisra/

Introduction to Internet of Things 1

What is Data Handling
 Data handling

L
 Ensures that research data is stored, archived or disposed off in a safe and secure
manner during and after the conclusion of a research project

E
 Includes the development of policies and procedures to manage data handled
electronically as well as through non‐electronic means.

PT
 In recent days, most data concern –
 Big Data N
 Due to heavy traffic generated by IoT devices
 Huge amount of data generated by the deployed sensors

Introduction to Internet of Things 2

What is Big Data
 “Big data technologies describe a new generation of technologies and architectures,

L
designed to economically extract value from very large volumes of a wide variety of
data, by enabling the high-velocity capture, discovery, and/or analysis.”

E
[Report of International Data Corporation (IDC)]
 “Big data shall mean the data of which the data volume, acquisition speed, or data

PT
representation limits the capacity of using traditional relational methods to conduct
effective analysis or the data which may be effectively processed with important
horizontal zoom technologies.”
N
[National Institute of Standards and Technology (NIST)]

Introduction to Internet of Things 3

Types of Data
 Structured data
 Data that can be easily organized.

L
 Usually stored in relational databases.

E
 Structured Query Language (SQL) manages structured data in databases.
 It accounts for only 20% of the total available data today in the world.

PT
 Unstructured data
 Information that do not possess any pre‐defined model.
 Traditional RDBMSs are unable to process unstructured data.
 Enhances the ability to provide better insight to huge datasets.

N
It accounts for 80% of the total data available today in the world.

Introduction to Internet of Things 4

Characteristics of Big Data
 Big Data is characterized by 7 Vs –

L
 Volume
 Velocity

E
 Variety
 Variability

PT
 Veracity
 Visualization
 Value N
Introduction to Internet of Things 5
Characteristics of Big Data (Contd.)
 Volume
 Quantity of data that is generated

L
 Sources of data are added continuously

E
 Example of volume ‐
 30TB of images will be generated every night from the Large Synoptic Survey Telescope

PT
(LSST)
 72 hours of video are uploaded to YouTube every minute

N
Introduction to Internet of Things 6
Characteristics of Big Data (Contd.)
 Velocity
 Refers to the speed of generation of data

L
 Data processing time decreasing day‐by‐day in order to provide real‐time services

E
 Older batch processing technology is unable to handle high velocity of data
 Example of velocity –

PT
 140 million tweets per day on average (according to a survey conducted in 2011)
 New York Stock Exchange captures 1TB of trade information during each trading
session
N
Introduction to Internet of Things 7
Characteristics of Big Data (Contd.)
 Variety
 Refers to the category to which the data belongs

L
 No restriction over the input data formats

E
 Data mostly unstructured or semi‐structured
 Example of variety –

PT
 Pure text, images, audio, video, web, GPS data, sensor data, SMS, documents, PDFs, flash
etc.

N
Introduction to Internet of Things 8
Characteristics of Big Data (Contd.)
 Variability
 Refers to data whose meaning is constantly changing.

L
 Meaning of the data depends on the context.

E
 Data appear as an indecipherable mass without structure
 Example:

PT
 Language processing, Hashtags, Geo‐spatial data, Multimedia, Sensor events
 Veracity
 Veracity refers to the biases, noise and abnormality in data.
 It is important in programs that involve automated decision‐making, or feeding the data
N
into an unsupervised machine learning algorithm.
 Veracity isn’t just about data quality, it’s about data understandability.

Introduction to Internet of Things 9

Characteristics of Big Data (Contd.)
 Visualization
 Presentation of data in a pictorial or graphical format

L
 Enables decision makers to see analytics presented visually

E
 Identify new patterns

PT
 Value
 It means extracting useful business information from scattered data.
 Includes a large volume and variety of data
N
 Easy to access and delivers quality analytics that enables informed decisions

Introduction to Internet of Things 10

Data Handling Technologies
 Cloud computing
 Essential characteristics according to NIST

L
 On‐demand self service

E
 Broad network access
 Resource pooling

PT
 Rapid elasticity
 Measured service
 Basic service models provided by cloud computing
 Infrastructure‐as‐a‐Service (IaaS)
N
 Platform‐as‐a‐Service (PaaS)
 Software‐as‐a‐Service (SaaS)

Introduction to Internet of Things 11

Data Handling Technologies (Contd.)
 Internet of Things (IoT)
 According to Techopedia, IoT “describes a future where every day physical

L
objects will be connected to the internet and will be able to identify themselves
to other devices.”

E
 Sensors embedded into various devices and machines and deployed into fields.

PT
 Sensors transmit sensed data to remote servers via Internet.
 Continuous data acquisition from mobile equipment, transportation facilities,
public facilities, and home appliances
N
Introduction to Internet of Things 12
Data Handling Technologies (Contd.)
 Internet of Things (IoT)
 According to Techopedia, IoT “describes a future where every day physical

L
objects will be connected to the internet and will be able to identify themselves
to other devices.”

E
 Sensors embedded into various devices and machines and deployed into fields.

PT
 Sensors transmit sensed data to remote servers via Internet.
 Continuous data acquisition from mobile equipment, transportation facilities,
public facilities, and home appliances
N
Introduction to Internet of Things 13
Data Handling Technologies (Contd.)
 Data handling at data centers
 Storing, managing, and organizing data.

L
 Estimates and provides necessary processing capacity.

E
 Provides sufficient network infrastructure.
 Effectively manages energy consumption.

PT
 Replicates data to keep backup.
 Develop business oriented strategic solutions from big data.
 Helps business personnel to analyze existing data.
 Discovers problems in business operations.
N
Introduction to Internet of Things 14
Flow of Data

E L
Generation Acquisition Storage Analysis

PT
 Enterprise data  Data collection  Hadoop  Bloom filter
 IoT data  Data transportation  MapReduce  Parallel computing
 Bio‐medical data
N
 Data pre‐processing  NoSQL databases  Hashing and
 Other data indexing

Introduction to Internet of Things 15

Data Sources
 Enterprise data  Bio‐medical data
 Online trading and analysis data.  Masses of data generated by gene

L
 Production and inventory data. sequencing.

E
 Sales and other financial data.  Data from medical clinics and medical
 IoT data R&Ds.

PT
 Data from industry, agriculture,  Other fields
traffic, transportation  Fields such as – computational biology,
 Medical‐care data, astronomy, nuclear research etc
 Data from public departments, and
families.
N
Introduction to Internet of Things 16
Data Acquisition
 Data collection
 Log files or record files that are automatically generated by data sources to record

L
activities for further analysis.
 Sensory data such as sound wave, voice, vibration, automobile, chemical, current,

E
weather, pressure, temperature etc.
 Complex and variety of data collection through mobile devices. E.g. – geographical

PT
location, 2D barcodes, pictures, videos etc.
 Data transmission
 After collecting data, it will be transferred to storage system for further processing and
N
analysis of the data.
 Data transmission can be categorized as – Inter‐DCN transmission and Intra‐DCN
transmission.

Introduction to Internet of Things 17

Data Acquisition (Contd.)
 Data pre‐processing
 Collected datasets suffer from noise, redundancy, inconsistency etc., thus, pre‐

L
processing of data is necessary.
 Pre‐processing of relational data mainly follows – integration, cleaning, and

E
redundancy mitigation

PT
 Integration is combining data from various sources and provides users with a uniform
view of data.
 Cleaning is identifying inaccurate, incomplete, or unreasonable data, and then
modifying or deleting such data.
N
 Redundancy mitigation is eliminating data repetition through detection, filtering and
compression of data to avoid unnecessary transmission.

Introduction to Internet of Things 18

Data Storage
 File system
 Distributed file systems that store massive data and ensure – consistency, availability,

L
and fault tolerance of data.
 GFS is a notable example of distributed file system that supports large‐scale file

E
system, though it’s performance is limited in case of small files
 Hadoop Distributed File System (HDFS) and Kosmosfs are other notable file systems,

PT
derived from the open source codes of GFS.
 Databases
 Emergence of non‐traditional relational databases (NoSQL) in order to deal with the
N
characteristics that big data possess.
 Three main NoSQL databases – Key‐value databases, column‐oriented databases, and
document‐oriented databases.

Introduction to Internet of Things 19

 L
Data Handling Using Hadoop

E
Reliable, scalable, distributed data handling

PT
N
Introduction to Internet of Things 20
 What is Hadoop

 Hadoop is a software framework for

L
distributed processing of large datasets
across large clusters of computers.

E
 Hadoop is open-source implementation for

PT
Google ‘s GFS and MapReduce.
 Apache Hadoop's Map Reduce and Hadoop
Distributed File System (HDFS)
components originally derived respectively
N
from Google's MapReduce and Google File
System (GFS) .
Source: https://www.cloudnloud.com/hadoop-hdfs-operations/

Introduction to Internet of Things 21

Building Blocks of Hadoop
 Hadoop Common

L
 A module containing the utilities that support the other Hadoop components
 Hadoop Distributed File System (HDFS)

E
 Provides reliable data storage and access across the nodes
 MapReduce

PT
 Framework for applications that process large amount of datasets in parallel.
 Yet Another Resource Negotiator (YARN)
 Next‐generation MapReduce, which assigns CPU, memory and storage to applications
N
running on a Hadoop cluster.

Introduction to Internet of Things 22

Hadoop Distributed File System (HDFS)
 Centralized node

L
 Namenode
 Maintains metadata info about files

E
 Distributed node

PT
 Datanode
 Store the actual data
 Files are divided into blocks
N
 Each block is replicated
Source: http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html

Introduction to Internet of Things 23

Name and Data Nodes
 Namenode

L
 Stores filesystem metadata.
 Maintains two in‐memory tables, to map the datanodes to the blocks, and vice versa

E
 Datanode

PT
 Stores actual data
 Data nodes can talk to each other to rebalance and replicate data
 Data nodes update the namenode with the block information periodically
N
 Before updating datanodes verify the checksums.

Introduction to Internet of Things 24

Job and Task Trackers
 Job Tracker –
 Runs with the Namenode

L
 Receives the user’s job
 Decides on how many tasks will run (number

E
of mappers)
 Decides on where to run each mapper

PT
(concept of locality)
 Task Tracker –
 Runs on each datanode
N
 Receives the task from Job Tracker
 Always in communication with the Job Source: http://developeriq.in/articles/2015/aug/11/an-introduction-to-
Tracker reporting progress apache-hadoop-for-big-data/

Introduction to Internet of Things 25

Hadoop Master/Slave Architecture
 Master‐slave shared‐nothing architecture
 Master

L
 Executes operations like opening, closing,
and renaming files and directories.

E
 Determines the mapping of blocks to
Datanodes.

PT
 Slave
 Serves read and write requests from the
file system’s clients.
N
 Performs block creation, deletion, and
replication as instructed by the Namenode.
Source: http://ankitasblogger.blogspot.in/2011/01/hadoop-cluster-setup.html

Introduction to Internet of Things 26

References
 R. Ahmed and G. Karypis, “Algorithms for Mining the Evolution of Conserved Relational States in Dynamic Networks,”
Knowledge and Information Systems, vol. 33, no. 3, pp. 603‐630, Dec. 2012.
 M.H. Alam, J.W. Ha, and S.K. Lee, “Novel Approaches to Crawling Important Pages Early,” Knowledge and Information Systems,

L
vol. 33, no. 3, pp 707‐734, Dec. 2012.
 S. Aral and D. Walker, “Identifying Influential and Susceptible Members of Social Networks,” Science, vol. 337, pp. 337‐341,

E
2012.
 A. Machanavajjhala and J.P. Reiter, “Big Privacy: Protecting Confidentiality in Big Data,” ACM Crossroads, vol. 19, no. 1, pp. 20‐
23, 2012.

PT
 S. Banerjee and N. Agarwal, “Analyzing Collective Behavior from Blogs Using Swarm Intelligence,” Knowledge and Information
Systems, vol. 33, no. 3, pp. 523‐547, Dec. 2012.
 E. Birney, “The Making of ENCODE: Lessons for Big‐Data Projects,” Nature, vol. 489, pp. 49‐51, 2012.
 S. Borgatti, A. Mehra, D. Brass, and G. Labianca, “Network Analysis in the Social Sciences,” Science, vol. 323, pp. 892‐895, 2009.
 J. Bughin, M. Chui, and J. Manyika, Clouds, Big Data, and Smart Assets: Ten Tech‐Enabled Business Trends to Watch. McKinSey
Quarterly, 2010.
N
 D. Centola, “The Spread of Behavior in an Online Social Network Experiment,” Science, vol. 329, pp. 1194‐1197, 2010.
 http://hadoop.apache.org/

CSE, IIT Kharagpur Introduction to Internet of Things 27

 EL
PT
N
Introduction to Internet of Things 28