SSRG International Journal of Electronics and Communication Engineering ( SSRG - IJECE ) - Volume 5 Issue 7 - July 2018

A Python based Virtual

A Python Assistant
based Virtual using Raspberry
Assistant using Pi
for Home
Raspberry Pi forAutomation
Home Automation
KishoreKishore
Kumar R1,R1,
Kumar Ms.Ms.J.J.Jayalakshmi 2
Jayalakshmi2, Karthik Prasanna
, Karthik S3
Prasanna S3
1
1 UG,Saveetha
UG, Department of ECE, Department of ECE, Saveetha
Engineering Engineering
College, College, India;
India; rkishorekumar1996@gmail.com
2
2 Associate
Associate Professor,
Professor, Department
Department of ECE,
of ECE, SaveethaEngineering
Saveetha Engineering College,
College, India;
India;
3UG, Department of ECE, Saveetha Engineering College, India;
jjayalakshmi@saveetha.ac.in
3
UG, Department of ECE, Saveetha Engineering College, India; karthik2096@gmail.com

various applications of IoT can be developed. Knowledge of

Abstract— The main aim of the work is to develop an Linux Operating System will be required to work with a
economically effective and performance wise efficient virtual Raspberry Pi board. It runs on a Quad core ARM Cortex A-53
assistant using Raspberry Pi for home automation based on the at a clock speed of 1.2GHz. It has a 1 Gigabyte DDR2 RAM
concepts of Internet of Things, Speech Recognition, Natural
Language Processing and Artificial Intelligence. People who are with a clock speed of 900MHz. It has inbuilt Wi-Fi and
using it can give voice inputs and the device itself responds Bluetooth connectivity.
through voice commands by itself. It can fetch the date, time,
weather, play your favourite music and fetch search results from
the internet along with controlling the home appliances.
NodeMCU chips are used to control the appliances which
receives the command from the Raspberry Pi. The Raspberry Pi
processes the speech inputs online given by the user through the
mic and converts it into text and executes the command. The
whole project is put in action through a python script which
includes online Speech to Text conversion and Text to Speech
conversion codes written. The NodeMCU is coded separately
using the Arduino IDE to make it control the appliances and
allow it to be accessed through its IP address. The device will
respond to the user in a casual manner so that the user has a
friendly experience with the device and feels it like his or her own
assistant. This device makes the day by day processes easier.
Fig.1 - Raspberry Pi 3 Model B

Index Terms—Artificial Intelligence, Home Automation NodeMCU is an Arduino-like hardware IO which comes
System, Natural Language Processing, NodeMCU, Raspberry Pi, along with an inbuilt ESP8266Wi-Fi module. It is a
Speech Recognition, Speech to Text, Text to Speech. development kit which helps us to create prototypes of our IoT
product using Lua Script. We can also use the Arduino IDE to
code the NodeMCU. NodeMCU has to be included in the IDE
I. INTRODUCTION which will help us use the kit with it.

T HIS work is constructed based on the basic concepts on

Internet of things (IoT) and Natural Language Processing
(NLP). We have put forth a method to integrate all the home
appliances through a central command center. This will
eventually reduce the human interaction with the appliances
and make the work easier. This system is designed to provide
a user friendly experience as well as an easier interface so that
anyone can use this effortlessly. This project has been
implemented with the help of Raspberry Pi 3 Model B and
NodeMCU which is an open source platform in Internet of
Things that runs on the ESP8266Wi-Fi SoC and the ESP-12
module based hardware.
Raspberry Pi is known as the credit-card sized computer
which was initially designed for education, inspired by the Fig.2 - NodeMCU
1981 BBC Micro. It is a low-cost device through which
ISSN: 2348 - 8549 www.internationaljournalssrg.org Page 22
 SSRG International Journal of Electronics and Communication Engineering ( SSRG - IJECE ) - Volume 5 Issue 7 - July 2018

III. EXISTING SYSTEM

The Internet of Things is a major breakthrough in technology. There are Home Automation Systems which are voice
It is nothing but communication of things (devices) with each controlled. The primary objective of these systems is
other using the internet as a medium. The major protocol used switching i.e., Switching ON and OFF the appliances and
in IoT is the MQTT protocol. The MQTT protocol works nothing more than that. The Existing system restricts the
based on the TCP/IP protocol. It is where the concept of possibility of technological advancements which can be put
message brokers come into play. The brokers act as a vector forth as outdated. [5]
which directs the information that is needed to be There are some systems which use mobile phones to
communicated throughout the devices. Knowing this MQTT process the Natural Languages and put them into work. But
protocol will help in better understanding of this system. Also that remains a void of a dedicated device which can be relied
basic knowledge of TCP, UDP will also be needed for better upon. So the existing system is not very much reliable. There
interpretation. In this project the devices communicate with are certain virtual assistants available as dedicated devices
each other with the help of MQTT protocol. which are very much costlier to afford. So it becomes
Natural Language Processing is simply a bridge-way that economically ineffective.
reduces the distance between human communication and Though devices like Amazon Echo, Google Home are
machine communication. The main objective of NLP is to available, additional configurations and setup maybe required
make the machines understand the natural human language so to make it viable for automation of the appliances which
that the usage becomes very much comfortable. In technical requires technical assistance. Moreover, there has to be some
terms, NLP is the algorithm which analyzes and synthesizes physical interaction with the devices to wake it up in order to
human speech. This algorithm is based on artificial perform the tasks in the existing system.[10]
intelligence and computational linguistics. [1] The Open Source platforms like Jasper which can be used
to develop voice controlled applications can be used. But it
II. RELATED WORKS IN THE FIELD requires a lot of practical knowledge in Raspberry Pi and
There are a lot of commercial companies which are into Linux Operating Systems. It takes too much time in set up as it
Home Automation like Schneider, Home Automation Inc., works default in the CMUSphinx Speech engines which are
Logitech, Honeywell, etc., They have made successful basically offline.[2][4]
attempts in integrating the home appliances to a central server As to overcome all the issues of the existing system, our
and controlling them using the same. Certain companies project works on a dedicated single python script which
include Security also. requires the most minimum configuration and setup and
Natural Language Processing is a real difficult part in this makes it easy for the installation.
domain. Many legends including Google, Amazon, Apple,
Microsoft have NLP in action in the form of Google Now, IV. PROPOSED SYSTEM
Alexa, Siri and Cortana respectively. The NLP is based on This work focuses on providing with the easiest and the
machine learning which breaks down the words to the smallest most effective method to communicate with the system by
unit called phonemes and compare it with the pre-recorded giving voice commands through natural languages. This
speech which had been trained previously. Speech to Text also project eliminates the hectic process of tiring configurations
comes under the NLP. The primary aim of NLP is to make the and setups and overheating of the system which ultimately
human to computer interaction easier by making it more like affects the performance. The conversation by natural
human to human interaction. By the term human to human languages to the system and its response makes the user feel
interaction, we mean the communication through native like he is talking to another human and makes him ignore the
languages like English, French, Spanish, etc., The computers fact that a system is performing all the tasks.
usually communicate through artificial languages like C, C++,
Java, Python, etc., Our project aims to combine all of this and A. Architecture
create a Virtual Assistant which would be handy in Home
Automation also.[9]
Devices like Amazon Alexa, Google Home are existing
systems which are dedicated personal assistants which comes
with a lot of features. But they are equally expensive which
cannot be affordable by everyone. This system is developed to
be less expensive than the existing system providing features
that they provide along with add-ons.
Few years back, Facebook CEO Mark Zuckerberg tried
developing an Artificially Intelligent personal assistant named
‘Jarvis’ inspired from the movie ‘Iron Man’. He quite
succeeded in the project. It took so much effort and so much
coding to make it possible. These are some of the related Fig.3 - Architecture
works in the field which inspired us to start with this project.

ISSN: 2348 - 8549 www.internationaljournalssrg.org Page 23

 SSRG International Journal of Electronics and Communication Engineering ( SSRG - IJECE ) - Volume 5 Issue 7 - July 2018

The Google cloud converts the speech into text and we

Fig.3 shows the architecture of the proposed system. The can fetch the data and store it in a string. This string will
work initially seemed difficult as a whole. So to make it consist of unnecessary phrases which may not be required
simpler it was divided into modules to make the work easier in the programming. So the string is broken into words and
and also for debugging. only the required words are looked for and corresponding
actions are assigned to them under separate functions.[3][5]
C. Command Module
The three modules are as follows:
 Speech to Text Module This is the module which consists most of the
 Command Module programming including the code of NodeMCU. First we
 Text to Speech Module will look at the part of NodeMCU setup. The IO of
NodeMCU is similar to that of Arduino. So people with
B. Speech to Text Module prior knowledge to Arduino will find it easier to code.
The first module is the Speech to Text Module whose Initially the NodeMCU is considered the server in which
main function is to convert the speech signals into text so the SSID and the password of the router will be provided
that it can be easier to put it in the form of a program. before-handed in the code. The hardware connections are
Initially the Raspberry Pi is set up to obtain the audio very minimal. The board is connected to a 5V relay circuit
through a USB microphone in the default mode. Also and the appliances are connected through the relay. The
ALSA configuration is done for better audio experience. relay acts as the switching. The power supply to the relay
After the basic setup is done, an STT engine must be will be given from the NodeMCU.
installed in the Raspberry Pi.
There are various STT engines namely Pocketsphinx,
Wit.ai STT, Google STT, Julius, AT&T STT. Pocketsphinx
engine is known for its offline capabilities and works well
without internet. It is specially designed to work well with
systems like Raspberry Pi. The disadvantages are that it Fig.5 - Command Module
takes so much time to configure it. It also requires a lot of
dependencies. In that case, Wit.ai or Google STT could be a Motor drivers must be used for interfacing High Voltage
better choice.[4] motors and also other suitable drivers must be used for Air
Wit.ai is designed to work based on the wit.ai cloud Conditioners (HVAC) and water heaters which uses up high
services. It uses an unique technique called the voltage for their functioning.[5]
crowdsourcing to train the voice recognition algorithms. A The software part includes assignment of a static IP to the
fast reliable internet connection is required to use this NodeMCU. Once the IP is assigned to the board, a code is
engine. written to create a webpage which consists of ON and OFF
Google STT is directly from the Google. It requires an buttons respective to the number of devices to be controlled.
active internet connection to work. It is the one that is used The buttons can be accessed through the IP which can be
in android phones for speech inputs. So it is reliable as in included in the python script of the Raspberry Pi.
recognition. Separate functions are created to fetch the date, time,
In this project, We have used Google STT which is open- weather (using any weather API), Google Search and to
source and easily accessible and known to provide access the NodeMCU through the browser. These functions
promising results. A speech recognition library is installed are programmed to be executed based on the keywords
which will be imported into the python script. The whole found by using ‘if’ statements.
script will be written based on the library. It should be noted that all the required libraries should be
properly installed in the corresponding directories to avoid
unnecessary bugs in the program.
This project was initially started with the aim of
connecting it to Jasper. The configuration of Jasper took a
lot of time and also the setup did not complete successfully
which caused many errors which eventually crashed the
Fig.4 - STT Module system repeatedly. So after deep learning about Jasper, it
was found to be nothing but a combination of many python
Initially, the microphone is assigned as source and then scripts. So we decided to create a dedicated python script
the input is acquired from the user. Before that, it is made to which will consist of only the required functions and rule
adjust itself according to the ambient noise or else the off all the unnecessary functions and modules which makes
microphone would mistake noise as inputs. Usually the it lighter and lesser complex when compared to Jasper.
system takes about a second to adapt itself to the Thus this command module serves as the brain to our
surrounding noise and get used to it. Once it gets adapted, system. Most of the processes takes place here. This script
the system will filter out the noise and take in the speech is made executable and also made to run default on startup
audio separately and sends it to the Google cloud. using Linux commands.

ISSN: 2348 - 8549 www.internationaljournalssrg.org Page 24

 SSRG International Journal of Electronics and Communication Engineering ( SSRG - IJECE ) - Volume 5 Issue 7 - July 2018

D. Text to Speech Module The script is coded in such a way that the user feels like
This Text to Speech Module is used mainly as a feedback he is talking to a person himself. It is programmed in such a
response which is executed at the successful or unsuccessful way that it gives personalized replies based on the user
completion of any task. It uses the Text to Speech Engine queries. It also is connected to Google and Wikipedia to
whose primary function is to convert the text inputs into find out the general questions that the user asks and
Voice outputs. interprets the required information and reads it out to the
There are various Text to Speech Engines available both user. The system is designed to give a personalized feel at
online and offline like eSpeak, Festival, Flite, SVOX Pico home.
TTS, Ivona TTS, Google TTS, Mary TTS, Mac OS X
TTS.[4]
eSpeak is an offline Speech synthesizer. It is very handy V. RESULTS AND DISCUSSION
and light weighted and it is compatible with many The above mentioned modules were systematically
platforms. It is open source and has easier configuration implemented using the necessary hardware and software and
process. There are choices of voice from 5 male voices and tested. The results were promising and this section consists of
4 female voices and also the speed at which the words are snap shots of some of the results.
uttered can be controlled based on the requirement.
Google TTS uses API which will require an active and
fast internet connection to synthesize the text to speech. It
sounds good yet the requirement of connectivity doubts the
privacy of the users.
Mary TTS is an offline speech synthesizer written based
on Java. To use Mary TTS, a server must be set up and it
can also be hosted on the same machine. As it takes in a lot
of steps, it is quite a hectic process. We have used the
eSpeak TTS to get the output voice feedback from the
system.
Initially the eSpeak engine is installed in the Raspberry
Pi using ‘sudo apt-get install espeak’ command. The engine
Fig.7 - Virtual Assistant Setup
can be called easily through a python script. The speed,
voice modulation can be adjusted as per the necessity and Fig.7 shows the setup of the prototype of the above stated
comfort.[4] system. The Raspberry Pi is connected with a USB
microphone for audio input and a USB powered speaker for
audio output.
The Raspberry Pi needs a 5V/2A power supply to function
properly. Any changes in the voltage rating affects the board
such as rebooting often, not performing efficiently, etc., So it
is necessary to give it a proper power supply.
The peripherals used in this prototype does not need a
separate power supply as they get their power from the USB
ports of the Raspberry Pi.

Fig.6 - TTS Module

The TTS is called for interacting with the user to receive

the voice commands to perform tasks as well as to
acknowledge the task if completed or not. The eSpeak is an Fig.8 - Output screenshot of the system
offline synthesizer so it does not require an active internet
connection. The Voice outputs from the system are pre- The proposed system was implemented and the outputs
programmed which are put in random orders. The were also printed on screen for the follow up of the working
commands are executed based on the users input in a process. Fig.8 shows the initial output during the startup of the
random manner. program. It talks to the user and acquires the output through
the microphone in the form of natural language.

ISSN: 2348 - 8549 www.internationaljournalssrg.org Page 25

 SSRG International Journal of Electronics and Communication Engineering ( SSRG - IJECE ) - Volume 5 Issue 7 - July 2018

browser opens up with the IP address corresponding to the

NodeMCU and makes the assigned pin LOW which switches
OFF the relay which in turn cuts off the supply to the
appliance which ultimately switches it off. Similarly the
motors and high voltage devices can be controlled using the
NodeMCU.
The outputs were taken from the Raspberry Pi which runs
on the latest version of the Raspbian OS at the time of
publication of this paper. A lot of troubleshooting was done
before the prototype gave successful results. It takes much
patience and repeated trials to make this a successfully
working model. The voice recognition is initially slower for
new voices. So testing must be done repeatedly to make it get
used to the voice. This is also a part of the training process.
Fig.9 - Fetching Weather
Also noise reduction codes must be included to avoid the
misinterpretation of the words.
In Fig.9, the user asks the system for the weather. The
system fetches the weather from the internet and displays and
as well as reads out the weather. Similarly, the system can
fetch the date, time, play any music you wish from YouTube, VI. CONCLUSION
perform searches and much more. The existing voice recognition based Home Automation
systems basically run by only a fixed set of commands which
makes it a stereotypic functioning. The user loses interest as
time moves on. To break the stereotype and to overcome all
the issues and problems in the existing system, this proposed
system takes in a dedicated python script which completely is
interactive not with a fixed set of commands but with
dynamically changing responses. This kind of approach makes
the user feel personal while using the system.[7][8]
The software and hardware implemented in this system are
mostly open-source and inexpensive when produced in huge
quantities will reduce the cost of production very much which
ultimately becomes affordable to almost everyone.

Fig.10 - Controlling NodeMCU (Switch ON) VII. FUTURE WORKS THAT CAN BE IMPLEMENTED
The only major drawback in any speech recognition system
Fig.10 shows the output where the user asks the system to is its dependency on an active internet connection. It is easier
switch on the lights and as a result a browser window opens to recognize speech through internet yet at times or at remote
up with an IP address along with the ON command. This places where a reliable connection may not be available all the
enables the assigned pin to HIGH which switches the relay time, the device becomes difficult to be used.
ON which in turn switches on the appliance. To overcome this, offline speech recognition system should
be implemented. It will be a great breakthrough when it comes
to independent stand-alone home automation devices or virtual
assistants because most of the Assistants are connected to the
internet.[6]
Another important improvement which can be done is the
addition of native languages. It is a hectic process yet it will be
useful to all the people who do not know English. It makes
this device usable by almost all of the people in the world. By
the addition of native languages, the device becomes much
more user friendly and easily accessible.
Machine Learning should be implemented completely into
this system. By that way, the system will be able to learn new
processes by itself and adapt to the user based on its past
experiences. This makes it easier for the user to interact with
Fig.11 - Controlling NodeMCU (Switch OFF) the assistant as well.
Memory can be improved and let the system store the new
Fig.11 shows the output where the user asks the system to information gathered from the user and use it in the future if
switch off the lights and similar to the previous output, the
ISSN: 2348 - 8549 www.internationaljournalssrg.org Page 26
 SSRG International Journal of Electronics and Communication Engineering ( SSRG - IJECE ) - Volume 5 Issue 7 - July 2018

required. Much more modules can be created based on the

necessities.

VIII. REFERENCES

[1] Mrs. Paul Jasmin Rani, Jason Bakthakumar, Praveen Kumaar B,

Praveen Kumaar U, “Voice Controlled Home Automation System Using
Natural Language Processing (NLP) and Internet Of Things (IOT)”,
International Conference on Science and Technology Engineering &
Management, March 2017.
[2] Tatjana Erić, Sandra Ivanović, Sunčica Milivojša, Milica Matić,
Nikola Smiljković, “Voice Control for Smart Home Automation:
Evaluation of Approaches and Possible Architectures”, IEEE 7th
International Conference on Consumer Electronics – Berlin (ICCE -
Berlin), September 2017.
[3] Eleonora Nan, Una Radosavac, Istvan Papp, Marija Antić,
“Architecture of Voice Control Module for Smart Home Automation
Cloud”, IEEE 7th International Conference on Consumer Electronics –
Berlin (ICCE - Berlin), September 2017.
[4] Jasper Project Documentation, [Online] Available:
http://jasperproject.github.io/documentation/
[5] Yash Mittal, Paridhi Toshniwal, Sonal Sharma, Deepika Singhal, Ruchi
Gupta, V. K. Mittal, “A Voice-Controlled Multi-Functional Smart Home
Automation System”, Annual IEEE India Conference (INDICON),
December 2015.
[6] Milica Matić, Igor Stefanović, Una Radosavac, Milan Vidaković,
“Challenges of integrating smart home automation with cloud based
voice recognition systems”, IEEE 7th International Conference on
Consumer Electronics – Berlin (ICCE - Berlin), September 2017.
[7] Sushant Kumar, S.S Solanki, “Voice and Touch Control Home
Automation”, 3rd International Conference on Recent Advances in
Information Technology (RAIT), March 2016.
[8] Sangeeta Mishra, Rohit Kankal, Sudeep Lunawat, Prashant Ram,
Vivek Gupta, “Interactive Automation Interface using android app and
Speech Recognition over wireless LAN and Internet”, International
Conference & Workshop on Electronics & Telecommunication
Engineering (ICWET), February 2016.
[9] G B Karan, Dhananjay Kumar, Kiran Pai, J. Manikandan, “Design of a
phoneme based voice controlled home automation system”, IEEE
International Conference on Consumer Electronics-Asia (ICCE-Asia),
October 2017.
[10] Ayush Panwar, Anandita Singh, Renu Kumawat, Siddharth Jaidka,
Kumkum Garg, “Eyrie smart home automation using Internet of
Things”, Computing Conference, July 2017.

ISSN: 2348 - 8549 www.internationaljournalssrg.org Page 27