See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/342961223

Recommendation Systems in Big Data Era

Research · October 2019

CITATION READS

1 2,477

4 authors, including:

Neelam Singh Devesh Pratap Singh

Graphic Era University Graphic Era University
29 PUBLICATIONS 102 CITATIONS 98 PUBLICATIONS 656 CITATIONS

SEE PROFILE SEE PROFILE

Bhasker Pant

38 PUBLICATIONS 146 CITATIONS

SEE PROFILE

All content following this page was uploaded by Neelam Singh on 28 July 2020.

The user has requested enhancement of the downloaded file.

 International Journal of Innovative Technology and Exploring Engineering (IJITEE)
ISSN: 2278-3075, Volume-8 Issue-12S3, October 2019

Recommendation Systems in the Big Data Era

Neelam Singh, Shivanshi Tripathi, Devesh Pratap Singh, Bhasker Pant, Vijay Kumar

Abstract—Rapid progression in technology and increasing 1990‟s marked the beginning of research on recommender
use of social media platforms like Facebook, Instagram and systems and from that time onwards research in this area
Twitter has altered the way of articulating people’s judgment, got expanded with introduction to new approaches with
observation and sentiments about specific product, services, more better and accurate recommendation result. The task
and more. This leads to the production and accumulation of of information filtering in recommender systems can be
massive amount of data. Recommendation systems are getting
performed by the following three algorithms, namely
impetus when it comes to find insights from this data to make
decisions that can be represented in various statistical and “collaborative filtering”, “content based filtering” and
graphical forms. They have proven useful in predicting or “hybrid filtering”[1].
recommending products ranging from food, movies, Eventually novel strategies progressed from these basic
restaurants etc. This paper presents an overview about
categories with improved recommendations with the
recommendation systems and a review of generation of
recommendation methods based on categories like content- inclusion of information from social networking platforms,
based, collaborative, and hybrid approaches. The paper will Internet of Things, location information, and genetic
enlist the limitations which the present recommendation algorithm based methods etc. Recommender has gained
system faces and the possible improvements required in their impetus in both industry as well as academia with more
capabilities to fit into a wider range of application areas. advance research undergoing in these sectors.
Recommender systems offers a wide range of possibilities
Index Terms:Recommender system, Recommendation, and challenges for research as well as implementations. A
Models, Data analytics. wide range of applications includes recommendations in
web search, books, movies, music, restaurants, food,
I. INTRODUCTION apparels, vehicles, targeted advertisements, medicines,
news, probable consumers for companies and many more.
Recommendation has become the need of the hour and has Recommender systems are extensively employed by E-
drastically changed the interaction among user and web commerce platforms for improving user experience and
sites. Felds like education, finance and scientific research, thus increasing the usage. These system enables to convert a
etc are some application areas of the recommender systems. web surfer into a buyer by giving suggestions.
Rapid advancement in the field of information technology
has increase the volume of digital information available and The paper reviews recommender algorithms, their
being generated. classification based in methodology used and application
area. The paper organization is given as section II
Search Engines like Google and Bing are often used in this deliberates several type of recommendation approaches
field of machine learning to search and filter the with a comparison among the methods presented in section
information based on movies, music, or articles by III. Section IV enumerates challenges and limitations
deploying big data analysis tools and techniques. By addressed by recommender systems are listed and also how
implementing recommender systems on websites various e- recommendation system works with big data is listed in
commerce based organizations and retail companies section V and in section VI we conclude and future scope is
leveraging the advantage of data and advancing their sales. discussed.
Hence, these systems intend to forecast users‟ interests and
also, prescribe items that are much off their interests for II. TYPES OF RECOMMENDATION SYSTEM
them. User ratings provided after watching a movie or
purchase of an article, from search engine queries or Recommender system deals in two types of information
feedback becomes a key source of data needed for such as:
recommender systems. Applications like Netflix or  Characteristic information. This category of
YouTube suggests playlist or video recommendations by information relates to values like keywords,
utilizing the data. categories, etc and users information about their
favorites, profiles, etc.
 User-item interactions. It contains values like
ratings, quantity of buying, likes, etc.

On the basis of information, recommender system are

Revised Manuscript Received on October 02, 2019.
Neelam Singh, Dept. of Computer Science and Engineering, Graphic classified as [1] as shown in fig.1:
Era deemed to be University, Dehradun, India. a. content-based, employing characteristic information,
Shivanshi Tripathi, Dept. of Computer Application, Graphic Era b. collaborative filtering, which utilizes user-item
deemed to be University, Dehradun, India. interactions.
Devesh Pratap Singh, Dept. of Computer Science and Engineering,
Graphic Era deemed to be University, Dehradun, India
Bhasker Pant, Dept. of Computer Science and Engineering, Graphic
Era deemed to be University, Dehradun, India.
Vijay Kumar, Dept. of Physics, Graphic Era Hill University,
Dehradun, India.

Published By:
Retrieval Number: L100610812S319/2019©BEIESP Blue Eyes Intelligence Engineering
DOI: 10.35940/ijitee.L1006.10812S319 80 and Sciences Publication
 Recommendation Systems in the Big Data Era

c. Hybrid systems that merge both types of information in Attribute common among multiple objects are given higher
order to avoid conflicts that are incorporated when working preference as compared to others. These attribute weight
with only one type of information [2]. together with the history plays an important role in
generating user preference model. All objects in the
database are compared with the model and values are
A. Recommender system utilizing Collaborative allotted based on its likeness with the user profile and then
filtering: scores recommendations are made.
To create tailored recommendations on the Web, After examining the User-Based and Item-Based
collaborative filtering (CF) method is frequently used. Collaborative methods that utilizes the interactions of the
Popular websites example Amazon, Netflix, iTunes, etc. users with the various types of items in order to build
utilizes collaborative filtering technology. Algorithms for recommendations, we recognizes that these systems faces
collaborative filtering are employed for automatic some problems like:
predictions about a user's interests by accumulating likings  Cold-start problem for new users.
of various other users[3]. For example, a site such as Netflix  New-item problem.
may recommend that the users who watch movie A and B  Scarcity problem.
may watch movie C as well. This could be performed by  Transparency issues
comparing the past preferences of those who have watched Content-Based Recommender Systems are being generated
the same movies. Thus, Collaborative Filtering doesn‟t by taking into consideration the content of each item for
require anything else other than users‟ past preferences on a recommendation purpose, and trying to resolve the problems
set of items. The core assumption here is that the users past that are illustrated above. A content based system requires
approval is likely to be considered for agreeing in future as the data given by the user, either explicitly or implicitly.
well. Hence based on user preferences, Collaborative Thus depending upon the data provided, a user profile is
Filtering can be expressed into two categories as [3]: produced, which can be further used to generate
The first category is the explicit rating in which user rates recommendations[5].
an item on a sliding scale, like 5 stars for Samsung Galaxy  Merits of content based filtering:
S10. This is the most straight away feedback from users that
a. User independence: Unlike collaborative filtering,
shows how much they liked an item.
Second category is Implicit Rating that recommends user content based method only examines the items and
preferences indirectly based on shopping history, page user profile for suggestion.
views, clicks etc[4]. b. Transparency: Collaborative Filtering provides you
The methods for performing collaborative filtering are with the suggestions based on some anonymous
nearest neighborhood and Matrix factorization. users, who have the similar choices, but content
 Benefits of Collaborative Filtering centered
based filtering suggest the items based on the
Recommender System[4]:
a. Collaborative filtering focused systems are more available features.
flexible as they can be applied to any domain, and c. No Cold start problem: In case of Content Based
when used in appropriate way can also give inter method, before grading by a given number of users
domain recommendations. new items can be advised [6].
b. Collaborative filtering based systems perform best on  Challenges with content based filtering[6]:
a large user space. a. Content Based method requires items Meta data, thus
c. Collaborative filtering engines can overcome "filter entail domain modeling and it asserts problem if their
bubble" problem, as user finds and connects subspace extension is applied to different domains.
in the item space. b. CB engines are relatively impervious to user size.
 Limitations to Collaborative Filtering based c. CB engines are focused on providing items relevant
Recommender System: and similar to user tastes as there can be counter
a. New rater problem: As a CF drawback, they suffer actions to increase variety.
from the "new item" problem much more than CB
engines (both the engines are constrained roughly
equally by the "new user" difficulty, though their
resolutions for initial profiling might vary).
b. Transparency: Collaborative filtering provides you
the recommendations depending upon some
unknown users those have the same taste like you, so
they are not able to tell what features are responsible
for the recommendation.
B. Content based recommender system:
In order to give suggestions to users this method of
recommender system entails two significant categories of
information. The primary type of information required is
the supplementary information related to the item provided
by a set of attributes assigned to them. Another information
required is user profile to find out the user interactions with
items with a specific set of attributes.

Published By:
Retrieval Number: L100610812S319/2019©BEIESP Blue Eyes Intelligence Engineering
DOI: 10.35940/ijitee.L1006.10812S319 81 and Sciences Publication
 International Journal of Innovative Technology and Exploring Engineering (IJITEE)
ISSN: 2278-3075, Volume-8 Issue-12S3, October 2019

Fig. 4 Hybrid type 3

4. Attributes of both collaborative filtering and

content based filtering are merged to develop
Combination model as displayed in fig. 5.

Fig. 1 Classification of Recommendation Sytem

C. Hybrid based recommender system:

These systems takes the advantage of both collaborative and
content based filtering by evading their distinct restrictions.
Fig. 5 Hybrid type 4
Joining collaborative and content based filtering methods to
define a fusion system can be performed in many ways as
classified below [7]: Hybrid Recommendation Systems classification:
1. Execution of both Collaborative and content based Taxonomy for the hybrid recommendation systems
filtering is done separately and afterwards their presented by Burke categorizes Hybrid Recommendation
results are combined as shown in fig.2. Systems into seven classes as following [7] [8]:
1. Weighted hybrid: Hybrid class in which additive
formula is taken to find cumulative scores of each
class.
2. Switching hybrid: In this approach a section from
the available recommendation components system
is selected and applied.
3. Mixed hybrid: Suggestion of different
recommender system are merged and multiple
rated lists are transformed into single rated list.
4. Feature Combination hybrid: Two sections namely
contributing and actual recommender are used.
Actual recommender is dependent on altered data
Fig. 2 Hybrid type 1
through contributing method where it throws
2. By applying collaborative filtering properties for features of one source on to the other sections
content based recommendation as shown in fig.3. source.
5. Feature Augmentation hybrid: Analogous to
feature combination method with more flexibility
as in this case contributor gives novel
characteristic.
6. Cascade hybrid: In this class plays tie breaker
method is used such that for every recommender
priority is assigned and according to that lower
priority recommenders plays a tie breakers part
over higher priority.
Fig. 3 Hybrid type 2 7. Meta-level hybrid: Both contributing and actual
recommenders exist but the initial section entirely
3. Fig. 4 discuss the use of content based filtering substitutes the data for the latter section.
properties in collaborative based method.

Published By:
Retrieval Number: L100610812S319/2019©BEIESP Blue Eyes Intelligence Engineering
DOI: 10.35940/ijitee.L1006.10812S319 82 and Sciences Publication
 Recommendation Systems in the Big Data Era

III. COMPARISON AMONG METHODS Working of Big Data based Recommendation System
For comparing the techniques available for building the A Recommendation system with big data is divided into
recommender system we have plotted a bar graph in python precise, rational phases as data acquisition, ratings, and
using pandas based on the Fig. 6 and table 1 below: filtering. And these phases are discussed as under [10]:

A. Data Acquisition phase

For example we consider that a user of YouTube or any
other website is browsing a video or a movie for watching.
Each time the user searches for a video or a movie, thus
based on the search there comes n-number of results for that
user. Depending upon the technology used, the event
category could differ. Event is then entered into a NoSQL
database. This entry is technical in nature but for a laymans
it could be interpreted as something like “User a watched
movie Z once or twice”. This is how record are being
captured and stored for upcoming suggestions.
For example, a customer who searched YouTube for Kabir
Singh‟s “tujhe kitna chahne lage hai” was interestingly
given several recommendations on other songs of this
searched movie as shown in fig. 7:

Fig. 6 Model comparison

Table 1 Comparison chart between among the models

Fig. 7 Recommended list of songs

B. Ratings phase
Rating plays a significant role in telling about a user‟s
perception for a product or an item. User‟s feelings for an
item can be revealed to a level according to their actions
such as likes, addition to watch list, watching or only
searching. Based on user actions, Recommendation systems
can provide implicit ratings. 5 are given as the highest
rating which a user can give to an item. Let suppose, 4
rating can be assigned to watching, 3 to likes, searching can
IV. BIG DATA AND THE RECOMMENDER SYSTEM get 2 and so on. Recommender engines also consider the
Our day to day life has been majorly impacted by the ratings and the feedback given by the users.
existence of the Recommendation systems. Let‟s take an
example for this as our online movie watching or say online C. Filtering phase
book shopping experience is being redefined. The Now this Filtering phase deals with the cleaning of the item
Recommendation system provides suggestions of items one details means based on ratings and other customer details
person might be interested in as that person browses provided with the help of big data techniques. Thus, based
through the available items. Despite of the facts that on the data provided, Recommender systems works on three
business or consumer, Recommendation systems have been classes of filtering techniques as: collaborative, user-based
vastly valuable. Thus, the main concept behind the and hybrid methods. In collaborative filtering,
implementation of the Recommendation systems is big data. commendations are provided on an assessment of user‟s
A job in a classic Recommendation system is totally based preferences. Let suppose an example, if customer A likes
on the availability of sufficient data failing which it cannot products such as @, #, $, and % and customer B likes
perform. Thus, to suit this purpose big data supply excess products such as @, #, $, % and &, then it is likely that
amount of user‟s data in the form of past purchases, customer A will be provided with the recommendation of
browsing patterns and history, and feedback to the the product „&‟ as and when seen between the similarities
Recommendation systems in order to generate significant of both user A and B, there are lot more in common. Thus,
and effective suggestions [9]. Without the use of big data, depending upon similarities the recommendation has been
even the most advanced recommendations may not be done.
effective.

Published By:
Retrieval Number: L100610812S319/2019©BEIESP Blue Eyes Intelligence Engineering
DOI: 10.35940/ijitee.L1006.10812S319 83 and Sciences Publication
 International Journal of Innovative Technology and Exploring Engineering (IJITEE)
ISSN: 2278-3075, Volume-8 Issue-12S3, October 2019

V. LIMITATIONS OF RECOMMENDATION instantaneously and chronologically. Another method

SYSTEM provided to address the scalability issue is to perform
Recommendations generated by recommender system are computation on multiple machines in parallel using
based on data analysis which is susceptible to imperfection. distributed algorithm[18].
The shortcomings of recommendation system are discussed
in the subsequent section [11]-[12]: D. Overspecialization
A. Cold Start Problem Overspecialization in recommender system leads to
Cold start problem means to a condition in which the suggesting items highly rated by the user thus can end up
recommendation engine is not able to provide necessary and providing recommendation of items already purchased or
required predictions or suggestions because of the experienced by the user. This can make user loose interest
deficiency of primary ratings of a user or an article[13]. in the system because it does not work according to user
Also, this problem in recommender system can be found in preferences. Neighborhood collaborative filtering is
two scenario i.e. when a new item or user is provided to the proposed to handle overspecialization [19]. Randomness
system. The problem of new item is found when there is an introduction using genetic algorithm or by removing similar
addition of an item into the system and that item does not items was also proposed.
have any primary rating before. Thus, the likelihood of
suggesting an unrated product is minimum and thus they E. Serendipity
might become unobserved. In order to solve this problem A very crucial objective to be achieved by each
the most likely way is to select a group of users who will recommender system is Serendipity, which focus on
give grating every new product. Major intention behind the attaining user trust and loyalty. In contrast to user previous
new user problem is deficit of ratings for new user. Thus, in rating for items they will get significantly novel and
this condition, suggesting anything to a new user is relevant suggestions. It is challenging to apprehend the idea
impossible. When the user provide their initial ratings in the of serendipity completely due to its subjective nature and is
system they starts getting suggestions. But there will be a rarely seen in real life scenarios. Solutions like re-ranking
problem that the system will not be able to provision the the accuracy results has been introduced to achieve
required and necessary recommendation to the new user serendipity [20].
because of the redundancy or the lack of the data provided
by new user [13]. VI. CONCLUSION
B. Sparsity Recommender System research is on the path of
Cold start is not the only problem with recommender improvement and standardization leading to better accuracy
system and relevance. New implementations and works came into
Another problem faced is of Sparsity which is the situation existence which are promising but the approach needs to be
in which the number of existing items to be rated are much more focused towards removing the underlying challenges
more in comparison to already rated objects by the user. faced by recommender systems. In this study we enlisted
Thus, when a user article matrix is created based on the data some of the significant implementations in this field with a
available than there are very few entries that are to be comparative analysis between them using python. Also
marked which results the matrix to be sparse which results investigated challenges associated with recommender
in reduced recommendations [14][15]. So, for this problem, systems and its relationship with Big Data.
one of the possible solutions is to provide suggestions to a
user by checking user profiles and analyzing the similarities REFERENCES
so that if two users share a common interest than one
product may be recommended to other but this situation is
1. Lu, Ji et al. "Recommender system application developments: a
not really necessary to conclude only based on the
survey." Decision Support Systems, vol. 74 pp. 12-32, 2015
similarity of items that they have rated. This class of 2. Burke, Robin. "Hybrid web recommender systems." The adaptive
filtering method is also called as demographic filtering. To web. Springer, Berlin, Heidelberg. pp. 377-408, 2007
address the sparsity problem Singular Value Decomposition 3. Babu, Maddali Surendra Prasad, and Boddu Raja Sarath Kumar. "An
(SVD) to reduce the dimensionality of sparse rating matrix Implementation of the User-based Collaborative Filtering
is also proposed [16]. Algorithm." IJCSIT) International Journal of Computer Science and
Information Technologies vol. 2 no.3, pp. 1283-1286, 2011.
C. Scalability 4. Chen, Rui, et al. "A survey of collaborative filtering-based
Due to the increase in no of users, products and ratings the recommender systems: From traditional methods to hybrid methods
issue of scalability is found in recommender systems. As based on social networks." IEEE Access, vol. 6, pp. 64301-64320,
and when the information about a product or an item 2018.
5. Sorde, Roshni K., and Sachin N. Deshmukh. "Comparative study on
increases than also the recommender systems are
approaches of recommendation system." International Journal of
anticipated to answer quickly giving suggestions to the
Computer Applications, vol. 118 no. 2, 2015.
customers and this situation requires greater scalability[17]. 6. P. Nagarnaik and A. Thomas, "Survey on recommendation system
But the execution of such systems becomes strenuous and methods," 2015 2nd International Conference on Electronics and
exorbitant. Thus, the key challenge is to design an efficient Communication Systems (ICECS), Coimbatore, pp. 1603-1608,
and effective data model which can adapt itself to handle 2015. doi: 10.1109/ECS.2015. 7124857.
such massive datasets which keeps on growing. So for this
problem, solution proposed by G. Takács et al. [17] to make
use of an online learning algorithm which is capable of
processing the updates associated to each user

Published By:
Retrieval Number: L100610812S319/2019©BEIESP Blue Eyes Intelligence Engineering
DOI: 10.35940/ijitee.L1006.10812S319 84 and Sciences Publication
 Recommendation Systems in the Big Data Era

7. Choi, Keunho, et al. "A hybrid online-product recommendation University, Dehradun, India. A meritorious student with research interest
system: Combining implicit rating-based collaborative filtering and in Machine Learning, Big Data.
sequential pattern analysis." Electronic Commerce Research and
Applications vol.11, no.4, pp. 309-317, 2012. Dr. Devesh Pratap Singh, Professor and Head of
8. Mu, Ruihui. "A survey of recommender systems based on deep Computer Science and Engineering department at Graphic
Era Deemed to be University Dehradun India. He has
learning." IEEE Access, vol. 6, pp. 69009-69022. 2018.
received M. Tech degree in Computer Science and
9. Zhang, Y., 2016. GroRec: a group-centric intelligent recommender Engineering from Uttarakhand Technical University
system integrating social, mobile and big data technologies. IEEE Dehradun India in 2009. He has also received Ph.D. in 2015. His research
Transactions on Services Computing, 9(5), pp.786-795. interests include Information Security, Wireless Sensor Networks, Internet
10. Wang, Y., Wang, M. and Xu, W., 2018. A sentiment-enhanced of Things and Soft Computing. He has published more than fifty research
hybrid recommender system for movie recommendation: a big data papers in his area of expertise. He is the member of ACM.
analytics framework. Wireless Communications and Mobile
Computing, 2018.
Dr. Bhasker Pant, Dean Research & Development and
11. Bai, Xiaomei, et al. "Scientific paper recommendation: A Associate Professor in Department of Computer Science
survey." IEEE Access 7 (2019): 9324-9339. and Engineering. He is Ph.D. in Machine Learning and
12. Misale, Mohini, and Pankaj Vanwari. "A survey on recommendation Bioinformatics from MANIT, Bhopal.Has more than 15
system for technical paper reviewer assignment." 2017 International years of experience in Research and Academics. He has
conference of Electronics, Communication and Aerospace till now guided as Supervisor 3 Ph.D. candidates
Technology (ICECA). Vol. 2. IEEE, 2017. (Awarded).and 5 candidates are in advance state of work. He has also
guided 28 MTech. Students for dissertation. He has also supervised 2
13. Chakraborty, Jayeeta, and Vijay Verma. "A survey of diversification
foreign students for internship. He has more than 70 research publication in
techniques in Recommendation Systems." 2016 International National and international Journals. He has also chaired a session in
Conference on Data Mining and Advanced Computing (SAPIENCE). Robust Classification & Predictive Modelling for classification held at
IEEE, 2016. Huangshi, China.
14. Akhil, P. V., and Shelbi Joseph. "A Survey Of Recommender System
Types And Its Classification." International Journal of Advanced
Research in Computer Science 8.9 (2017).
15. Nagarnaik, Paritosh & Thomas, A. (2015). Survey on
recommendation system methods. 2nd International Conference on
Electronics and Communication Systems, ICECS 2015. 1603-1608.
10.1109/ECS.2015.7124857.
16. Sarwar, B., Karypis, G., Konstan, J. and Riedl, J., 2002, December.
Incremental singular value decomposition algorithms for highly
scalable recommender systems. In Fifth international conference on
computer and information science (Vol. 1, No. 012002, pp. 27-8).
17. Takács, G., Pilászy, I., Németh, B. and Tikk, D., 2009. Scalable
collaborative filtering approaches for large recommender
systems. The Journal of Machine Learning Research, 10, pp.623-
656.
18. Mohamed, Marwa Hussien, Mohamed Helmy Khafagy, and
Mohamed Hasan Ibrahim. "Recommender systems challenges and
solutions survey." In 2019 International Conference on Innovative
Trends in Computer Engineering (ITCE), pp. 149-155. IEEE, 2019
19. Sarwar, B.M., Karypis, G., Konstan, J. and Riedl, J., 2002,
December. Recommender systems for large-scale e-commerce:
Scalable neighborhood formation using clustering. In Proceedings of
the fifth international conference on computer and information
technology (Vol. 1, pp. 291-324). Almazro, Dhoha, et al. "A survey
paper on recommender systems." arXiv preprint
arXiv:1006.5278 (2010).
20. De Gemmis, M., Lops, P., Semeraro, G. and Musto, C., 2015. An
investigation on the serendipity problem in recommender
systems. Information Processing & Management, 51(5), pp.695-717.

AUTHORS PROFILE

Neelam Singh, Assistant Professor, Department of

Computer Science and Engineering, Graphic Era
Deemed to be University, Dehradun, India. She has
received M.Tech (CSE) degree in Computer Science and
Engineering, GEU, Dehradun, India in 2014. Her
research interests include Big Data, Machine Learning,
Data Mining and Artificial Intelligence.

Shivanshi Tripathi, Student (M.Sc IT), Department of

Computer Application, Graphic Era Deemed to be

Published By:
Retrieval Number: L100610812S319/2019©BEIESP Blue Eyes Intelligence Engineering
DOI: 10.35940/ijitee.L1006.10812S319 85 and Sciences Publication
View publication stats