Volume 7, Issue 8, August – 2022 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Web Crawler: A Survey

S. S. Bhamare
School of Computer Sciences
Kavayitri Bahinabai Chaudhari North Maharashtra University
Jalgaon (M.S) India

Abstract:- In world wide web, Web Crawler is working as In Web, generally Web pages formatted in HTML (i.e.
a software agent which helps in web indexing. Web Hyper Text Markup Language) are found on this network of
crawler at times called as spider or internet bot explicitly computers. Contains of web pages such as pictures, videos
operated by many search engines. In Information and other online content can be accessed through a different
Retrieval to colleting an information Web crawler play an Web browser.
important role. For effective searching and web indexing
is mostly depends on web crawlers. General Purpose, Web crawling supports information retrieval for
Distributed and Focused crawling are the types of automatic retrieval of appropriate documents, while at the
crawling techniques of web crawling. This paper same time it makes sure that retrieval of irrelevant documents
discussed overall survey on web crawlers, its types and is also avoided.
working of web crawlers.
The main aim of web crawler is to make content index
Keywords:- Web Crawler, Crawling techniques, WWW, of web sites through the Internet thus these websites can be
Search engine. seen in results of search engine. Web crawler, or spider is
typically functioned by search engines like Google and Bing
I. INTRODUCTION etc.

World Wide Web is “a wide-area hypermedia Generally, crawling activity of web crawlers begin by
information retrieval initiative aiming to give universal downloading the robot.txt file of websites. This text file
access to a large universe of documents.” In simpler terms, consists of sitemaps that list the URLs of web site the search
the Web is a computer network that allows users of one engine can crawl. Once web crawlers start crawling a web
computer to access information stored on another through the page, new web pages are discovering through links. All these
world-wide network called the Internet [22]. The Web's crawlers contain newly discovered URLs to the crawl queue
application follows a standard client-server model. Currently, so that they can be crawled later and index every individual
the World Wide Web (WWW) contains massive amounts of page that is connected to others by these Web crawlers.
web pages which are accessible by users. This provides a
very useful and helpful means of collecting information. Web pages change frequently, and it is also necessary to
identify exactly how search engines should crawl them. To
Web Search is the most important application on the determine factors various algorithms are used by Web Search
World Wide Web. It is based on information retrieval (IR), engine crawlers like how frequently an existing page would
which is an area of research that enables the user to discover be recrawled and how many number of web pages on a web
essential information from a large collection of text site would be indexed.
documents. An IR system discovers a set of documents that is
associated to the query from its underlying collection. In III. WEB CRAWLING TECHNIQUES
information retrieval through search engines, web crawler
plays significant role for searching and indexing the web Mostly Web Crawlers are used following web crawling
documents. techniques:
A. General Purpose Crawling
II. WEB CRAWLER AND ITS WORKING
In this general-purpose crawling techniques Web Crawler
In last decade World Wide Web expand very rapidly gathers all the pages from a specific collection of URLs and
makes it the largest widely accessible data source in the their links. To fetch many pages from different locations this
world. The amount of data/information on the Web is huge technique helps to the crawler. Due to fetching all the pages
and is still growing. this technique can slow down the speed and network
bandwidth.
Most of the information on the Web is linked. There are
Hyperlinks among various Web pages within and across B. Focused Crawling
different web sites. Within a site, hyperlinks manage In focused crawler is used to collect documents only on a
information structures. Implicit transmission of authority to particular or focus topic. This technique reduces amount of
the focus pages is represented by hyperlinks in various sites network traffic and downloads. Main objective of this
focused crawler is for search only selective web pages that
are appropriate to a particular set of problems. The related
sections only crawls of the web and have an advantage to
considerable savings of hardware and network resources.

IJISRT22AUG259 www.ijisrt.com 613

Volume 7, Issue 8, August – 2022 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
C. Distributed Crawling [11] present a new Genetic PageRank Algorithms, is a
In distributed crawling, it allows users to extend their search and optimization technique which is used in
individual computing and bandwidth resources to crawling computing to find optimal solutions.
web pages by increasing the load of these tasks across many
computers. Several methods are used to crawl and download [12] proposed a method for detecting web crawlers in
pages from the Web [20]. real time. Researchers use decision trees to categorize
requests in real time, as start from a crawler or human, while
IV. LITERATURE SURVEY their session is ongoing.

In literature survey, there exist few research work on [13] offered a method to correctly determine the quality
web crawler carried out by many researchers to improve the of hyperlinks that is not retrieved but this hyperlink is
effectiveness of web crawler, accessible to them. So that researcher applies an algorithm
such as AntNet routing algorithm.
[2] The key objective of the paper is to extract data
from websites using provided hyperlinks. This extracted data [14] present a method which employs mobile agents to
are mainly unstructured data. In the end, the authors show a crawl the web pages. These mobile crawlers filter out any
difference between TF-IDF algorithm and the BFS algorithm unnecessary data locally before moved it back to the search
to show the accuracy rate suggesting TF-IDF algorithm engine. These mobile crawlers decrease the network load by
offers more precise results. reducing the amount of data transmitted over the network.

[3] New web crawler was implemented that is also [20] discussed different existing research work on Web
called a web spider, that browses the web in a methodical Crawlers and its techniques has been carried out by various
manner to gather information. This system was designed, researchers.
developed, and implemented using python. To develop this
system an algorithmic program is designed for implemented V. CHALLENGES AND ISSUES
the developed module on the required system.
The main challenge is increasing size of web. Every day
[6] are suggested the design of an efficient parallel large number of web pages emerging on web and crawling of
crawler. Due to size of web increases rapidly every day. It is this large size of web is new challenge for the web crawlers.
necessary to make a crawling process parallelize. In a The major challenge is also associated with crawling
sufficient amount of time, it supports to complete multimedia data, Web crawler execution time and scaling of
downloading web pages. Researchers propose system of the web size.
measurement to evaluate a parallel web crawler and compare
the proposed architectures using lots of pages collected from Many methods and options have been designed and
the Web. developed according to above challenges. Due to rate of
increase of data on web makes web crawling a more
[7] proposed a novel model and its design of the Web challenging and difficult task. Here is need that continuously
Crawler using multiple HTTP connections to World Wide updating in web crawling algorithms to match up with the
Web. It begins with a URL, Crawler visits the URL and it requiring data from users and increasing data on web.
detects all the hyperlinks available in the web page and adds
them to the list of URLs to visit, known as the crawl frontier. VI. CONCLUSION
Up to the level five from every home page of web sites URLs Web crawlers plays an important role for all search
is repeatedly visited and then stops while trying to retrieve engines which helps to search web data. To develop an
information from the internet. efficient web crawler that match with todays need is not
[9] proposed design architecture for execute multiple difficult task. Developing such web crawler with proper
crawling processes (C-procs) as a parallel crawler. Each approach and architecture leads to implement a smart web
process executes the dynamic tasks as a single process crawler for smart searching. Many researchers developed
crawler executes. By downloading pages from the World various web crawler algorithms for searching but smart web
Wide Web, it stores the web pages locally and extracts URLs crawler algorithms need to implement for better results and
from web pages by following their hyperlinks. The crawling high performance.
process executing these tasks may be over on the same local REFERENCES
network or at geographically remote locations.
[1.] D. Debraj and P. Das, "Study of deep web and a new
[10] proposed and developed new web Crawler called form-based crawling technique," International Journal
PyBot which is based on algorithm standard Breadth-First of Computer Engineering and Technology (IJCET),
Search strategy. Primarily this crawler takes a URL and it’s Vol. 7, No. 1, pp. 36-44, 2016.
all hyperlinks. From that page hyperlinks, crawler crawls [2.] Ahmed, Tanvir & Chung, Mokdong ―Design and
again till to the point that have a no other hyperlinks found. application of intelligent dynamic crawler for web data
This crawler crawls and downloads all the Web Pages. It mining, Korea Multimedia Society, Spring Conference
uses download web pages and web structure in Excel CSV 2019.
format for the ranking pages, page rank algorithm used for [3.] F. M. Javed Mehedi Shamrat, Zarrin Tasnim, A.K.M
produces ranking order of pages with page list. Sazzadur Rahman, Naimul Islam Nobel, Syed Akhter

IJISRT22AUG259 www.ijisrt.com 614

Volume 7, Issue 8, August – 2022 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Hossain An Effective Implementation Of Web Crawling [18.] Douglis, F., A. Feldmann, B. Krishnamurthy, and J.
Technology To Retrieve Data From The World Wide Mogul. Rate of change and other metrics: A live study
Web (Www) International Journal Of Scientific & of the World Wide Web. USENIX Symposium on
Technology Research Volume 9, Issue 01, January 2020 Internet Technologies and Systems, 1997.
Issn 2277-8616 1252 Ijstr©2020 www.ijstr.org [19.] Fetterly, D., M. Manasse, M. Najork, and J. Wiener. A
[4.] Z. Guojun, J. Wenchao, S. Jihui, S. Fan, Z. Hao, L. large-scale study of the evolution of Web pages. WWW
Jiang, et al., "Design and application of intelligent ‘03, 669-678, 2003.
dynamic crawler for web data mining," Proceeding of [20.] Md. Abu Kausar, V. S. Dhaka, Sanjeev Kumar Singh
2017 32nd Youth Aca demic Annual Conference of “Web Crawler: A Review” International Journal of
Chinese Associ ation of Automation (YAC) IEEE , pp. Computer Applications (0975 – 8887) Volume 63–
1098-1105, 2017. No.2, February 2013.
[5.] Berners-Lee, Tim, “The World Wide Web: Past, Present [21.] Kim, J. K., and S. H. Lee. An empirical study of the
and Future”, MIT USA, Aug 1996, available at: change of Web pages. APWeb ‘05, 632-642, 2005.
http://www.w3.org/People/Berners-Lee/1996/ppf.html. [22.] The World-Wide Web, Henrik Frystyk, July 1994
[6.] Junghoo Cho and Hector Garcia-Molina “Parallel https://www.w3.org/People/Frystyk/thesis/WWW.html
Crawlers”. Proceedings of the 11th international
conference on World Wide Web WWW '02”, May 7–
11, 2002, Honolulu, Hawaii, USA. ACM 1-58113-449-
5/02/0005.
[7.] Rajashree Shettar, Dr. Shobha G, “Web Crawler On
Client Machine”, Proceedings of the International
MultiConference of Engineers and Computer Scientists
2008 Vol II IMECS 2008, 19-21 March, 2008, Hong
Kong
[8.] K. Sharma, J.P. Gupta and D. P. Agarwal
“PARCAHYD: An Architecture of a Parallel Crawler
based on Augmented Hypertext Documents”,
International Journal of Advancements in Technology,
pp. 270-283, October 2010.
[9.] Shruti Sharma, A.K.Sharma and J.P.Gupta “A Novel
Architecture of a Parallel Web Crawler”, International
Journal of Computer Applications (0975 – 8887)
Volume 14– No.4, pp. 38-42, January 2011
[10.] Alex Goh Kwang Leng, Ravi Kumar P, Ashutosh
Kumar Singh and Rajendra Kumar Dash “PyBot: An
Algorithm for Web Crawling”, IEEE 2011
[11.] Lili Yana, Zhanji Guia, Wencai Dub and Qingju Guoa
“An Improved PageRank Method based on Genetic
Algorithm for Web Search”, Procedia Engineering, pp.
2983-2987, Elsevier 2011
[12.] Andoena Balla, Athena Stassopoulou and Marios D.
Dikaiakos (2011), “Real-time Web Crawler Detection”,
18th International Conference on Telecommunications,
pp. 428-432, 2011
[13.] Bahador Saket and Farnaz Behrang “A New Crawling
Method Based on AntNet Genetic and Routing
Algorithms”, International Symposium on Computing,
Communication, and Control, pp. 350-355, IACSIT
Press, Singapore, 2011
[14.] Anbukodi.S and Muthu Manickam.K “Reducing Web
Crawler Overhead using Mobile Crawler”,
PROCEEDINGS OF ICETECT, pp. 926-932, 2011
[15.] K. S. Kim, K. Y. Kim, K. H. Lee, T. K. Kim, and W. S.
Cho “Design and Implementation of Web Crawler
Based on Dynamic Web Collection Cycle”, pp. 562-
566, IEEE 2012
[16.] MetaCrawler Search Engine, available at:
http://www.metacrawler.com.
[17.] Cho, J. and H. Garcia-Molina. The evolution of the
Web and implications for an incremental crawler.
VLDB ’00, 200-209, 2000.

IJISRT22AUG259 www.ijisrt.com 615