Algorithmic techniques and tools used in web

content mining
Firth Author#1, Second Author*2, Third Author#3
“First-Third Department, First-Third University”
Address Including Country Name
1first.author@firth-third.edu
3third.author@firth-third.edu

*Second Company
Address including Country Name
2second.author@second.com

Abstract - As the web is growing rapidly, we can consider
web as a pool of information. There is massive tendency of
people using web for every information’s which they want
to know. Large amount of text documents, multimedia files,
and images are available in the web and it is still increasing
its forms. That’s why we want web content mining to
extract potential data from internet. Web mining is a part
of data mining which relates to various research
communities such as information retrieval, database
management systems, and artificial intelligence. this topic
mainly focused on the web content mining tasks along with
its techniques and algorithms.
I. INTRODUCTION
With the tremendous growth of the amount of data or
information available on internet or world wide web, it is
considered as a collection of documents, images, text files and
other forms of data in structured, semi structured and
unstructured forms. it is also huge, diverse and dynamic. The
primary objective of web mining is to extract useful
information and knowledge from web. Web mining is a
multidisciplinary field. it includes data mining, machine
learning, natural language processing, statistics, database,
information retrieval, multimedia, etc. Nowadays web is
becoming the major data source for the users in many domains.
this fact increases the users on web and most of the users are
inexperienced. The web mining becomes the challenging task
due to the heterogeneity and lack of structure in web resource.
Since inexperienced, most of the web users could encounter the
following problems while interaction with the web.
This problem is a data-triggered process.
Here the web user has to extract potentially useful
information from a collection of available contents.
C. Personalizing Data
This is associated with the type and
presentation of information, as it is likely that people
differ in the contents and presentations they prefer
while interacting.
D. Analyzing Individual User Preferences
This deal with the problem of encountering
the needs of web users. This includes personalization
of individual user, website design and management,
customizing user information etc.
The web is noisy it contains mixture of many kinds of
information. The web mining techniques can be used to solve
those issues.[1][2]
II. BODY
As stated above web content mining is a challenging task
because in new society number of users on web is huge and also
high percentage of them is inexperienced so web data has
become unstructured. Also, web is noisy it contains mixture of
many kinds of information. Apart from that the web is also
dynamic because the information on the web changes
constantly. So, to apply a solution for this first of all we have to
understand what is known as web content mining. [1,2,3]

A. Finding Appropriate Information

When a user wants to find specific
information in the web, they input a simple keyword
query. The query response will be the list of pages
ranked depends on their similarity to the query. This
problem is a query-triggered process. Natural
language processing is a solution to such kind of
problems.
B. Creation of New Knowledge from the Web
 into more structured forms and indexing the information to
retrieve it quickly. These algorithms are used for web content
mining Correlation algorithm for relevance ranking and Cluster
Hierarchy Construction Algorithm (CHCA)[3,4]

So, to optimize the algorithmic approach we have to

compare each algorithm as an example we can compare Web
structure mining algorithms PageRank, weighted PageRank
and HITS. [2,3]
TABLE 1
Mining techniques
Algorithm PageRank Weighted HITS
PageRank
Mining WSM WSM WSM and
Figure 1 – Web content mining technique WCM
used
So, if we discuss what is actually web content mining, Working Computes Computes Computes
it is the Data Mining Technique That Automatically Discovers scores at scores at hub and
or Extracts the information From Web Documents. It Consists indexing indexing authority
of Following Tasks. Resource finding involves the task of time. Results time. Results scores of n
retrieving intended web documents. It is the process by which are sorted are sorted highly
we extract the data either from online or offline text resources according to according to relevant
available on web. Information selection and pre-processing importance page pages on the
involves the automatic selection and preprocessing of specific of pages. importance. fly.
information from retrieved web resources. This process I/P Blacklinks Blacklinks, Blacklinks,
Parameters forward forward
transforms the original retrieved data into information. The
links links and
transformation could be renewal of stop words, stemming or it
content
may be aimed for obtaining the desired representation such as Complexity O (log N) <O (log N) <O (log N)
finding phrases in training corpus. Generalization Limitations Query Query Topic drift
automatically discovers general patterns at individual web sites independent independent and
as well as across multiple sites. Data Mining techniques and efficiency
machine learning are used in generalization. Analysis It problem
involves the validation and interpretation of the mined patterns. Search Google Research Clever
It plays an important role in pattern mining. A human plays an Engine model
important role in information on knowledge discovery process
on web. [1]
Another approach is to use tools in web content mining to
To develop a solution to above stated problem we have reduce complexity. In that approach first we have to consider
to break web content mining into three categories those are what are Web Content Mining tools. They are software that
Web content mining, Web Structure mining, Web Usage helps to download the essential information for users as it
mining. The challenge for Web structure mining is to deal with collects appropriate and perfectly fitting information. Some of
the structure of the hyperlinks within the Web itself. Link the tools are
analysis is an old area of research. The Web contains a variety
of objects with almost no unifying structure, with differences 1. Web Info Extractor (WIE) - This is a tool for data
in the authoring style and content much greater than in mining, extracting Web content, and web content
traditional collections of text documents. The link analysis Analysis and it can extract structured or unstructured
algorithms contain PageRank, weighted PageRank and HITS. data from Web page, reform into local file or save to
Meaning of web usage mining is discovery of meaningful database, place into Web server.
patterns from data generated by client-server transactions on 2. Mozenda - This is a tool to enable users to extract and
one or more Web servers. Typical Sources of Data such as manage Web data. The Users can setup agents that
Automatically generated data stored in server access logs, normally extract, store, and also publish data to
referrer logs, agent logs, and client-side cookies and E- multiple destinations.
commerce and product-oriented user events. Web content
mining is the process of retrieving the information from WWW
 3. Screen Scrapper - This is a tool for extracting/mining REFERENCES
information from web sites. It is used for searching a
[1] R. Malarvizhi, K. Saraswathi "Web Content Mining Techniques
database, which interfaced with software to attain
Tools & Algorithms – A Comprehensive Study” International Journal
content mining needs. of Computer Trends and Technology (IJCTT), V4(8):2940-2945
4. Web Content Extractor - WCE is a powerful and easy August Issue 2013.
to use data extraction tool for Web scraping, and data
extraction from the Internet. This offers a friendly, [2] Sandhya, Mala Chaturvedi, “A Survey On Web Mining Algorithms”
wizard-driven interface that will help through the The International Journal Of Engineering And Science, (IJES) Volume
2 Issue 3 Pages 25-30 2013.
process of building a data extraction pattern and
creating crawling rules in a simple point-and click [3] Marghny, M. H., and A. F. Ali. "Web mining based on genetic
manner. algorithm." In AIML 05 Conference, pp. 19-21. 2005.
5. Automation Anywhere - AA is a Web data extraction
tool used in getting web data, screen scratch from Web
pages or use it for Web mining [3].

Another approach is to use Ranking and Evaluation of Search

Results through Web Content Mining. For that following
framework can be used [3]

To use web content mining following framework can be used

• Pre-processing
• Full-word profile generation
• Term frequency computation with the domain
directory
• Correlation co-efficient computation
• Rank the relevant document [3]

III. CONCLUSION

Web content mining is the Data Mining Technique That

Automatically Discovers or Extracts the information From
Web Documents. As the web continues to increase in size the,
complexity of web content mining is also increased. Hence
extract the information from the web becomes more complex.

To develop a solution, we have to break web content mining

into three categories those are Web content mining, Web
Structure mining, and Web Usage mining. Web structure
mining is to deal with the structure of the hyperlinks within the
Web itself. PageRank, weighted PageRank and HITS are the
used algorithms in web structure mining and they have
compared under various categories.

Apart from that, we can use tools to reduce the complexity of

web content mining. Web Info Extractor, Mozenda, Screen
Scrapper, Web Content Extractor are some well-known
software tools which are involved in web content mining. In
this research paper we have exposed some popular algorithms
and tools which is used to reduce the complexity of the web
content mining