Scribd Logo
Upload

Welcome to Scribd!

  • IRS Module 5 & 6 Web Search

    Uploaded by

    Pravin Shinde
    12 views37 pages
    This document discusses the components and architecture of web crawlers, including their crawling policies for selection, re-visiting pages, and politeness. It covers the key components of a web crawler like the frontier, history and page repository, fetching, parsing, URL extraction and canonicalization. The document also touches on topics like graph search problems, URL normalization, multithreaded crawlers, and crawler identification.

    Original Description:

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document discusses the components and architecture of web crawlers, including their crawling policies for selection, re-visiting pages, and politeness. It covers the key components of a web crawler like the frontier, history and page repository, fetching, parsing, URL extraction and canonicalization. The document also touches on topics like graph search problems, URL normalization, multithreaded crawlers, and crawler identification.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    12 views37 pages

    IRS Module 5 & 6 Web Search

    Uploaded by

    Pravin Shinde
    This document discusses the components and architecture of web crawlers, including their crawling policies for selection, re-visiting pages, and politeness. It covers the key components of a web crawler like the frontier, history and page repository, fetching, parsing, URL extraction and canonicalization. The document also touches on topics like graph search problems, URL normalization, multithreaded crawlers, and crawler identification.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 37

    WEB SEARCH IRS

    Prof.Pravin V.Shinde
    Web Crawler
    Use of Web Crawler
    Continue..
    Crawling Policies
    Selection Policy
    Re-Visit Policy
    Re-Visit Policy
    Politeness Policy
    Parallelization Policy
    Components
    Contd..
    Contd..
    Web Crawler Architecture
    Crawling Infrastructure
    Contd..
    Graph Search Problem
    Contd..
    Frontier
    History and Page Repository
    Contd..
    Fetching
    Fetching Contd..
    Parsing
    URL Extraction and
    Canonicalization
    Canonicalization Procedure
    Stoplisting
    Stemming
    HTML Tag Tree
    Example
    Contd..
    URL Normalization
    Crawler Identification
    Multithreaded Crawler
    Contd..
    Thank You

    You might also like