Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## ISSearch
Class ISDBCrawler

    java.lang.Object
      ISSearch.ISDBCrawler

All Implemented Interfaces:
  ISCrawlerInterface, ISDBCrawlerInterface, java.lang.Runnable

    public class ISDBCrawler

extends java.lang.Object
implements ISDBCrawlerInterface

The Crawler class of the Web search engine. This class is used to start and stop the Crawler, to reset the engine and to control crawling parameters.

Field Summary

Fields inherited from interface ISSearch.ISCrawlerInterface
`RUNNING, STOPPED`

Constructor Summary
`ISDBCrawler()` Creates a new instance of ISCrawler

Method Summary
`void` | `addLink(java.net.URL link)` Adds a new link to the URL queue, if the link is not yet visited.
`void` | `closeDB()` Closes the database connection of the built-in database interface.
`java.net.URL` | `getBest()` Returns the best candidate to be visited next.
`java.lang.String` | `getContentType(java.net.URLConnection urlConnection)` Returns the ContentType of the current document.
`int` | `getCrawlingDepth()` Returns the current maximum allowed crawling depth.
`ISDocumentInterface` | `getCurrentDocument()` Returns the last document visited by the Crawler.
`java.net.URL` | `getCurrentURL()` Returns the last URL visited by the Crawler.
`ISDBinterface` | `getDBInterface()` Returns the built-in database interface of the crawler
`int` | `getMaxQueueSize()` Returns the maximum allowed size of the URL Queue
`java.net.URL` | `getNextURL()` Returns the next URL to be searched.
`int` | `getQueueSize()` Returns the current size of the URL queue
`int` | `getState()` Returns the current state of the crawler.
`int` | `getTimeout()` Returns the current timeout of the crawler.
`boolean` | `isDataStructureEmpty()` Checks if our data structure is empty or not.
`boolean` | `isVisited(java.net.URL doc)` Checks if the URL of the given document is already visited by the crawler.
`boolean` | `openDB()` Initializes the internal database interface and opens its database connection
`void` | `reset()` Resets the crawler.
`boolean` | `robotSafe(java.net.URL url)` Checks if there exists a robots.txt on the server and checks it contains a "Disallow:".
`void` | `run()`
`ISDocumentInterface` | `runParser(java.io.Reader r)` Starts the parser.
`void` | `setCrawlingDepth(int depth)` Sets the maximum allowed crawling depth.
`void` | `setCurrentDocument(ISDocumentInterface isd)` Sets the last document visited by the Crawler.
`void` | `setQueueMaxSize(int m)` Set the maximum allowed size of the URL queue
`void` | `setState(int state_code)` Sets the current state of the crawler.
`void` | `setTimeout(int t)` Sets the current timeout of the crawler.
`void` | `start()` Starts the thread of the crawler and changes the engine state to `RUNNING`
`void` | `stop()` Stops the crawler.
`boolean` | `store(java.net.URL link, ISDocumentInterface doc)` Stores the crawled document and its URL into the database

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### ISDBCrawler

    public ISDBCrawler()

  Creates a new instance of ISCrawler

Method Detail

### store

    public boolean store(java.net.URL link,
                         ISDocumentInterface doc)

  Stores the crawled document and its URL into the database

  Specified by:
    `store` in interface `ISDBCrawlerInterface`
  Parameters:
    `link` - the URL of the crawled document
    `doc` - extracted terms and links from the document
  Returns:
    true, if the storage was successful; false otherwise.

### openDB

    public boolean openDB()

  Initializes the internal database interface and opens its database connection

  Specified by:
    `openDB` in interface `ISDBCrawlerInterface`
  Returns:
    true, if the connection to the database was successful, false otherwise.

### closeDB

    public void closeDB()

  Closes the database connection of the built-in database interface.

  Specified by:
    `closeDB` in interface `ISDBCrawlerInterface`

### getDBInterface

    public ISDBinterface getDBInterface()

  Returns the built-in database interface of the crawler

  Specified by:
    `getDBInterface` in interface `ISDBCrawlerInterface`
  Returns:
    the database interface of the crawler

### addLink

    public void addLink(java.net.URL link)

  Adds a new link to the URL queue, if the link is not yet visited.

  Specified by:
    `addLink` in interface `ISCrawlerInterface`
  Parameters:
    `link` - The URL link representation of the new target

### getBest

    public java.net.URL getBest()

  Returns the best candidate to be visited next. The result must have the highest priority (in the sense of the selected ordering strategy) under all available links.

  Specified by:
    `getBest` in interface `ISCrawlerInterface`
  Returns:
    The best target to be visited by the Crawler next, `null` if the queue is empty.

### getCrawlingDepth

    public int getCrawlingDepth()

  Returns the current maximum allowed crawling depth.

  Specified by:
    `getCrawlingDepth` in interface `ISCrawlerInterface`
  Returns:
    The current allowed craling depth.

### getCurrentDocument

    public ISDocumentInterface getCurrentDocument()

  Returns the last document visited by the Crawler.

  Specified by:
    `getCurrentDocument` in interface `ISCrawlerInterface`
  Returns:
    The last visited document as object that implements ISDocumentInterface (and contains all extracted links, words and their stems); null if no documents were crawled yet.

### setCurrentDocument

    public void setCurrentDocument(ISDocumentInterface isd)

  Description copied from interface: `ISCrawlerInterface`
  Sets the last document visited by the Crawler.

  Specified by:
    `setCurrentDocument` in interface `ISCrawlerInterface`

### getCurrentURL

    public java.net.URL getCurrentURL()

  Returns the last URL visited by the Crawler.

  Specified by:
    `getCurrentURL` in interface `ISCrawlerInterface`
  Returns:
    The last visited URL; null if no links were crawled yet.

### getMaxQueueSize

    public int getMaxQueueSize()

  Returns the maximum allowed size of the URL Queue

  Specified by:
    `getMaxQueueSize` in interface `ISCrawlerInterface`
  Returns:
    The max allowed Queue size

### getQueueSize

    public int getQueueSize()

  Returns the current size of the URL queue

  Specified by:
    `getQueueSize` in interface `ISCrawlerInterface`
  Returns:
    The current size of the URL queue.

### getState

    public int getState()

  Returns the current state of the crawler. Possible states are `RUNNING` and `STOPPED`.

  Specified by:
    `getState` in interface `ISCrawlerInterface`
  Returns:
    The current state of the crawler, `RUNNING` oder `STOPPED`

### setState

    public void setState(int state_code)

  Description copied from interface: `ISCrawlerInterface`
  Sets the current state of the crawler. Possible states are `RUNNING` and `STOPPED`.

  Specified by:
    `setState` in interface `ISCrawlerInterface`

### getTimeout

    public int getTimeout()

  Description copied from interface: `ISCrawlerInterface`
  Returns the current timeout of the crawler.

  Specified by:
    `getTimeout` in interface `ISCrawlerInterface`
  Returns:
    The current timeout of the crawler.

### setTimeout

    public void setTimeout(int t)

  Description copied from interface: `ISCrawlerInterface`
  Sets the current timeout of the crawler.

  Specified by:
    `setTimeout` in interface `ISCrawlerInterface`

### isVisited

    public boolean isVisited(java.net.URL doc)

  Checks if the URL of the given document is already visited by the crawler.

  Specified by:
    `isVisited` in interface `ISCrawlerInterface`
  Returns:
    `true` if the engine was able to recognize the given URL as already visited, `false`.

### setCrawlingDepth

    public void setCrawlingDepth(int depth)

  Sets the maximum allowed crawling depth.

  Specified by:
    `setCrawlingDepth` in interface `ISCrawlerInterface`
  Parameters:
    `depth` - The maximum allowed craling depth.

### setQueueMaxSize

    public void setQueueMaxSize(int m)

  Set the maximum allowed size of the URL queue

  Specified by:
    `setQueueMaxSize` in interface `ISCrawlerInterface`
  Parameters:
    `m` - The maximum allowed Queue size

### start

    public void start()

  Starts the thread of the crawler and changes the engine state to `RUNNING`

  Specified by:
    `start` in interface `ISCrawlerInterface`

### stop

    public void stop()

  Stops the crawler. This method stops crawling and sets the engine status to `STOPPED`.

  Specified by:
    `stop` in interface `ISCrawlerInterface`

### reset

    public void reset()

  Resets the crawler. This method stops the crawling, resets the URL queue, and the list of visited links. Finally, it sets the crawler status to `STOPPED`,

  Specified by:
    `reset` in interface `ISCrawlerInterface`

### run

    public void run()

  Specified by:
    `run` in interface `java.lang.Runnable`

### getNextURL

    public java.net.URL getNextURL()

  Description copied from interface: `ISCrawlerInterface`
  Returns the next URL to be searched. It is doing the job for `getBest()`.

  Specified by:
    `getNextURL` in interface `ISCrawlerInterface`
  Returns:
    next URL in the queue.

### isDataStructureEmpty

    public boolean isDataStructureEmpty()

  Description copied from interface: `ISCrawlerInterface`
  Checks if our data structure is empty or not.

  Specified by:
    `isDataStructureEmpty` in interface `ISCrawlerInterface`
  Returns:
    boolean value which is true if our data structure is empty

### runParser

    public ISDocumentInterface runParser(java.io.Reader r)

  Description copied from interface: `ISCrawlerInterface`
  Starts the parser.

  Specified by:
    `runParser` in interface `ISCrawlerInterface`
  Returns:
    `ISDocumentInterface`.

### getContentType

    public java.lang.String getContentType(java.net.URLConnection urlConnection)

  Description copied from interface: `ISCrawlerInterface`
  Returns the ContentType of the current document.

  Specified by:
    `getContentType` in interface `ISCrawlerInterface`
  Parameters:
    `urlConnection` - The current document given by a URLConnection.
  Returns:
    The ContentType of the current document as String.

### robotSafe

    public boolean robotSafe(java.net.URL url)

  Description copied from interface: `ISCrawlerInterface`
  Checks if there exists a robots.txt on the server and checks it contains a "Disallow:".

  Specified by:
    `robotSafe` in interface `ISCrawlerInterface`
  Parameters:
    `url` - URL which should be checked.
  Returns:
    true if the document can be parsed.

Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD