Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## ISSearch
Class ISCrawler

    java.lang.Object
      ISSearch.ISCrawler

All Implemented Interfaces:
  ISCrawlerInterface, java.lang.Runnable

    public class ISCrawler

extends java.lang.Object
implements ISCrawlerInterface

The Crawler class of the Web search engine. This class is used to start and stop the Crawler, to reset the engine and to control crawling parameters.

See Also:
  `Runnable`, `Thread`, `InetAddress`, `URL`, `HttpURLConnection`, `InputStreamReader`, `BufferedReader`, `Exception`

Field Summary

Fields inherited from interface ISSearch.ISCrawlerInterface
`RUNNING, STOPPED`

Constructor Summary
`ISCrawler()` Creates a new instance of ISCrawler

Method Summary
`void` | `addLink(java.net.URL link)` Adds a new link to the URL queue, if the link is not yet visited.
`java.net.URL` | `getBest()` Returns the best candidate to be visited next.
`java.lang.String` | `getContentType(java.net.URLConnection urlConnection)` Returns the ContentType of the current document.
`int` | `getCrawlingDepth()` Returns the current maximum allowed crawling depth.
`ISDocumentInterface` | `getCurrentDocument()` Returns the last document visited by the Crawler.
`java.net.URL` | `getCurrentURL()` Returns the last URL visited by the Crawler.
`int` | `getMaxQueueSize()` Returns the maximum allowed size of the URL Queue
`java.net.URL` | `getNextURL()` Returns the next URL to be searched.
`int` | `getQueueSize()` Returns the current size of the URL queue
`int` | `getState()` Returns the current state of the crawler.
`int` | `getTimeout()` Returns the current timeout of the crawler.
`boolean` | `isDataStructureEmpty()` Checks if our data structure is empty or not.
`boolean` | `isVisited(java.net.URL doc)` Checks if the URL of the given document is already visited by the crawler.
`static void` | `main(java.lang.String[] args)`
`void` | `reset()` Resets the crawler.
`boolean` | `robotSafe(java.net.URL url)` Checks if there exists a robots.txt on the server and checks it contains a "Disallow:".
`void` | `run()`
`ISDocumentInterface` | `runParser(java.io.Reader r)` Starts the parser.
`void` | `setCrawlingDepth(int depth)` Sets the maximum allowed crawling depth.
`void` | `setCurrentDocument(ISDocumentInterface isd)` Sets the last document visited by the Crawler.
`void` | `setQueueMaxSize(int m)` Set the maximum allowed size of the URL queue
`void` | `setState(int state_code)` Sets the current state of the crawler.
`void` | `setTimeout(int t)` Sets the current timeout of the crawler.
`void` | `start()` Starts the thread of the crawler and changes the engine state to `RUNNING`
`void` | `stop()` Stops the crawler.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### ISCrawler

    public ISCrawler()

  Creates a new instance of ISCrawler

Method Detail

### addLink

    public void addLink(java.net.URL link)

  Adds a new link to the URL queue, if the link is not yet visited.

  Specified by:
    `addLink` in interface `ISCrawlerInterface`
  Parameters:
    `link` - The URL link representation of the new target

### getBest

    public java.net.URL getBest()

  Returns the best candidate to be visited next. The result must have the highest priority (in the sense of the selected ordering strategy) under all available links.

  Specified by:
    `getBest` in interface `ISCrawlerInterface`
  Returns:
    The best target to be visited by the Crawler next, `null` if the queue is empty.

### getCrawlingDepth

    public int getCrawlingDepth()

  Returns the current maximum allowed crawling depth.

  Specified by:
    `getCrawlingDepth` in interface `ISCrawlerInterface`
  Returns:
    The current allowed craling depth.

### getCurrentDocument

    public ISDocumentInterface getCurrentDocument()

  Returns the last document visited by the Crawler.

  Specified by:
    `getCurrentDocument` in interface `ISCrawlerInterface`
  Returns:
    The last visited document as object that implements ISDocumentInterface (and contains all extracted links, words and their stems); null if no documents were crawled yet.

### setCurrentDocument

    public void setCurrentDocument(ISDocumentInterface isd)

  Sets the last document visited by the Crawler.

  Specified by:
    `setCurrentDocument` in interface `ISCrawlerInterface`
  Parameters:
    `The` - last visited document as object that implements ISDocumentInterface (and contains all extracted links, words and their stems).

### getCurrentURL

    public java.net.URL getCurrentURL()

  Returns the last URL visited by the Crawler.

  Specified by:
    `getCurrentURL` in interface `ISCrawlerInterface`
  Returns:
    The last visited URL; null if no links were crawled yet.

### getMaxQueueSize

    public int getMaxQueueSize()

  Returns the maximum allowed size of the URL Queue

  Specified by:
    `getMaxQueueSize` in interface `ISCrawlerInterface`
  Returns:
    The max allowed Queue size

### getQueueSize

    public int getQueueSize()

  Returns the current size of the URL queue

  Specified by:
    `getQueueSize` in interface `ISCrawlerInterface`
  Returns:
    The current size of the URL queue.

### getState

    public int getState()

  Returns the current state of the crawler. Possible states are `RUNNING` and `STOPPED`.

  Specified by:
    `getState` in interface `ISCrawlerInterface`
  Returns:
    The current state of the crawler, `RUNNING` oder `STOPPED`

### setState

    public void setState(int state_code)

  Sets the current state of the crawler. Possible states are `RUNNING` and `STOPPED`.

  Specified by:
    `setState` in interface `ISCrawlerInterface`
  Parameters:
    `The` - current state of the crawler, `RUNNING` oder `STOPPED`

### getTimeout

    public int getTimeout()

  Returns the current timeout of the crawler.

  Specified by:
    `getTimeout` in interface `ISCrawlerInterface`
  Returns:
    The current timeout of the crawler.

### setTimeout

    public void setTimeout(int t)

  Sets the current timeout of the crawler.

  Specified by:
    `setTimeout` in interface `ISCrawlerInterface`
  Parameters:
    `t` - The current timeout of the crawler in ms.

### isVisited

    public boolean isVisited(java.net.URL doc)

  Checks if the URL of the given document is already visited by the crawler.

  Specified by:
    `isVisited` in interface `ISCrawlerInterface`
  Returns:
    `true` if the engine was able to recognize the given URL as already visited, `false`.

### setCrawlingDepth

    public void setCrawlingDepth(int depth)

  Sets the maximum allowed crawling depth.

  Specified by:
    `setCrawlingDepth` in interface `ISCrawlerInterface`
  Parameters:
    `depth` - The maximum allowed craling depth.

### setQueueMaxSize

    public void setQueueMaxSize(int m)

  Set the maximum allowed size of the URL queue

  Specified by:
    `setQueueMaxSize` in interface `ISCrawlerInterface`
  Parameters:
    `m` - The maximum allowed Queue size

### start

    public void start()

  Starts the thread of the crawler and changes the engine state to `RUNNING`

  Specified by:
    `start` in interface `ISCrawlerInterface`

### stop

    public void stop()

  Stops the crawler. This method stops crawling and sets the engine status to `STOPPED`.

  Specified by:
    `stop` in interface `ISCrawlerInterface`

### reset

    public void reset()

  Resets the crawler. This method stops the crawling, resets the URL queue, and the list of visited links. Finally, it sets the crawler status to `STOPPED`,

  Specified by:
    `reset` in interface `ISCrawlerInterface`

### run

    public void run()

  Specified by:
    `run` in interface `java.lang.Runnable`

### getNextURL

    public java.net.URL getNextURL()

  Returns the next URL to be searched. It is doing the job for `getBest()`.

  Specified by:
    `getNextURL` in interface `ISCrawlerInterface`
  Returns:
    next URL in the queue.

### isDataStructureEmpty

    public boolean isDataStructureEmpty()

  Checks if our data structure is empty or not.

  Specified by:
    `isDataStructureEmpty` in interface `ISCrawlerInterface`
  Returns:
    boolean value which is true if our data structure is empty

### runParser

    public ISDocumentInterface runParser(java.io.Reader r)

  Starts the parser.

  Specified by:
    `runParser` in interface `ISCrawlerInterface`
  Parameters:
    `Reader` - which contains the URL to be parsed.
  Returns:
    `ISDocumentInterface`.

### robotSafe

    public boolean robotSafe(java.net.URL url)

  Checks if there exists a robots.txt on the server and checks it contains a "Disallow:".

  Specified by:
    `robotSafe` in interface `ISCrawlerInterface`
  Parameters:
    `url` - URL which should be checked.
  Returns:
    true if the document can be parsed.

### getContentType

    public java.lang.String getContentType(java.net.URLConnection urlConnection)

  Returns the ContentType of the current document.

  Specified by:
    `getContentType` in interface `ISCrawlerInterface`
  Parameters:
    `urlConnection` - The current document given by a URLConnection.
  Returns:
    The ContentType of the current document as String.

### main

    public static void main(java.lang.String[] args)

Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD