org.archive.crawler.datamodel
Class RobotsDirectives

java.lang.Object
  extended by org.archive.crawler.datamodel.RobotsDirectives
All Implemented Interfaces:
java.io.Serializable

public class RobotsDirectives
extends java.lang.Object
implements java.io.Serializable

Represents the directives that apply to a user-agent (or set of user-agents)

See Also:
Serialized Form

Field Summary
(package private)  PrefixSet allows
           
(package private)  float crawlDelay
           
(package private)  PrefixSet disallows
           
 
Constructor Summary
RobotsDirectives()
           
 
Method Summary
 void addAllow(java.lang.String path)
           
 void addDisallow(java.lang.String path)
           
 boolean allows(java.lang.String path)
           
 float getCrawlDelay()
           
 void setCrawlDelay(float i)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

disallows

PrefixSet disallows

allows

PrefixSet allows

crawlDelay

float crawlDelay
Constructor Detail

RobotsDirectives

public RobotsDirectives()
Method Detail

allows

public boolean allows(java.lang.String path)

addDisallow

public void addDisallow(java.lang.String path)

addAllow

public void addAllow(java.lang.String path)

setCrawlDelay

public void setCrawlDelay(float i)

getCrawlDelay

public float getCrawlDelay()


Copyright © 2003-2009 Internet Archive. All Rights Reserved.