org.archive.crawler.datamodel
Class RobotsDirectives
java.lang.Object
org.archive.crawler.datamodel.RobotsDirectives
- All Implemented Interfaces:
- java.io.Serializable
public class RobotsDirectives
- extends java.lang.Object
- implements java.io.Serializable
Represents the directives that apply to a user-agent (or set of
user-agents)
- See Also:
- Serialized Form
| Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
disallows
PrefixSet disallows
allows
PrefixSet allows
crawlDelay
float crawlDelay
RobotsDirectives
public RobotsDirectives()
allows
public boolean allows(java.lang.String path)
addDisallow
public void addDisallow(java.lang.String path)
addAllow
public void addAllow(java.lang.String path)
setCrawlDelay
public void setCrawlDelay(float i)
getCrawlDelay
public float getCrawlDelay()
Copyright © 2003-2009 Internet Archive. All Rights Reserved.