|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectjavax.management.Attribute
org.archive.crawler.settings.Type
org.archive.crawler.settings.ComplexType
org.archive.crawler.settings.ModuleType
org.archive.crawler.framework.Processor
org.archive.crawler.prefetch.QuotaEnforcer
public class QuotaEnforcer
A simple quota enforcer. If the host, server, or frontier group associated with the current CrawlURI is already over its quotas, blocks the current URI's processing with S_BLOCKED_BY_QUOTA.
| Nested Class Summary |
|---|
| Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType |
|---|
ComplexType.MBeanAttributeInfoIterator |
| Field Summary | |
|---|---|
protected static java.lang.String |
ATTR_FORCE_RETIRE
whether to force-retire when over-quote detected |
protected static java.lang.String |
ATTR_GROUP_MAX_ALL_KB
group max all fetch bytes (including error responses) |
protected static java.lang.String |
ATTR_GROUP_MAX_FETCH_RESPONSES
group max fetch responses (including error codes) |
protected static java.lang.String |
ATTR_GROUP_MAX_FETCH_SUCCESSES
group max successful fetches |
protected static java.lang.String |
ATTR_GROUP_MAX_SUCCESS_KB
group max successful fetch bytes |
protected static java.lang.String |
ATTR_HOST_MAX_ALL_KB
host max all fetch bytes (including error responses) |
protected static java.lang.String |
ATTR_HOST_MAX_FETCH_RESPONSES
host max fetch responses (including error codes) |
protected static java.lang.String |
ATTR_HOST_MAX_FETCH_SUCCESSES
host max successful fetches |
protected static java.lang.String |
ATTR_HOST_MAX_SUCCESS_KB
host max successful fetch bytes |
protected static java.lang.String |
ATTR_SERVER_MAX_ALL_KB
server max all fetch bytes (including error responses) |
protected static java.lang.String |
ATTR_SERVER_MAX_FETCH_RESPONSES
server max fetch responses (including error codes) |
protected static java.lang.String |
ATTR_SERVER_MAX_FETCH_SUCCESSES
server max successful fetches |
protected static java.lang.String |
ATTR_SERVER_MAX_SUCCESS_KB
server max successful fetch bytes |
protected static java.lang.Boolean |
DEFAULT_FORCE_RETIRE
|
protected static java.lang.Long |
DEFAULT_GROUP_MAX_ALL_KB
|
protected static java.lang.Long |
DEFAULT_GROUP_MAX_FETCH_RESPONSES
|
protected static java.lang.Long |
DEFAULT_GROUP_MAX_FETCH_SUCCESSES
|
protected static java.lang.Long |
DEFAULT_GROUP_MAX_SUCCESS_KB
|
protected static java.lang.Long |
DEFAULT_HOST_MAX_ALL_KB
|
protected static java.lang.Long |
DEFAULT_HOST_MAX_FETCH_RESPONSES
|
protected static java.lang.Long |
DEFAULT_HOST_MAX_FETCH_SUCCESSES
|
protected static java.lang.Long |
DEFAULT_HOST_MAX_SUCCESS_KB
|
protected static java.lang.Long |
DEFAULT_SERVER_MAX_ALL_KB
|
protected static java.lang.Long |
DEFAULT_SERVER_MAX_FETCH_RESPONSES
|
protected static java.lang.Long |
DEFAULT_SERVER_MAX_FETCH_SUCCESSES
|
protected static java.lang.Long |
DEFAULT_SERVER_MAX_SUCCESS_KB
|
protected static int |
GROUP
|
protected static int |
HOST
|
protected static java.lang.String[][] |
keys
|
protected static int |
NAME
|
protected static int |
RESPONSE_KB
|
protected static int |
RESPONSES
|
protected static int |
SERVER
|
protected static int |
SUCCESS_KB
|
protected static int |
SUCCESSES
|
| Fields inherited from class org.archive.crawler.framework.Processor |
|---|
ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules |
| Fields inherited from class org.archive.crawler.settings.ComplexType |
|---|
definition, definitionMap |
| Constructor Summary | |
|---|---|
QuotaEnforcer(java.lang.String name)
Constructor. |
|
| Method Summary | |
|---|---|
protected boolean |
applyQuota(CrawlURI curi,
java.lang.String quotaKey,
long actual)
Apply the quota specified by the given key against the actual value provided. |
protected boolean |
checkQuotas(CrawlURI curi,
CrawlSubstats.HasCrawlSubstats hasStats,
int CAT)
Check all quotas for the given substats and category (server, host, or group). |
protected void |
innerProcess(CrawlURI curi)
Classes subclassing this one should override this method to perform their custom actions on the CrawlURI. |
| Methods inherited from class org.archive.crawler.framework.Processor |
|---|
checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, initialTasks, innerRejectProcess, isContentToProcess, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn |
| Methods inherited from class org.archive.crawler.settings.ModuleType |
|---|
addElement, listUsedFiles |
| Methods inherited from class org.archive.crawler.settings.Type |
|---|
addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient |
| Methods inherited from class javax.management.Attribute |
|---|
getName |
| Methods inherited from class java.lang.Object |
|---|
clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
| Field Detail |
|---|
protected static final int SERVER
protected static final int HOST
protected static final int GROUP
protected static final int NAME
protected static final int SUCCESSES
protected static final int SUCCESS_KB
protected static final int RESPONSES
protected static final int RESPONSE_KB
protected static final java.lang.String[][] keys
protected static final java.lang.String ATTR_SERVER_MAX_FETCH_SUCCESSES
protected static final java.lang.Long DEFAULT_SERVER_MAX_FETCH_SUCCESSES
protected static final java.lang.String ATTR_SERVER_MAX_SUCCESS_KB
protected static final java.lang.Long DEFAULT_SERVER_MAX_SUCCESS_KB
protected static final java.lang.String ATTR_SERVER_MAX_FETCH_RESPONSES
protected static final java.lang.Long DEFAULT_SERVER_MAX_FETCH_RESPONSES
protected static final java.lang.String ATTR_SERVER_MAX_ALL_KB
protected static final java.lang.Long DEFAULT_SERVER_MAX_ALL_KB
protected static final java.lang.String ATTR_HOST_MAX_FETCH_SUCCESSES
protected static final java.lang.Long DEFAULT_HOST_MAX_FETCH_SUCCESSES
protected static final java.lang.String ATTR_HOST_MAX_SUCCESS_KB
protected static final java.lang.Long DEFAULT_HOST_MAX_SUCCESS_KB
protected static final java.lang.String ATTR_HOST_MAX_FETCH_RESPONSES
protected static final java.lang.Long DEFAULT_HOST_MAX_FETCH_RESPONSES
protected static final java.lang.String ATTR_HOST_MAX_ALL_KB
protected static final java.lang.Long DEFAULT_HOST_MAX_ALL_KB
protected static final java.lang.String ATTR_GROUP_MAX_FETCH_SUCCESSES
protected static final java.lang.Long DEFAULT_GROUP_MAX_FETCH_SUCCESSES
protected static final java.lang.String ATTR_GROUP_MAX_SUCCESS_KB
protected static final java.lang.Long DEFAULT_GROUP_MAX_SUCCESS_KB
protected static final java.lang.String ATTR_GROUP_MAX_FETCH_RESPONSES
protected static final java.lang.Long DEFAULT_GROUP_MAX_FETCH_RESPONSES
protected static final java.lang.String ATTR_GROUP_MAX_ALL_KB
protected static final java.lang.Long DEFAULT_GROUP_MAX_ALL_KB
protected static final java.lang.String ATTR_FORCE_RETIRE
protected static final java.lang.Boolean DEFAULT_FORCE_RETIRE
| Constructor Detail |
|---|
public QuotaEnforcer(java.lang.String name)
name - Name of this processor.| Method Detail |
|---|
protected void innerProcess(CrawlURI curi)
Processor
innerProcess in class Processorcuri - The CrawlURI being processed.
protected boolean checkQuotas(CrawlURI curi,
CrawlSubstats.HasCrawlSubstats hasStats,
int CAT)
curi - CrawlURI to mark up with resultshasStats - holds CrawlSubstats with actual values to testCAT - category index (SERVER, HOST, GROUP) to quota settings keys
protected boolean applyQuota(CrawlURI curi,
java.lang.String quotaKey,
long actual)
curi - CrawlURI whose processing is subject to a potential quota
limitationquotaKey - settings key to get applicable quotaactual - current value to compare to quota
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||