Skip to content

The feed crawler

RSS and Atom feeds are fetched periodically by the feed crawler.


The crawler identifies itself with a User-Agent string that contains OohDirectory/1.0.

For example:

OohDirectory/1.0 (+ 1 subscribers)

However, if the crawler consistently gets a "403 Forbidden" response then it will start to use a different User-Agent in an attempt to emulate a Chrome browser as closely as possible.


The frequency of requests for a single feed varies depending on an estimate of how frequently new items appear in the feed. With each request, if there are no new items, then the frequency is reduced. Conversely, if there is a new item, the frequency is increased.

Currently a single feed is requested at most once per hour, and at a minimum once per 24 hours.