Feed updates
  • benalves February 2011

    Hi there,

    I have two questions, both probably have to do with each other.

    1. How often do the feeds refresh? I think I have seen hourly and once a day
    2. I created two new feeds today. One had articles in it right away, the other didn't. Could this be because of a refresh timing issue?

    Thanks,
    Ben

  • Mike February 2011

    1. You're not crazy, Ben. We have an adaptive crawling schedule. That means we crawl sites that change frequently more than sites that change rarely. However, we always try to at least crawl once a day.

    2. It may be due to the fact that a URL that you entered was already registered in our system (by someone else previously). In that case, you'll be able to see a history of updates right away. If you don't want that, you can always just look at the timestamp in each item.

  • benalves February 2011

    Thanks for your responses. Both make since, but could you elaborate more on the first? I have some feeds that update almost daily. Will the crawler schedule figure this out fast and update hourly at least, or will it take time to learn the schedule?

  • Mike February 2011

    The crawl scheduler is designed so that it initially crawls at the highest frequency. Then, if it finds that it often sees no changes, then it eventually learns to slow down the crawl to match the frequency that the site changes. However, there is a lower cap at once per day. If this isn't the behavior that you are seeing, let us know and we can look into it.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership

In this Discussion

Tagged