page still being crawled, after alert is turned off
  • I unselected an alert 2 days back and checked that the site had recorded the change by opening the diffbot alerts page again. The alert was unselected as intended, but the alerts haven't stopped at all.

  • eleith March 2010

    are you still getting email alerts? if so, can you forward us an email that you received from one of these alerts?


  • Ah... let me clarify the above. I'm having diffbot monitor a specific "page" which is actually a SCRIPT on my site. Whenever the script is called, it (MY SCRIPT) emails me a "status report".

    I have never received any "email alerts" from Diffbot because the page has NEVER changed - that's intentional. However, the SCRIPT is still being called by diffbot despite NOT being selected.

  • eleith March 2010

    thanks for the clarification.

    currently when an alerts users turns "off" a url, they are just unsubscribing from email alerts. diffbot still monitors the URL to build it's archive.

    if you want diffbot to stop monitoring the page entirely, for now you have to remove the alert all together, not just turn it "off".

    i'll look in changing this behavior in the future


  • Interestingly enough - it was STILL calling that page even after I deleted it... When I checked http://www.diffbot.com/inbox?start=alerts - my page was NOT mentioned on the RIGHT hand side, but was present on the LEFT hand list. I have no idea what's that for, but I've removed it from there as well and it may stop calling my page now! :)

  • eleith April 2010

    if a source is in either of those lists, it will be crawled. let me know if you notice it is still crawling your page, and we'll look into it.


  • Woah! Just checked this morning and YES! It's still crawling my page! How are you storing these lists?

  • eleith April 2010

    either another account is subscribed to this (maybe an older account of yours?) or crawling was improperly not turned off for the source. could you send me a private message (click on my user name), and let me know the exact URL you are subscribe to?

    we'll look into it, and get back to you with our results.

  • eleith April 2010

    turns out, we purge our crawlers of old urls infrequently, so urls will still be watched until we purge them. that would explain the delays you've experienced. i've turned off the crawling for your particular case, but i we'll look into the feasibility of accomplishing this on the fly, without any delays.

    thanks for bringing this to our attention!

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership

In this Discussion