Just a quick follow-up on the whole Mediabot thing……
Many people seem to be jumping to the conclusion that adding AdSense code might be a new submission tool that will help a site get indexed faster. I think most webmasters would love it if that were the case, but I haven’t seen anything in all the examples I’ve looked at that would suggest that there is any truth to it .
All of the pages from this site that are in Google’s index cached in our Mediabot template originally appeared in the Google index cached in our Googlebot template. In each case, new pages that were created (and crawled by Mediabot within hours of being published) were not included in the Google index until after a visit by the regular Googlebot. In fact, we still have many pages indexed in the Googlebot template, even though they have been visited by the Mediabot after they were originally indexed.
I guess it’s possible that the initial collection of new urls found by Mediabot might be dumped into the Googlebot hopper, which in turn could potentially lead to Googlebot showing up sooner than it normally would, but I haven’t seen anything that would suggest that is happening.
The only thing I see happening is that content that already exists in Google’s database is occasionally being refreshed by Mediabot. And I don’t really have a problem with that happening. It makes sense from an efficiency standpoint.
What I do have a problem with is the fact Google didn’t think it was an important enough change to warrant any kind of public disclosure before it was implemented. Several months of BigDaddy discussions and not a single comment on the fact they were planning on making dramatic changes to they way they collect data.
The whole thing reminds me of when they decided to start crawling secure servers without telling anyone. That little fiasco caused all kinds of problems because no one bothered to exclude secure content because Google stated in their official documentation that they wouldn’t crawl it.
I would think that four years would be more than enough time to come up with a little better strategy when it comes to things like this. But I guess that’s not the case.
Anyone want to place some bets on how long it will take for Google to edit the AdSense FAQ ?
Comments
4 Responses to “ AdSense Bot Part 2 ”
Got something to say?
Same thing’s happening here Greg
One site, a blog, has adsense in some posts and not others.
Other sites have adsense on all pages. Others no adsense at all.
There is no difference in inclusion time
That said, it used to be 2-3 days, much longer now
Greg, your findings match very well my own on this. As an example, just take my own (Danish) blog at http://www.demib.dk - I had the domain for years but never put anything on it. In the beginning of March I launched my blog (with AdSense) and the mediabot is crawling every page just fine, but the regular bot has not been around much (just a few hits) and the result is still that not a single page has been indexed (yet).
Off course, it could be that all “demib” is just banned as default hehehe … nahh, waite, demib.com is still in, so I guess it’s just a matter of time - or it’s just a special demib-snadbox LOL
Anyway, my demib.dk blog should really be indexed - its as white as any site gets
Mikkel
I too started a wordpress blog about a month ago.
Brand new domain.
Pages are getting into the index and ranking well.
Some pages are dropping in and out depending on the DC though
One thing I don’t like though is that the feed usually ranks first for the search term.
I also think there could be duplicate content problems in the early days of a blog which could lead to missing pages.
If you create a widget category, then do a widget post, you end up with the same content in the post page, the category page, the feed the monthly archive and the index page of the blog itself.
Obviously as new posts go up the category and archive etc are no longer a problem.
Indexing is very poor at the moment. I have one very reliable and consistent site I believe G considers an ‘authority’, new pages used to be showing in the serps within 2-3 days of page being created. Can be a 1 or even 2 now
Can be a 1 or even 2 now
1 or 2 weeks that is