as the maintainer of this resource, for now, I've found no good answer to this question, hence this group project to make the catalogs people may want

completely agree. unfortunately dnsmasq, firewalls, running your own bind, or PF, table, or the other solutions you mention -- these all require a level of technical expertise far, far beyond the norm, or even 2 standard deviations above the norm of Internet users' technical ability (for some of them, including me). I had maintained the Facebook blocklist for hosts files because it was super simple for me to use, and to share with others.

definitely not ideal, not even complete, and requires work - BUT, nearly any Internet user can implement the solution done this way.

It would be really interesting to autogenerate the domain lists by running background scripts on AS numbers, polling DNS for every IP in the range, and cataloging the domains by script - say, daily, and then printing the list into a 0.0.0.0 prefixed hosts list. Thank you!

disclaimer: github user maintaining linked resource

Just keep in mind: People who end up at Stack Overflow are heavily biased compared to the general Internet population - because they are looking for something that lead them to search for and then click a link to Stack Overflow.

This same bias that selects the population will affect the choice of which search engine to use.

Nice theory, but the data doesn't support it.

We have other sites (cooking.stackexchange, diy.stackexchange) that are much more representative of the population at large. Even on those sites, you see a similar ratio... the second largest search referrer sends 1.6% as much traffic as Google (about twice as much as it sends to StackOverflow, but still virtually insignificant.)

How does the distribution of traffic to the homepages look? One thing search engines do is search for new information, answers to questions and the like. Another thing they do is spell-corrected dns.

Since stackexchange is mostly targeted at the former, I'd expect your numbers to be skewed in interesting ways, which you could partially correct by looking at homepage visits to capture the latter use. (This is a big confounding variable whenever people try to measure comparative statistics between search engines. I've heard that Yahoo and Bing get many more one-word searches than Google, which are likely to be of the spell-corrected dns kind.)

Perhaps the selection bias is some combination of "understands how the Internet works / not willing to accept paid results in searches / looking for real facts and data" that lead users to use both Google and Stack Exchange?

Clearly Google has an obvious and overt lead in all categories, and all search. I'm not sure how you can refute that the set of visitors to one site is not a biased sample compared to all Internet users.

Your results are different than other published search engine use comparisons. If both are factual, then the measurements must come from different populations.

"Your results are different than other published search engine use comparisons"

Link please?

The traffic for my site isn't as extreme as the stack exchange, but I'm seeing 9 times more traffic from google than bing. This leads to the same conclusion that Jeff came to. If Bing stopped sending me traffic tomorrow I wouldn't care. If google stopped sending me traffic I'd lose nearly all of my traffic.

Not to mention that while at work many of us are limited in our choice of browsers. Until recently I'd probably arrive at SO via Bing, since it's IE's default search provider and my employer actively blocked installation of Firefox & Chrome.

Now that they're not doing that anymore, I'll arrive by google most of the time.

From expriance (I work for a big publisher) the split between G Y and B looks about the same for the big name sites I work with. Google is even more dominant 90%+ in the Uk

I agree developers are more biased towards Google (or away from Bing) than the general population. But I used to run a real estate website which crosses a wider cross section of the population than stackoverflow and all the non-Google search engines did an exceedingly poor job of indexing and delivering traffic to that site as well.

> I agree developers are more biased towards Google (or away from Bing) than the general population.

Are they? I can easily imagine it being the other way around. I don't see how general population would know about other search engines or even consider that Google != Internet.

But of course, without hard data the whole topic is groundless.

I am basically thinking that Bing 'sponsors' many other pages to use it as the default search engine so they will stumble across it.