The FreeBSD Diary

The FreeBSD Diary (TM)

Providing practical examples since 1998

If you buy from Amazon USA, please support us by using this link.
[ HOME | TOPICS | INDEX | WEB RESOURCES | BOOKS | CONTRIBUTE | SEARCH | FEEDBACK | FAQ | FORUMS ]
Google blocking searches from unknown browsers 10 May 2003
Need more help on this topic? Click here
This article has 13 comments
Show me similar articles

Google appears to be redirecting all search queries submitted from non-recognized broswers to a 403-Forbidden page. I noticed the problem tonight when using the links browser. Similar queries from the same IP with Opera, IE, or lynx succeed. Search attempts by other links users from other IPs resulted in the same message. Additional tests showed that fetch and wget also failed.

The 403-Forbidden message refers you to their terms and conditions. I'm sure Google are trying to block automated queries, which do not comply with their T+C. I've written to them regarding this. I am also confident this problem will be fixed soon and that they were not targeting any particular tool directly.

The original page (with my IP address removed) is available at google-links-forbidden-original.php.

You can get around the problem by changing the referrer field via Setup->Network Options->HTTP options->Fake User-Agent. I used Opera/7.10 (UNIX; U) [en] and was able to get past their blockage.

For wget, you can do something like:

wget --user-agent="Opera/7.10 (UNIX; U) [en]" http://www.google.com/search?hl=en\&ie=ISO-8859-1\&q=things\&btnG= Google+Search

And for fetch, you can set the HTTP_USER_AGENT environment variable:

export HTTP_USER_AGENT="Opera/7.10 (UNIX; U) [en]"
It's not hard to set user agents for most tools. This will break some things for a while, but hopefully most tools can be adjusted.
Fixed! 13 May 2003
It's been fixed. For links at least. You no longer need a referrer entry.

Need more help on this topic? Click here
This article has 13 comments
Show me similar articles