The FreeBSD Diary
Providing practical examples since 1998If you buy from Amazon USA, please support us by using this link.
Google blocking searches from unknown browsers 10 May 2003
Google appears to be redirecting all search queries submitted from non-recognized broswers to a 403-Forbidden page. I noticed the problem tonight when using the links browser. Similar queries from the same IP with Opera, IE, or lynx succeed. Search attempts by other links users from other IPs resulted in the same message. Additional tests showed that fetch and wget also failed.
The 403-Forbidden message refers you to their terms and conditions. I'm sure Google are trying to block automated queries, which do not comply with their T+C. I've written to them regarding this. I am also confident this problem will be fixed soon and that they were not targeting any particular tool directly.
The original page (with my IP address removed) is available at google-links-forbidden-original.php.
You can get around the problem by changing the referrer field via Setup->Network Options->HTTP options->Fake User-Agent. I used Opera/7.10 (UNIX; U) [en] and was able to get past their blockage.
For wget, you can do something like:
And for fetch, you can set the HTTP_USER_AGENT environment variable:
It's not hard to set user agents for most tools. This will break some things for a while, but hopefully most tools can be adjusted.
Fixed! 13 May 2003
|It's been fixed. For links at least. You no longer need a referrer entry.|