|
Google appears to be redirecting all
search queries submitted from non-recognized broswers to a 403-Forbidden page. I noticed the problem
tonight when using the links
browser. Similar queries from the same IP with
Opera, IE, or lynx succeed. Search attempts by other links users
from other IPs resulted in the same message. Additional tests showed that fetch and wget also failed.
The 403-Forbidden message refers you to their terms and conditions.
I'm sure Google
are trying to block automated queries, which do not comply with their
T+C. I've written to them regarding this. I am also confident
this problem will be fixed soon and that they were not targeting
any particular tool directly.
The original page (with my IP address removed) is available at
google-links-forbidden-original.php.
You can get around the problem by changing the referrer field via Setup->Network Options->HTTP options->Fake User-Agent. I
used Opera/7.10 (UNIX; U) [en] and was able to get past their blockage.
For wget, you can do something like:
wget --user-agent="Opera/7.10 (UNIX; U) [en]" http://www.google.com/search?hl=en\&ie=ISO-8859-1\&q=things\&btnG=
Google+Search
And for fetch, you can set the HTTP_USER_AGENT environment variable:
export HTTP_USER_AGENT="Opera/7.10 (UNIX; U) [en]"
It's not hard to set user agents for most tools. This will break some things for a while, but hopefully most tools
can be adjusted.
|
