Changing Java User-Agent Name

[David McKee]

If you "Data Mine" using tools such as Web-Harvester or others that are Java Based, it is difficult to get in the front door of many sites because your user-agent string says something like this:

Java/1.6.0-rc

This is immediately detected and you are refused. Unfortuantly if you attempt to change this via the URLConnection objects properties, it does not work properly.

I have found that following, using Apache's HTTPClient class to do the trick:

import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.BasicHttpParams;

public class getter {
public final static void main(String[] args) throws Exception {

DefaultHttpClient httpclient = new DefaultHttpClient();
BasicHttpParams hp = new BasicHttpParams();
hp.setParameter("http.useragent", (Object)"Mozilla/1.0 (compatible; linux 2015 plus; yep)"); // you oughta change this into your own UA string....
httpclient.setParams(hp);

try {
HttpGet httpget = new HttpGet("http://zminer.com/");
ResponseHandler responseHandler = new BasicResponseHandler();
String responseBody = httpclient.execute(httpget, responseHandler);
System.out.println(responseBody);
} finally {
httpclient.getConnectionManager().shutdown();
}
}
}

You will find that this can set the user-agent name.

I am interested in knowing if anyone else has run into this issue, or has solved it in a different way...

Thanks.

-DTM

 

[campers]

yep, that's what you have to do. If you're using the underlying HTTPUrlConnection then the code would be

URL url = new URL(urlString);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("User-Agent", "Mozilla/4.0(compatible; MSIE 5.0; Windows XP; DigExt)");

 

[David McKee]

yep, that's what you have to do. If you're using the underlying HTTPUrlConnection then the code would be

URL url = new URL(urlString);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestProperty("User-Agent", "Mozilla/4.0(compatible; MSIE 5.0; Windows XP; DigExt)");

This may work in 1.5 and above, however I have noticed that some of the java versions append a "Java 1.xxx" bit at the end. I hear somewhere that you can first set the string to "" and then set it again - not sure if that works or not.

If you use "BadBeahvior" plugin in wordpress and attempt to use Java to perform XMLRPC, it will block you unless you perform this agent renaming (or declare a white-list allowing java which could be dangerous unless you also add specific ip's)

Whitelisting BB is fine, until you forget to back up the whitelist and go an perform a plugin update! Doh!

So anyway...

-DTM