Sunday January 08, 2006

Props from Jeremy on our anti-blog spam efforts are certainly appreciated. I know we don't have a spam-free index, however the amount of spam we keep out of the index is truly astonishing. Our ping interface is deluged with a torrent of rubbish but we do our best to scrub the nasty stuff out of our update stream. The problem defies conventional mail spam or even blog comment spam analytic techniques as the structure of blog spam is very different. Deep examination of the content and structure across a pattern of web sites is often required to distinguish it as spam but in the end, the indicators are there. Most spammers' publishing behaviors are statistical outliers by nature; the numbers speak for themselves.

We have a lot to do, on this and on many fronts but we try to pay attention to the gripes as a measure of priorities. The kudos are nice, too!

zawodny technorati spam

The levers and dials of character set encoding can be overwhelming, just looking at the matrix supported by J2SE 1.4.2 gives me vertigo. Java's encoding conversion support is simple enough, if not garrulous:

String iso88591String = request.getParameter("q");
String utf8String = new String(iso88591String.getBytes("UTF-8"));

• Rome has implemented character set encoding detection into it's XmlReader class
• There's a SourceForge Java port of the Mozilla encoding detector
• The J2SE kit comes with an interesting encoding detector and converter that works with common Japanese encodings. If it's one of Shift JIS, EUC-JP, or ISO 2022 JP but you're not sure which, use JISAutoDetect and you'll get a unicode string back:

  String q_unknown_japanese = request.getParameter(q);
  String q_unicode  = new String(q_unknown.getBytes("ISO8859_1"),"JISAutoDetect");
  

• Did I miss something? That seems to be the story so far but if there are more (and better) options available, I'd sure like to know about it.

java mozilla character sets