Add ability to ignore bad UTF-8 encoded characters

[ser]

Holden Karau's original email:

I've been doing some work parsing alot of RSS feeds in the wild, and one thing which I've found useful is removing invalid UTF-8 characters.Thispatch is against rexml_3.1.7 and modifies encodings/ICONV.rb & encodings/UTF-8.rb . I've attached the patch, but in case it gets mangled you can grab it from my website at http://www.holdenkarau.com/~holden/projects/rexml/rexml_strip_invalid.diff. I'm not sure if this should go into REXML, but I think it would be useful for a lot of people.

Solution

Holden's solution is pretty close to what we want to do. Add an UTF-8/Ignore encoding. Users can then construct their own sources when they want to parse malformed XML documents:

s = Source.new( xml_file, "UTF-8/Ignore" )
d = Document.new(s)

[ser]

Holden's patch to modify REXML's default UTF-8 behavior