[cwasko]

Well, the subject pretty much says it all. I have some jank XML docs which have non-ascii chars in them. I just want to remove them, or translate them to their � equiv. The big thing here though is, I want to retain the file as an XML file, so I don't want to convert the standard ascii chars. I've found many Perl scripts which will do the whole thing, but I really don't know how to do it for non-ascii.

[OreoCookie]

Check out recode. It can convert one character set into pretty much any other. Documentation of recode

Download the source here.

Before you compile recode, use

./configure --disable-nls

instead of just ./configure. Otherwise it won't compile.

Hope that helps.

[cwasko]

Thanks for the tip. Do you actually have it working on OSX? I tried compiling and it first bombed and said it couldn't determine the host. So I copied in the /usr/share/libtool stuff. It then got most of the way through and bombed on another part. At this point, a binary would be nice (hint-hint) or even just a command line shell script. I started to play around last night with tr and jot, but haven't finished things up yet.

[cwasko]

Well, I got it to compile. I downloaded the development version. However, I can't figure out how to make it strip the stuff I don't want. It seems to bomb when it gets to the parts in the file that I want to strip. Uhg.

recode -sf ..us < file.orig.xml > file.xml

[cwasko]

perl -pe 's/[^\040-\176\011\012\015]//g' inputfile > outputfile

This will strip everything except normal ascii characters and white-space characters. The odd thing is that this works beautifully on OSX, but when I do this on Linux RedHat 8.0, it doesn't strip all the chars when run on the same file. Any ideas?