ut8 madness

[gleeco]

marc et al -
question is more in the world of common::sense and utf8, but i'm seeing
this in AE, so am seeking clarification/hints here on this list.

Does Perl tag a whole data structure non-utf8 if part of it is not?
What happens in complex data structures with some part that is not utf8?

tl;dr - 2 fetches of JSON; one has entities escaped in the output JSON as
\uNNN and the other ends up getting messed up.
here's the relevant pseudo code:
---start----
my %data;
# Anyevent::HTTP
http_get  http://foo.com/,  sub  {
   $data{bar}  = JSON::XS->new->decode( $_[0] );
}
http_get http://bar.com,  sub {
   $data{bar}  = JSON::XS->new->decode( $_[0] );
}
# later
print JSON::XS->new->encode( \%data );  # broken!

---end-pseudo--

What i'm seeing is foo.com has JSON with \uNNN inside of it, and
bar.comhas a bad case of mojibake when shared in %data
Curiously though, i print the raw body  --or the JSON::XS parse inside the
bar.com CB, and guess what? it's fine!

But after i put the data in to the container %data, bar.com json object
gets scrambled chars.

If i got the source input correct in both, then this problem seems to
disappear.

WTF?!?

thanks in advance!

-gleeco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schmorp.de/pipermail/anyevent/attachments/20120329/c0981824/attachment.html>