I'm having trouble with unicode versus octets

[Marc Lehmann]

On Mon, Oct 15, 2012 at 02:16:59PM -0400, Fulko Hew <fulko.hew at gmail.com> wrote:
> In my app, I receive strings, massage them, and use push_write() to enqueue
> them.

When you receive strings, and massage them (thereby treating them as
characters) you first need to decode your strings (e.g. with Encode::decode).

Decoding would give you a unicode string.

When you then want to write out these strings over e.g. a socket, you need to
encode them again, with e.g. Encode::encode.

You can encode/decode in many ways, but you need to do it, as all the I/O
functions only work with octets, not with characters.

> I want/need? to be able to treat the AnyEvent handle/data-path as an octet
> path

Yes indeed, sockets or anything else you use with AnyEvent::Handle (or perl's
builtin I/O functions) need octets.

> and not unicode (because any unicode-iness is between the originator
> and the ultimate destination, and not my code, nor handle in between.)

Well, somehow your code does create unicode characters in the string,
and characters with values > 255 - if you only create unicode strings
with codes < 256 then AnyEvent::Handle will (wrongly) treat your strings
as octet strings as it cannot detect this (perl strings do not have an
inherent encoding - the code decides which encoding a string is in).

> a) coerce the character string into an octet string
>    (for the rest of its life, ie. in subsequent modules)
>    so the warning/dying goes away.

Your character string cnanot be coerced into an octet string ever, as it
contaisn character values >255, which cannot, ever, be coerced into an
octet string, except by e.g. throwing those values away, which is likely
not what you want.

> b) add something (???) so that it go through that code.

You need to encode your strings into some encoding, for example utf-8. If
you need to use another ecnoding, and you really wnat to destroy the
characters it cannot trepresent, then you sitll need to encode it -
Encode::encode for example can do this for you.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\