Anything wrong with using AnyEvent::Handle for files?

Daniel Carosone daniel.carosone at gmail.com
Sat Aug 8 09:53:45 CEST 2015


I've recently implemented something to watch a logfile directory tree with
Linux::Inotify2 and basically 'tail -f' all the files in there, feeding me
a line at a time as they grow, opening and closing files as needed.  It's a
quick hack, using blocking IO for the file operations to start with. I was
trying to think of a good interface model to hide all the buffering needed
for making it use aio_, and I think extending Handle will work quite well.

For my usage, I can present a single Handle on the whole directory tree,
that does the inotify and aio internally.

Thanks for tickling the neurons needed! :)

On 8 August 2015 at 13:42, Marc Lehmann <schmorp at schmorp.de> wrote:

> On Fri, Aug 07, 2015 at 08:19:13AM -0700, Mike Schilli <
> office at perlmeister.com> wrote:
> > Although the documentation seems to discourage using AnyEvent::Handle
> > for anything but streaming things like sockets and pipes, using it for
> > files seems to work fine in the code below.
> > Any reason not to use it in this way?
>
> Daniel summarised the technical aspects quite nicely in his reply - I'll
> just add some philosophical thoughts and a few more obscure technical
> issues to it.
>
> First of all, if you use it, and it works for you, and you won't come to me
> to complain if it breaks, feel free to use it on files.
>
> Unlike for packet sockets, where AnyEvent::Handle refuses to work because
> it
> can't work properly with them, AnyEvent::Handle does best effort for files,
> for reasons explained later.
>
> However, there are limitations. For example, AnyEvent::Handle doesn't care
> for append mode or other file flags - it treats its handles as streaming
> files, so only "normal" file handles have good chances of working
> reliably. Moreso, AnyEvent::Handle relies on AnyEvent, and AnyEvent
> doesn't support files in its I/O watchers (it's indeed documented).
>
> The reason for this is that, although all sane event mechanisms (poll,
> select...) do the right thing with nonblocking file handles, as usual,
> epoll literally fails with them. This failure case isn't well documented,
> so most implementations get it wrong and fail in various ways. For
> example, tmux (based on libevent) ran into this problem, as libevent
> doesn't handle files when it is using epoll as it's backend.
>
> EV (based on libev) does handle files correctly, and most other event
> loops still use selector poll nowadays. For those backends who do use
> epoll, I would assume they won't handle files until I have verified that
> they do.
>
> Now to something different: AnyEvent::Handle is made to handle event-based
> streaming, which means pipes and sockets usually, and often makes little
> sense for files indeed. There are, however, exceptions to this:
>
> First, it should be possible to use AnyEvent::Handle on STDIN/STDOUT, and
> have it work both for pipes and files. While it might make sense to use
> normal perl I/O for files, it makes little sense to write two versions of
> your parser, one with AnyEvent::Handle, one with native perl I/O.
>
> Second, AnyEvent::Handle *can* be nicer to parse files than the native perl
> I/O. For example, reading concatenated old JSON texts from a file is
> probably
> easier with AnyEvent::Handle, and it's hard to see why one should be forced
> to resort to another technique when one already knows how to do it with
> AnyEvent.
>
> So, I *do* think that AnyEvent::Handle should support files, but it's not
> as
> easy as changing the documentation. Basically, there are multiple
> alternatives:
>
> 1. make AnyEvent::Handle detect this case and use e.g. AnyEvent::IO. This
>    is not trivial (AnyEvent::IO is ghastly low level :), and of course
>    limited to AnyEvent::Handle, but would have the benefit of actually not
>    blocking on slow disk or network I/O.
>
> 2. guarantee that files work with AnyEvent's I/O watchers and make
>    them work. this would be beneficial for people who do their own
>    AnyEvent::Handle style stuff, but would be "blocking" on I/O. It might
>    incur a rather large performance overhead for some backends though, and
>    AnyEvent doesn't control all backends, and was written basically to
>    support the common features of existing loops.
>
> 3. convince existing backend authors to support files in their I/O
> watchers.
>    AnyEvent->io will then just work, as well as AnyEvent::Handle, at least
>    for normal files, regardless of the documentation.
>
> Of course, I'd preferably go for 3), and maybe later 1)...
>
> Right now, however, neither of these are sufficiently high on my TODO list
> to lift this limitation soon.
>
> ... as a module author, you *could* just use AnyEvent::Handle and blame
> any backends that don't get it right, of course.
>
> --
>                 The choice of a       Deliantra, the free code+content
> MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
>       -=====/_/_//_/\_,_/ /_/\_\
>
> _______________________________________________
> anyevent mailing list
> anyevent at lists.schmorp.de
> http://lists.schmorp.de/mailman/listinfo/anyevent
>



-- 
Dan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schmorp.de/pipermail/anyevent/attachments/20150808/5c646172/attachment.html>


More information about the anyevent mailing list