Issue with AnyEvent::AIO

[Dominik Csapak]

On 6/30/23 05:08, Marc Lehmann wrote:
> On Thu, Jun 29, 2023 at 03:24:54PM +0200, Dominik Csapak <d.csapak at proxmox.com> wrote:
>> We have an issue with AnyEvent::AIO, basically we don't use it directly, but as
>> soon as it's installed on a machine, our AnyEvent based daemon (a http-sever basically)
>> uses 100% cpu time on one core (per forked subprocess).
> 
> Hi!

Hi,

thanks for your prompt answer!

> 
> IF the event loop is initialised before fork (which happens during
> AnyEvent->condvar in your example, and generally in the first call), you
> must not use it after fork. In general, after fork, you need to either
> call exec or _exit (not exit!), without doing much in between.
> 
> See the FORK section in the AnyEvent manpage.
> 
> The specific reason why AnyEvent::AIO seems to make a difference is
> because it also registers an I/O watcher during AnyEvent initialisation,
> not because it does anything special.
> 
> As a demo, if you replace the "return 1" in fork_child by POSIX::_exit 1,
> the program finishes, as you don't call into AnyEvent in the child.
> 
> While some event loops (such as EV) can be made to work after fork
> even when initialized, this generally requires careful control, as you
> typically have to be certain about what watchers you want to be active
> still - it's generally not worth it to try, and in any case, very fragile.
> 
> And this is genrally no help for AnyEvent, as it has to work with other
> event loops as well.
> 
> If you need an event loop in the child, you can try to have a look at
> AnyEvent::Fork, which creates subprocesses without forking your main
> proccess.
> 
> This will also help with lots of other problems, such as graphical
> toolkits, threads and so on, which all don't support fork except in a very
> limited way.
> 

Thanks for the explanation of that, but i looked over our codebase again,
and putting the condvar before the fork was my mistake, in the actual code,
we only use it after a fork.

When doing the fork correctly and only setting up the condvar after that
in my previously posted exmpale, it works as intended
(no 100% cpu usage from those processes)

It still happens in our production code, so i guess we inadvertently trigger
some AnyEvent initialization, though i couldn't find it (yet).
Is there any way to detect that during runtime? (so i could go and try to
detect the parts of the code where that might happen.)

basically what we do is

* setup some basic things (dirs, logging, reading config, socket, etc.)
* fork our workers
  - parent: handles sigchld, reload, etc.
  - children: set up their anyevent loop with the given socket, config, etc.

Since we only use AnyEvent in the children, i am not so sure what to look for.

when calling strace on such a process i see a constant stream of

epoll_wait(4, [{events=EPOLLIN, data={u32=8, u64=4294967304}}], 64, 3346) = 1

where the timeout (3346 in this example) begins at ~5000 and goes down.
then there is a

read(3, 0x7ffd1b8e0760, 8192)           = -1 EAGAIN (Resource temporarily unavailable)

and it starts again with the epoll_wait

(sorry if that is not helpful, as i said not really that deep into anyevent)

Is there any other information i can provide to help (some log output, strace, etc.?)
I would be very grateful if you could give any hint as to how i can
debug this

Kind Regards
Dominik