Some questions about libev

Marc Lehmann schmorp at schmorp.de
Fri May 22 16:02:03 CEST 2015


On Fri, May 22, 2015 at 01:26:45PM +0000, Dejan Markic <Dejan.Markic at mobik-ics.com> wrote:
> Once the thread starts, it create hiredis async handler and attaches libev with its libev handler and sets up all the callbacks.
> After all is initialized and connected, I send commands from main thread to all servers via redisAsyncCommand.

I don't know the details of that adapter library, so I can only comment
generally:
> 
> Questions:
> 
> 1. Hiredis libev adapter uses two ev_io watchers, one for reading and one for writing. It's simply calling start and stop on those watchers based on what it tries to do.
>     This does not work for me, unless I register another ev_async watcher, that fires everytime hiredis tries to start/stop the io_watcher. Is that expected? I would say that once
>     libev goes into poll/select/etc wait, it will not move out of wait if you stop the watcher, or start the stopped watcher, right?

The question is, how do you call ev_io_start or stop when the thread is
blocked? The only way to do that is from another thread, which requires
locking, and from your description, you don't do any locking.

In general, if you use multiple threads, you have to use some kind
of locking (or another suitable synchronisation mechanism) on shared
data, you cannot just access data structures from other threads without
synchronising with them. If you do, then watchers not firing is the
leats of your problems, because its easy to corrupt data structures when
different threads have different ciews of them.

There are many ways to achieve that, either you use an async watcher to
wake up the thread that uses the ev loop (and in its callback, you can
then start/stop other watchers), or you lock the event loop (see THREAD
LOCKING EXAMPLE in the libev docs), or a combinatzion of both.

> 2. I tried another approach I saw on the internet. Single ev_io
> watcher, but everytime hiredis tries to stop/start read/write, I to
> ev_io_stop(...) and then I set EV_READ and/or EV_WRITE flags via
> ev_io_set and then ev_io_start again. Also in that case, I need ev_async
> watcher to wakeup the libev. Would you that that single IO watcher
> approach would be better?

As long as you use ev_io_start/stop in the same thread aqs your other
libev calls, thats fine, of course also without ev_async. As soon as you
use another thread to manipulate the same libev loop, yoiu have to lock it
or otherwise synchronise.

> Both 1. and 2. I tried protecting changes in watchers with mutex, but no success. 

How did you protect changes in the watchers? In general, you have to lock
every call, including ev_run, so inside a watcher your loop will likely
still be locked.

> 3. My test program hangs after few minutes, maybe even few seconds (it's working for atleast 1000 iterations, even 100000 sometimes) after I fire it (sending SET/GET to one or many servers). It hangs in epoll_wait or select (tried different EVBACKEND_*). It doesn't receive the ev_async event it seems. I also have a feeling that there might be some memory corruptions, but I cannot find any using efence or valgrind.

Modern cpus are very asynchronous - hoping that memory somehow
synchronises itself between different cores is likely going to result in
unexpected behaviour like you describe.

> I've enabled EV_VERIFY 3 and sometimes I get "libev: pending watcher not on pending queue" when on EVBACKEND_SELECT.

Same thing, if you don't lock shared data structures, corruption is almost
guaranteed.

> Any hints before I go and completely rewrite it?

I don't know your exact design, so I don't know what you could/should
redesign, if anything, but in general, if you use threads, you need to be
extreemly careful and not forget locking.

The documentation gives a complete example in the THREAD LOCKING EXAMPLE,
which likely doesn't exactly match your use case, but it shows how to
relinquish a mutex while libev polls for new events, which allows you to
modify libevs watchers from other threads. These changes will only take
effect on the next event loop iteration, so additionally, you need to have an
async watcher to take up your thread from another thread.

Depending on your design, it might be simpler to only have an ev_async
watcher and do the required modifications in its callback. In the latter
case, you might not need extra locking, as you are allowed to call
ev_async_send from other threads without locking (you might need a mutex to
protect any variables you want to pass to that async watcher of course).

In general, with threads, do never assume that you can share data
structures between threads without synchronisation or locking.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp at schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\



More information about the libev mailing list