multi-ev_loop on UDP socket

Brandon Black blblack at
Mon Jun 4 19:36:03 CEST 2012

On Mon, Jun 4, 2012 at 8:40 AM, Marc Lehmann <schmorp at> wrote:
> On Mon, Jun 04, 2012 at 02:44:19PM +0200, Joachim Nilsson <troglobit at> wrote:
>> On Mon, Jun 4, 2012 at 2:29 PM, Marc Lehmann <schmorp at> wrote:
>> > On Mon, Jun 04, 2012 at 08:41:38AM +0800, 钱晓明 <mailtoantares at> wrote:
>> >> If there are more than one event loop on same udp socket, and each of
>> >> them in different thread, datagrams will be processed by all threads
>> >> concurrently? For example, one thread is processing udp message while
> If the processing of the udp packet takes comparatively long, then you will
> have few threads selecting on the fd (in the best case only one on average),
> because the others are busy processing something else.

In the specific case of connectionless UDP packets, if the processing
overhead to handle each incoming packet is very optimized and fast, on
Linux hosts, I've found that the Linux socket code itself becomes the
scaling limit.  One of my projects is an open source authoritative DNS
server where I did a lot of perf testing on this scenario.  Spawning
multiple threads looping on a single socket doesn't help, as they're
all waiting on some lower-level socket serialization in the kernel, so
you get basically the same throughput with 8 threads as you do with 1,
even if you have a bunch of CPU cores to use.

But you can scale up to more packets/sec handled on a given machine by
doing one thread-per-socket (-per-core) and distributing the packets
to these multiple sockets by some other mechanism.  e.g. in the DNS
case with an 8-core server, you might use some front-end hardware
loadbalancer (or some linux iptables/ipvsadm type hacks) to
re-distribute the incoming UDP packets on port 53 to local ports
1060-1067, run a thread+loop per socket on each of your 8 cores
listening on those ports, and of course have the LB stuff remap the
responses to source port 53 afterwards as well.  Then you can see real
scaling over anything you might try to do against a single UDP socket.

This all depends, as Marc was saying, on how much per-packet
processing overhead you have in your daemon.  If the overhead was
higher then yes you might be better off with several threads/loops on
a single socket.  I'd say as a very very rough rule of thumb: if your
processing code is light and optimal enough that it could handle
somewhere in the ballpark of say 20-50K (or higher) pps on a single
CPU core on the target machine, you're probably in the territory where
the socket is the limit and it's time to do thread-per-socket as
described above.

-- Brandon

More information about the libev mailing list