are new benchmarks needed?

[Charles Kerr]

On Tue, Dec 28, 2010 at 12:21 PM, Marc Lehmann <schmorp at schmorp.de> wrote:

> I don't see how it is inconsistent - there was always little practical
> difference in performance with those libraries, especially when applied
> to that benchmark. Neither the document nor me mention any practical
> differences.

You have frequently and prominently asserted that libev is faster than
libevent. It's on the website. It's in the README. It's a central
selling point!

1. Libev's README describes libev "is modelled (very losely) after
libevent and the Event perl module, but is faster, scales better and
is more correct, and also more featureful."

2. The 2008 document you referred to begins its summary with: "The
benchmark clearly shows that libev has much lower costs and is
consequently faster than libevent."

3. The 2008 document concludes with: "It is still much faster than
libevent even when using the libevent emulation API."

4. The original libev announcement was titled "announcing libev,
towards a faster and more featureful libevent." It was the first
adjective you ever used to describe libev with! :)

5. Your July 14 2010 mail to me ended with "Of course, there is no way
libevent can ever surpass libev (giving they both use similar
backends), as the libev API itself is faster (much less memory to move
around etc.)."

In each statement: faster, faster, faster, faster. In one case, even
"much faster."  You never said anything like "faster, but not enough
to make any practical difference." :)


> Interpreting benchmark results is notoriously difficult. To be honest,
> it seems to me as if you tried to use the benchmark results to prove
> something that the results don't show, and now you want to push this in
> some way, or make my words mean something that they don't really say.

I don't think I'm twisting your words. But I admit I'm having trouble
reconciling the five statements above with the other things you've
said:

6. Your statement in the "reloaded" thread: "Effectively, there is now
little (practical) performance difference between libev and libevent,
unless one has a really high number of fds to attend to."

7. Your statement just now: "There was always little practical
difference in performance with those libraries, especially when
applied to that benchmark."


> What both benchmarks show is that libev is faster even with emulation

I'm 100% positive you can interpret these benchmarks better than I
can, but to my eye they seem to show that libev is sometimes faster,
sometimes tied, and sometimes worse than libevent2. In cases where
libev is faster, libevent2 generally loses by smaller margins than
libevent1 did.

No Timeouts graphs
================
2008: http://libev.schmorp.de/dat.t0.png
2010: http://i.imgur.com/86CCE.png

* Total time per iteration: libev wins in 2008; libevent2 wins in 2010.
* Time spent processing 100 active clients: a tie in both 2008. an
overall tie in 2010, but libevent2 wins in the [1,000...10,000] file
descriptor range.
* Time spent processing 1000 active clients: in both 2008 and 2010,
libev handily beats all others :)

With per-connection idle timeouts graphs
================================
2008: http://libev.schmorp.de/dat.t1.png
2010: http://i.imgur.com/kLfsX.png (2010)

* Total time per iteration: libev wins in 2008; a tie in 2010.
* Time spent processing N active clients: libev wins by a wide margin
in 2008, and wins by a much smaller margin in 2010.


> Could you honestly admit what your actual agenda is?

Sure: I think that when libev compares itself to libevent, it should
do so with reasonably up-to-date information so that developers can
make better-informed decisions. That's not currently the case on
libev's website.

Over the summer I was researching libev with the intention of using it
instead of libevent in Transmission because users in IRC had passed me
the URL to libev's benchmarks. When reading up on libev I saw the
"reloaded" thread in the ML and couldn't understand why the the other
benchmarks were still so central to libev's website. I mailed you
about it, you said you it would be unfair to libevent2 to compare it
while still in alpha.  That seemed correct to me, so I filed the
question away until libevent2-stable came out. So here we are today.

I challenge anyone to look at the 2008 and 2010 graphs side-by-side --
try popping them up in tabs in a browser and flipping back and forth
-- and conclude that "there is little principal difference between the
old versions and the current versions of either libraries."  It's also
it's hard to understand how you could say "Neither the document nor me
mention any practical differences" when you frequently and prominently
do.

> I invested a lot of time into documenting facts as they are and have
> taken great care to avoid anything that could be misleading. There is
> nothing misleading in that document whatsoever - I even went as far as
> including version numbers, exact dates, compiler verison and flags and so
> on.

I agree with that and appreciate the attention to detail.

> Claiming that it is misleading is dishonest, and makes me wonder what
> your real intentions are.
>
> Please provide the parts of that document which are misleading, and
> explain why they are misleading.
>
> If you cannot, please take back your claim that the benchmark is
> misleading.

What I wrote: "If libevent has narrowed the gap as you've said, then
it's misleading to keep the older benchmarks on libev's webpage."

I don't dispute the 2008 benchmarks. I dispute its continued use in 2010.

cheers,
Charles