3e8.org

Benchmarking lighttpd on OS X with httperf, I found that persistent connections that were fetching files of certain sizes (some big, some small) had their first request satisfied in under 1ms, but experienced a 40ms delay on subsequent requests. Apache did not have this problem. Only one guy on the entire Internet reported seeing something like this, and he got (surprise) no response.

Studying ktrace/strace and tcpdumps from both sides indicated it might be a Nagle problem, and since httperf does disable Nagling, that left lighttpd. Indeed, I found that lighttpd does not (as of 1.4.18) set the TCP_NODELAY flag on its sockets. I guess it assumes the OS disables the Nagle optimization globally, which is generally true on Linux, but not OS X. Or evidently Solaris, since Sun said they patched lighttpd themselves.

I rebuilt the lighttpd from MacPorts and added a small patch:

port uninstall lighttpd
port -d extract lighttpd    # -d to see extract path
(patch network.c with TCP_NODELAY option)
port install lighttpd 

and it solved the issue.