I have not tried the preempt kernel. I have tried the lowlatency kernel and found that it improved overall performance, not just latency.
Turns out that the cost of NUMA balancing (https://www.kernel.org/doc/Documentation/sysctl/kernel.txt) was outweighing any benefit we might get from it. The problem is that NUMA balancing is implemented by periodically unmapping pages, triggering a page fault if the page is accessed later. This lets the kernel move the page to the NUMA node that is accessing the memory. The page fault is not free, however; it has a measurable cost, and happens even for processes that have never migrated to another core. The lowlatency kernel turns off NUMA balancing by default.
Instead of switching to the lowlatency kernel, we set `kernel.numa_balancing = 0` in `sysctl.conf` and accepted the consequences.
Turns out that the cost of NUMA balancing (https://www.kernel.org/doc/Documentation/sysctl/kernel.txt) was outweighing any benefit we might get from it. The problem is that NUMA balancing is implemented by periodically unmapping pages, triggering a page fault if the page is accessed later. This lets the kernel move the page to the NUMA node that is accessing the memory. The page fault is not free, however; it has a measurable cost, and happens even for processes that have never migrated to another core. The lowlatency kernel turns off NUMA balancing by default.
Instead of switching to the lowlatency kernel, we set `kernel.numa_balancing = 0` in `sysctl.conf` and accepted the consequences.