If your server's running Limux, it's going to kill your process with no questions asked if you run out of memory. You're better off practicing crash-only error recovery and having robust clients that can handle a reconnect.

HTTP is stateless already, so crash and restart all you want!

The OOM killer is more likely to kill some other process and trash your server.

Thankfully that sort of behavior has been vastly reduced since the thing was introduced, but disabling overcommit for high-reliability applications is still a reasonable course of action.

The OOM killer might eventually kill something, after it thrashes the system for a few hours.

I had a server last week in which swap hadn't been configured. A compilation job took all memory and the OOM started thrashing. Thankfully there's always one SSH session open but I couldn't kill anything, sync or shutdown; fork failed with insufficient memory.

Left if thrashing overnight and had to power-kill it next day.

I'd say letting that server bounce and having the watchdog/load balancer work to keep you at capacity is the best option there. You are going to need that infrastructure anyway and if you can't malloc enough for one more client, who is to say that the existing stuff is going to be able to continue either?

You should count on any particular instance bouncing when that happens, and design your system to survive that. You should also invest some effort to figure out why your system can get into that state. Consider if any particular instance should be attempting to accept more jobs than it has the ability to handle. I shouldn't be able to trigger an out of memory situation on your IRC server just by connecting as many irc clients as I can.