Wow, do you use Spark in production? I can't claim to have much experience with it, I just played around with it a loooong time ago, but I was impressed. I thought it went out of use along with Hadoop and HBase, glad to hear this is not the case. Are you hiring? :-D