I definitely don't come anywhere near Slack's scale but I've managed systems where over 3,000 cron jobs ran per day, half of which came from a cron job running every minute which usually finished in a few seconds. Some of these jobs run for X minutes too.
It's nice because there's properties you can configure for each cron job around retries and if it should be uniquely run or not. Maybe certain cron jobs should be re-tried if they fail, for others maybe it's ok to be picked up on the next interval if it fails.
Overall it's been super stable for almost 2 years which is when I started using them. Only a handful of jobs failed over this period of time and they weren't the result of Kubernetes, it was because the HTTP endpoint that was being hit from the cron job failed to respond and the cron job failure threshold was reached.
It's a good reminder that important jobs run on a schedule should be resilient to failure (saving progress, idempotent, etc.).
Do you capture all of your job code in a single image and reference execution paths on container startup per job? Or, are you building an image per job?
The jobs all run curl commands to a specific API endpoint with a specific bearer token. Those tokens are loaded through an env through SealedSecrets.
They all use the public curl image where I override the command in the Kubernetes cron job definition. The job container itself starts almost instantly since there's no app to boot.
If I had a case you're describing I would use the main app's image and run a specific command, in this case I'm assuming if there's not an API endpoint it would be some callable script that lives in your app's code / image.
I definitely don't come anywhere near Slack's scale but I've managed systems where over 3,000 cron jobs ran per day, half of which came from a cron job running every minute which usually finished in a few seconds. Some of these jobs run for X minutes too.
It's nice because there's properties you can configure for each cron job around retries and if it should be uniquely run or not. Maybe certain cron jobs should be re-tried if they fail, for others maybe it's ok to be picked up on the next interval if it fails.
Overall it's been super stable for almost 2 years which is when I started using them. Only a handful of jobs failed over this period of time and they weren't the result of Kubernetes, it was because the HTTP endpoint that was being hit from the cron job failed to respond and the cron job failure threshold was reached.
It's a good reminder that important jobs run on a schedule should be resilient to failure (saving progress, idempotent, etc.).