On issue we have is that although we've made our services very
resilient, by employing HA failover, load balancing and round robin DNS,
the one service that's difficult to do anything with is cron, because it
has traditionally been tied to a single machine.
For instance, we have a cluster of 4 Fedora servers which our end users
use as timeshare systems, using round robin DNS, and if one of the
servers is down it doesn't really matter too much. We don't even backup
the servers, relying on NFS home directories and rebuilding from scratch
using kickstart + cfengine if a server fails and can't be restarted.
However, the users have scattered crontab files around the 4 servers, so
that if the particular server a cron job is meant to run on dies, the
job doesn't run any more, and the crontab file may be permanently lost.
What we needed was a facility to allow crontabs in /var/spool/cron on
these 4 servers be NFS mounted from a single directory on our NetApp
filers (so giving us backups and snapshots), with any user able to run
"crontab -e" from any of the servers to manage a single shared crontab,
and for us in the IT Service to be able to set just one of the 4 servers
to run user crontab jobs at any time. However, we needed to keep
/etc/crontab and the files in /etc/cron.d/ specific to each individual
server still, and keep crond running on all 4 servers.