|
|
|
Back to newsletter 127 contents
The tale from highscalability.com about tape backup still being important (see this month's listed news items below) reminds me of how I first learned the lesson that you need some kind of data storage which allows you to restore data from periodic backups. It was sometime in the 1990s, and a major exchange trading system went down and was unavailable for several hours. I recall an insider telling me why it took so long to get back online even though it had a live failover capability.
The system went down when corrupt data was detected - in an exchange system any corrupt data is hugely serious, since who knows what trades that might affect and how! At that point, the operators could have failed over after only a short outage. But the problem is - if the system has corrupt data, how do you know that is not systemic, and that the failover system isn't going to cause or use the same corruption? Without knowing what caused the corrupt data, failing over was extremely dangerous. While the failover system could be quickly analysed for integrity to see it didn't have the corrupt data, if the problem causing the corruption was part of the system then kicking off the failover would just result in another corrupt system, but with this time no possibility of failing over to a 'good' system.
So before failing over, the operators had to determine the cause of the corruption, and then evaluate their options. As it happened in this case, the primary disks had been corrupted and this was isolated so the failover was safe. The details always stuck in my mind as a cautionary tale - having failover capability is not enough, having a live backup is not enough (that could have been corrupted before it was noticed); you need access to older data so that you can recover from your own system causing corruption.
Now on with this month's newsletter and all our usual Java performance tools, news, and article links. We also have a new cartoon over at fasterj.com evolve your performance over time using industry best practices, and, as usual, all the extracted tips from all of this month's referenced articles.
Java performance tuning related news.
Java performance tuning related tools.
Back to newsletter 127 contents