Recovery and Availability - One or Both?

A Brief Look at HA/DR

In this post I’ll be taking a look at recovery and availability, the technologies involved and what it all means to the two most interested departments involved in the recovery and availability decision making process: IT and Finance. Or to put it another way...

Separating the Two

Separating availability and recovery is fairly straight-forward and it comes down to tolerance of failure. If the application is required to be up 24/7, and any downtime results in financial losses, you need it to be highly available. If it’s something that can be down for a few hours or even days without anyone noticing, the clearly availability isn’t the key but more being able to recover service within a given time frame. A highly available solution will often involve a recovery strategy as well as a belt to go with the braces.

Availability and recovery decisions are driven by Recovery Point Objective (RPO) and the Recovery Time Objective (RTO), or in other words from which point in time do we need to recover and how quickly do we need it back, respectively. These are themselves linked to the service level agreement in place between the consumer of the service and whoever manages it, as well as the underpinning operating level agreements between the service provider and their third parties.

It can also be a question of scale. You wouldn't necessarily want your entire environment available 24/7, despite what  the individual users or groups might say., so you'd probably want key production systems protected and highly available and have a decent recovery solution in place, serving both to underpin the highly available systems and provide a recovery solution for the rest of the environment.

Availability

As mentioned briefly above, key applications may need to be available 24/7. For this we need to ensure data and services are accessible come what may. The concept of clustering isn’t a new one and has been used for many years. I was introduced to it in 1998 through the burgeoning Veritas Cluster Server, which provided two or more servers, initially at least in an “active-passive” relationship whereby one server, or “node” actively served the application while the other one or more servers in the cluster sat by ready and willing to take over, should something untoward befall he active node. Although not seamless in most cases, there being a short delay while the services came online on a passive node, it provided a tolerance that was good enough for most.

Since then there have been advances in the technology. Clustering can now (actually for quite a few years) be global, meaning that you can have clusters, even clusters of clusters, spanning geographical locations. Virtualisation technologies too can allow global clustering at the hypervisor level. In the case of VMWare, virtual machines can be moved about across vSphere servers to load balance and provide clustering capability.

These still rely on storage that can be shared between cluster nodes or replicated between locations. Much like Veritas, having been swallowed and eventually regurgitated by Symantec, the way storage is used for availability has seemingly come full circle. In what is seen my many as a return to the good/bad old days of local storage, the latest advances are doing away with the reliance on shared storage and replication technologies. What are now known as “availability groups”, at least in Microsoft land, can have synchronous copies of the same data existing across multiple servers in different geographical locations. MS SQL AlwaysOn Availability Group (AAG) and MS Exchange Database Availability Group (DAG) can support either of these services continuing uninterrupted in the event of a loss of a server or multiple servers, depending on configuration and budget.

Given enough priority and financial consideration, the loss of a datacenter can now potentially be no more than a slight inconvenience in productivity terms.

Recovery

How long can you afford for your application or service to be down? Exchange and SQL are critical applications that will cause most organisations to start taking a financial loss of some sort fairly quickly, but there are other applications that don’t warrant the spend – no point in spending tens of thousands on a subsystem that does not generate income or provide productivity infrastructure for doing so. In this case a robust recovery solution would be the best fit. A backup product periodically taking backups of data, servers, services or application stacks.

The Middle Ground

It is now common practice to make use of Software-as-a-Service (SaaS) solutions where the application is hosted on a cloud platform somewhere. The application’s availability and recoverability become the responsibility of the SaaS provider, if not de-risking the situation entirely then at least passing it on.

Comparison

So how do these options and methods stack up when you are attempting to define a recovery and/or availability strategy? The following table goes some way to explaining the choices and their implications. Clearly combining some of these can provide a more comprehensive and expansive HA DR strategy.

Method	Example RPO	Example RTO	Description	Considerations	Example Solutions
Active-Active Clustering	~ 0	~ 0	Availability groups maintain synchronous copies of data at disparate locations, meaning the data will be accessible from a second location in the event of a loss of server or site.	Ø  2 x sites required with global load-balancing Ø  Microsoft (or equivalent) availability licensing Ø  Resilient high bandwidth networking Ø  Resilient hardware (e.g. n+1) Ø  2 x data storage	Ø  Microsoft SQL AlwaysOn Availability Group (AAG) Ø  Microsoft Exchange Database Availability Group (DAG) Ø  Oracle Real Application Clusters (RAC)
Active-Passive Clustering	<15 minutes	<15 minutes	An active-passive clustered application should only undergo a minimal delay during failover while the passive node is brought online.	Ø  Cluster software & licensing Ø  Application cluster awareness	Ø  Microsoft Cluster Services Ø  Veritas Cluster Server Ø  VMWare ESXi
Server Replication	<1 hour	<4 hours	Replication applications can manage data replication over the network or storage layer and control of recovery of virtual machines through automation.	Ø  Adequate bandwidth for replication Ø  2 x data storage Ø  Replication software licensing	Ø  VMware Site Recovery Manager Ø  Veeam Ø  Zerto Ø  Storage array level replication (IBM, EMC, NetApp etc. all have their own)
Snapshots	<1 hour	<1 hour	Virtualisation or storage layer snapshots allow for point in time backups of servers or data that can be used to recover to a point in time.	Ø  Additional storage space needed for snapshot deltas Ø  Subject to the same vulnerability as the original data	Ø  Storage array based snapshots (NetApp, IBM, EMC etc.) Ø  Virtualisation hypervisor snapshots (VMware, Hyper-V)
Backup & Restore	Typically 12-24 hours	Typically 4-24 hours	The traditional view of disaster recovery where servers are backed up according to a schedule to disk and/or tape and recovered to the latest backup point. The belt to go with the braces and still a relevant DR strategy in its’ own right in some circumstances	Ø  Requires backup target hardware Ø  Management overhead Ø  Potential ong term warehouse storage	Ø  Veritas NetBackup Ø  CommVault Simpana Ø  EMC Networker Ø  CA ArcServe Ø  IBM TSM

Remember…

You need to stay on top of the recovery and availability strategy. It’s still all too often the case that HA and DR are implemented and then forgotten as I’ve mentioned in my post “Is DR still the IT poor relation?”.

A few things to keep in mind:

Review the recovery and availability strategies regularly

Application criticality changes What was critical last year may have moved up or down the priority list

Technologies change  There may be a better fit available

Is it still cost effective? Look at the availability, recovery and retention models you have in place to make sure it still makes sense financially

Test your recovery and availability strategies regularly. It worked last year but it may not work today.

Requirements change There’s a chance that something has been missed during implementation. It’s always better to find this out under test conditions

Applications change The way applications hang together and their associated availability and recovery implications can change between releases. Make sure it can still fail over or be recovered in the same way

People change It’s a fact of life, people move on. Make sure the recovery procedure is properly documented and run through the procedure during testing to ensure validity.

Technologies change Testing will ensure any interoperability issues are exposed, documented and resolved

Physical resources deteriorate over time Decay of magnetic media, also known as “bit rot” could mean that a restore required from 7+ years ago is not available. Whilst it is not usually required to restore data this old during day to day operation, and it is generally accepted that magnetic media will deteriorate, it is still advisable to test restores regularly and make plans for large scale backup data migration.[i]

You get what you pay for.

High availability isn’t cheap Everything needs to be redundant and monitored 24/7. I’ve stopped short of going into the infamous “9’s” discussion in this post but as an example a 99.99% uptime SLA commitment allows for just 52minutes 35.7 seconds a year in downtime before penalties are invoked. This is 263 seconds a month, 60.05 seconds a week or 8.6 seconds a day.

Not everything needs to be up 24/7 Find the balance, do the groundwork and only spend what you need to, i.e. spend on systems that will incur a tangible loss if down for any length of time.

Not everything needs to be kept forever Long term storage is an expensive commodity. There will be a certain amount of longer term retention in many companies, especially financials who are regulated by the FCA. The fact is not all of an FCA regulated company’s data needs to be subject to 7 or 10 year retention. Establish data classification and lifecycle management policies to ensure you are keeping only what you need to and for only as long as required.

---

[i] I was involved in the administration process of a large well known financial institution that went to the wall in the early 2000’s. The court cases went on for years, are probably ongoing now, where data was required from those old DLT tapes that were originally backed up in the 1990’s. Much of the time, the answer data simply wasn’t available due to media deterioration. Large scale data migration and validation added significantly to the overall cost and duration of the administration process.