Recovery and Availability - One or Both?

A Brief Look at HA/DR

In this post I’ll be taking a look at recovery and availability, the technologies involved and what it all means to the two most interested departments involved in the recovery and availability decision making process: IT and Finance. Or to put it another way...

Separating the Two

Separating availability and recovery is fairly straight-forward and it comes down to tolerance of failure. If the application is required to be up 24/7, and any downtime results in financial losses, you need it to be highly available. If it’s something that can be down for a few hours or even days without anyone noticing, the clearly availability isn’t the key but more being able to recover service within a given time frame. A highly available solution will often involve a recovery strategy as well as a belt to go with the braces.

Availability and recovery decisions are driven by Recovery Point Objective (RPO) and the Recovery Time Objective (RTO), or in other words from which point in time do we need to recover and how quickly do we need it back, respectively. These are themselves linked to the service level agreement in place between the consumer of the service and whoever manages it, as well as the underpinning operating level agreements between the service provider and their third parties.

It can also be a question of scale. You wouldn't necessarily want your entire environment available 24/7, despite what  the individual users or groups might say., so you'd probably want key production systems protected and highly available and have a decent recovery solution in place, serving both to underpin the highly available systems and provide a recovery solution for the rest of the environment.

Availability

As mentioned briefly above, key applications may need to be available 24/7. For this we need to ensure data and services are accessible come what may. The concept of clustering isn’t a new one and has been used for many years. I was introduced to it in 1998 through the burgeoning Veritas Cluster Server, which provided two or more servers, initially at least in an “active-passive” relationship whereby one server, or “node” actively served the application while the other one or more servers in the cluster sat by ready and willing to take over, should something untoward befall he active node. Although not seamless in most cases, there being a short delay while the services came online on a passive node, it provided a tolerance that was good enough for most.

Since then there have been advances in the technology. Clustering can now (actually for quite a few years) be global, meaning that you can have clusters, even clusters of clusters, spanning geographical locations. Virtualisation technologies too can allow global clustering at the hypervisor level. In the case of VMWare, virtual machines can be moved about across vSphere servers to load balance and provide clustering capability.

These still rely on storage that can be shared between cluster nodes or replicated between locations. Much like Veritas, having been swallowed and eventually regurgitated by Symantec, the way storage is used for availability has seemingly come full circle. In what is seen my many as a return to the good/bad old days of local storage, the latest advances are doing away with the reliance on shared storage and replication technologies. What are now known as “availability groups”, at least in Microsoft land, can have synchronous copies of the same data existing across multiple servers in different geographical locations. MS SQL AlwaysOn Availability Group (AAG) and MS Exchange Database Availability Group (DAG) can support either of these services continuing uninterrupted in the event of a loss of a server or multiple servers, depending on configuration and budget.

Given enough priority and financial consideration, the loss of a datacenter can now potentially be no more than a slight inconvenience in productivity terms.

Recovery

How long can you afford for your application or service to be down? Exchange and SQL are critical applications that will cause most organisations to start taking a financial loss of some sort fairly quickly, but there are other applications that don’t warrant the spend – no point in spending tens of thousands on a subsystem that does not generate income or provide productivity infrastructure for doing so. In this case a robust recovery solution would be the best fit. A backup product periodically taking backups of data, servers, services or application stacks.

The Middle Ground

It is now common practice to make use of Software-as-a-Service (SaaS) solutions where the application is hosted on a cloud platform somewhere. The application’s availability and recoverability become the responsibility of the SaaS provider, if not de-risking the situation entirely then at least passing it on.

Comparison

So how do these options and methods stack up when you are attempting to define a recovery and/or availability strategy? The following table goes some way to explaining the choices and their implications. Clearly combining some of these can provide a more comprehensive and expansive HA DR strategy.

Method	Example RPO	Example RTO	Description	Considerations	Example Solutions
Active-Active Clustering	~ 0	~ 0	Availability groups maintain synchronous copies of data at disparate locations, meaning the data will be accessible from a second location in the event of a loss of server or site.	Ø  2 x sites required with global load-balancing Ø  Microsoft (or equivalent) availability licensing Ø  Resilient high bandwidth networking Ø  Resilient hardware (e.g. n+1) Ø  2 x data storage	Ø  Microsoft SQL AlwaysOn Availability Group (AAG) Ø  Microsoft Exchange Database Availability Group (DAG) Ø  Oracle Real Application Clusters (RAC)
Active-Passive Clustering	<15 minutes	<15 minutes	An active-passive clustered application should only undergo a minimal delay during failover while the passive node is brought online.	Ø  Cluster software & licensing Ø  Application cluster awareness	Ø  Microsoft Cluster Services Ø  Veritas Cluster Server Ø  VMWare ESXi
Server Replication	<1 hour	<4 hours	Replication applications can manage data replication over the network or storage layer and control of recovery of virtual machines through automation.	Ø  Adequate bandwidth for replication Ø  2 x data storage Ø  Replication software licensing	Ø  VMware Site Recovery Manager Ø  Veeam Ø  Zerto Ø  Storage array level replication (IBM, EMC, NetApp etc. all have their own)
Snapshots	<1 hour	<1 hour	Virtualisation or storage layer snapshots allow for point in time backups of servers or data that can be used to recover to a point in time.	Ø  Additional storage space needed for snapshot deltas Ø  Subject to the same vulnerability as the original data	Ø  Storage array based snapshots (NetApp, IBM, EMC etc.) Ø  Virtualisation hypervisor snapshots (VMware, Hyper-V)
Backup & Restore	Typically 12-24 hours	Typically 4-24 hours	The traditional view of disaster recovery where servers are backed up according to a schedule to disk and/or tape and recovered to the latest backup point. The belt to go with the braces and still a relevant DR strategy in its’ own right in some circumstances	Ø  Requires backup target hardware Ø  Management overhead Ø  Potential ong term warehouse storage	Ø  Veritas NetBackup Ø  CommVault Simpana Ø  EMC Networker Ø  CA ArcServe Ø  IBM TSM

Remember…

You need to stay on top of the recovery and availability strategy. It’s still all too often the case that HA and DR are implemented and then forgotten as I’ve mentioned in my post “Is DR still the IT poor relation?”.

A few things to keep in mind:

Review the recovery and availability strategies regularly

Application criticality changes What was critical last year may have moved up or down the priority list

Technologies change  There may be a better fit available

Is it still cost effective? Look at the availability, recovery and retention models you have in place to make sure it still makes sense financially

Test your recovery and availability strategies regularly. It worked last year but it may not work today.

Requirements change There’s a chance that something has been missed during implementation. It’s always better to find this out under test conditions

Applications change The way applications hang together and their associated availability and recovery implications can change between releases. Make sure it can still fail over or be recovered in the same way

People change It’s a fact of life, people move on. Make sure the recovery procedure is properly documented and run through the procedure during testing to ensure validity.

Technologies change Testing will ensure any interoperability issues are exposed, documented and resolved

Physical resources deteriorate over time Decay of magnetic media, also known as “bit rot” could mean that a restore required from 7+ years ago is not available. Whilst it is not usually required to restore data this old during day to day operation, and it is generally accepted that magnetic media will deteriorate, it is still advisable to test restores regularly and make plans for large scale backup data migration.[i]

You get what you pay for.

High availability isn’t cheap Everything needs to be redundant and monitored 24/7. I’ve stopped short of going into the infamous “9’s” discussion in this post but as an example a 99.99% uptime SLA commitment allows for just 52minutes 35.7 seconds a year in downtime before penalties are invoked. This is 263 seconds a month, 60.05 seconds a week or 8.6 seconds a day.

Not everything needs to be up 24/7 Find the balance, do the groundwork and only spend what you need to, i.e. spend on systems that will incur a tangible loss if down for any length of time.

Not everything needs to be kept forever Long term storage is an expensive commodity. There will be a certain amount of longer term retention in many companies, especially financials who are regulated by the FCA. The fact is not all of an FCA regulated company’s data needs to be subject to 7 or 10 year retention. Establish data classification and lifecycle management policies to ensure you are keeping only what you need to and for only as long as required.

---

[i] I was involved in the administration process of a large well known financial institution that went to the wall in the early 2000’s. The court cases went on for years, are probably ongoing now, where data was required from those old DLT tapes that were originally backed up in the 1990’s. Much of the time, the answer data simply wasn’t available due to media deterioration. Large scale data migration and validation added significantly to the overall cost and duration of the administration process.

Storage Concepts: Software Defined Storage (SDS)

What is SDS?

Software Defined Storage, or SDS, is now integral to most virtualisation technologies. Indeed, the term is often used interchangeably with “Storage Virtualisation” – rightly or wrongly is material for another article. SDS could be said to define the suite of tools available to virtualise storage than the virtualisation itself, but I will confine this article to speaking interchangeably about SDS and storage virtualisation. As with all “Software Defined…” concepts, the key premise is abstraction, in this case of physical storage, and its presentation as a pool of usable data storage that can be allocated to devices, often through automation at the virtualisation layer. A much over-used expression is the Single Pane of Glass (SPOG) but it is applicable here in that SDS software provides an overall view of the storage without the need to monitor and manage individual storage devices. The software, quite literally, defines how the storage is allocated and used.

How is SDS used?

Imagine an environment with a mature storage estate that comprises a number of disparate storagedevices, let’s say EMC VNX, IBM V7000 and HDS HUS300, managed across several management platforms (EMC Ionix, IBM Flex System Management, Hitachi Device Manager, etc.). There is a big push for virtualisation through VMware but the storage side needs to remain in place until the virtualisation project is completed. Freeing up space from the virtualisation project, through consolidation, archive etc; means that these existing storage systems are more than adequate to service requirements of the virtualised platform. SDS software can aggregate the storage from all three storage devices and allow control of provisioning through the virtualisation software. The storage is aggregated into pools and provisioning can be manually or automatically done, depending on the organisation’s needs, entirely through the virtualisation software which in this case would be VMware vSphere. I guess I should say that other platform virtualisation technologies are available. Whether I believe it or not.

Automation

There is a high level of automation available to those who wish to virtualise their storage. It is possible, though maybe not always advisable, to allocate multiple tiers of storage into a large pool and let the underlying storage take care of the tiering. Consider a datastore has been created from a virtualised storage pool containing tier 1, 2 and 3 storage – let’s say a couple of EMC VNXs and an HP P4335 (odd mix, but you know how these things often happen). The storage can be pooled and allocated to a subset of virtual machines which all have differing storage performance requirements (IO profiles if you will) and the underlying storage will take care of the location of the storage through automated tiering of areas of high activity and performance requirement (hot-spots). Those virtual machines, after a period of learning, will have their performance requirements automatically met, assuming there is enough higher tier storage available.

Virtual Storage Appliances

Virtual Storage Appliances (VSA) are used to perform the storage virtualisation function. These will take an underlying storage system and allow it to be used with the abstraction layer’s software capabilities. One example of which would be where cross-site clustering is required but existing storage is unable to perform the synchronicity required to keep everything up to date. Implementing a VSA to sit above the storage and manage the replication and synchronisation allows a lower cost, cross-site storage solution to be implemented. Whether a low cost implementation of business critical clustered applications is appropriate is a discussion for those who will be picking up the bill.

Is SDS New?

As with many technical innovations, they are variations on an old theme. Mainframe computing used the concept of storage virtualisation way back in the ‘70s, believe it or not. IBM’s MVS (which stands for Multiple Virtual Storage) made use of shared memory across many virtual systems and workloads. Storage meant “memory storage” in this case but the principle was much the same. Storage systems as we would recognise them were still some way off.

What SDS Solutions Are Out There?

Here are a few of the better known SDS vendors:

• Nexenta
• Symantec (Storage Foundation/InfoScale)
• DataCore
• VMware
• EMC (ViPR, vVNX)
• HP (StoreVirtual VSA)

All provide the ability to centralise storage management and provide overall control of the storage landscape to virtualisation platforms. Some of these may be contentious and open to argument around whether they truly offer software defined storage, but they all put in a pretty good claim.

This article details these vendor’s offerings, and more, rather well.

Who Uses SDS?

Pretty simple to answer – anyone who requires a high level of automation, simplification and abstraction from their storage environment. That would include:

• Cloud service providers
• Anyone wishing to virtualise their estate
• Anyone with heterogeneous, legacy storage environments
• Anyone with a wide range of performance requirements

Anyone who wants to simplify storage management, in a nutshell.

Storage Concepts: Erasure Coding (EC)

What is Erasure Coding?

Wikipedia: “In information theory, an erasure code is a forward error correction code for the binary erasure channel, which transforms a message of k symbols into a longer message with n symbols such that the ...”

The Wikipedia entry is probably not the best place to start, let’s look at it from above the algorithms and formulae. What does Erasure Coding mean broadly, in terms of data storage? Well, erasure coding is a method of securing data on disk and can be thought of as an evolution of RAID in some respects, so to touch on RAID briefly before we look at Erasure Coding, traditional RAID will secure data by either mirroring it across one or more disks (RAID 1) or will write parity to one or more disks in a RAID set (RAID 5, RAID 6 aka RAID DP etc.) in order to ensure data is not lost in the event of a disk failure. Hot spares are kept in order to write the data from the failed disk onto, until the original disk is replaced. There are obvious benefits to this way of protecting data but the downside is becoming more evident as data volumes, and perhaps more pertinently, disk sizes increase. Mirroring and parity will require more and more disk as data volumes grow and replacing failed disks means  times on these vast disks is greatly increased, exposing the data to further risk.

Example: File F is split into 9(k) parts, 

which are themselves encoded into

 12(n) further parts, each containing 

effectively 1/9th of the data (it’s not quite

 as simple as that, but helps

 illustrate the principle). As long

 as we still have 9 of the 12 x 1/9th parts 

the original data can be reconstituted.

On to Erasure Coding. Data is split up, expanded with redundant data and sent to a target device in another location, which can be on the same array, local arrays in the same DC and/or remote arrays nationally or geographically. The data at the other ends can be decoded from this source or any other source of the data from any other location meaning that if this particular packet of data is lost, it can be retrieved from a subset of the same data from the other locations. To use a rather tenuous analogy, if you pack elements of your holiday wardrobe across three suitcases, you will arrive in Barbados safe in the knowledge that you will have something to wear even if one of your suitcases ends up in Luton. If two cases are lost, you may end up wearing just speedos and a smile all week. In other words, your suitcase fault tolerance is 2 out of 3.


In a little more detail, Erasure Coding splits data and shares it across locations to be reconstructed in the event of a corruption or loss of data in one or a number of the others. The concept is actually relatively simple to get a handle on. A file or object, F, is split up into k parts, which are then themselves encoded into n parts. To recreate F we would need k out of n parts to recreate the data. An EC configuration might require 9(k) from 12(n)parts, meaning that any 9 subsets of the data out of 12 can be used to reconstruct your original data (see example, right). Any less than that and you can’t reconstruct.

This number can be lower, for example 6 from 12, but then you would need a lot more space and bandwidth to accommodate the fact that all of the data needs to be retrievable from less locations, so therefore contain more redundant* data for the other locations – you might as well mirror or replicate.

* Redundant in this case taking the high availability meaning, as opposed to 'unused'

Is Erasure Coding new?

The theory isn’t. As is often the case with new technology, the principle has been around for some time in the form of Forward Error Correction, first proposed by Reed and Solomon in 1960. In order to ensure that transmissions were received, they were split and encoded, along with redundant data, with an algorithm. As long as the algorithm is known at the other end, any errors can be corrected accordingly. Put horribly simply, if I sent a transmission containing C1H2R3I4S5, and my fiendishly complex algorithm of adding a number to each letter was known at the other end, the receiver would know there was a problem if they received D4A3V2E1 along with the C1H2R3I4S5 in the same transmission. Erasure coding is Forward Error Correction for data replication, storage and security, but instead of correcting errors we are protecting against data loss.

Security

There is an inherent security in Erasure Coding. The fact that the data is broken up and recreated using an algorithm means that, unless that algorithm is known, the data is meaningless. This lends the data transmission an in-flight security of its’ own.

Who has implemented it?

There are a number of storage that have taken up Erasure Coding, often in conjunction with Object Based Storage:

•         SwiftStack
•         EMC Atmos
•         DDN ObjectAssure
•         Scality
•         Hadoop
•         NetApp
•         Cleversafe

Is DR still the IT poor relation?

DR – Disaster Recovery, not Diminishing Returns

Disaster Recovery solutions are still all too often seen as an afterthought, a necessity that drains money and resource, that will not provide an ROI and will most likely never be used in a real-life situation. The 2001 attack on the World Trade Center could be said to have been the trigger for many organisations taking disaster recovery far more seriously, resulting in a greatly increased focus on DR capability which led to the rapid deployment of additional hardware and resource to provide cover against similar disasters, whether man-made or natural.

Time and distance have eroded those early concerns over availability, which means that there are disaster recovery solutions that were put in place shortly after the events of 2001 that are now many years old, sparsely reviewed and rarely, if ever, tested.

Making appropriate plans for Disaster Recovery is essential.  There are some key questions that will factor in every DR decision making process, and they have a common theme: what are the consequencesof the organisation being out of action?

The answers will shape your Disaster Recovery solution.

Disaster Recovery doesn’t just refer to the infrastructure in place to support a recovery from a catastrophic failure. The right technology is essential of course, but just as important are the processes and the people who make up the day to day operation of the organisation. Effective DR planning needs to incorporate an equally effective Business Continuity strategy. Once an effective strategy has been decided upon, it needs to be documented and accessible to everyone who needs to know what to do in the event of DR invocation.

Having implemented a sound Disaster recovery and Business Continuity strategy it is vital that it be tested regularly. Full testing is imperative: the systems and services that are used in daily operation need to be offline and operation in in the DR location needs to be able to continue for as long as the recovery will take. Adequate provision for DR resources is an essential part of the DR solution, especially where there might be a significant delay factored into the return to business as usual.

One more point that will always stand repetition is that cost should not be a determining factor when planning a DR solution. There is a very simple equation which is ignored surprisingly often.

To put it simply, the cost of the DR solution should be relative to the cost of an outage. If your organisation would lose $10m an hour, your disaster recovery solution should be capable of recovering in a lot less than an hour and if it costs $10m to have the system in place to do it, the organisation wins in the event of DR invocation. If you spend $10k but your operation is down for 3 days… well the point is clear.

Now, some of this may not come as much of a surprise and it could be said that much of it is obvious. Maybe so, but it might just be worth taking a look at those DR and BCP plans to see if these obvious requirements are actually met. It might not be as simple to accomplish as it first seems so it is always worth seeking help from experienced professionals who make it their business to ensure your business continues.

Storage Concepts: Object Based Storage (OBS)

What is object-based storage?

Essentially, object-based storage, or OBS, a different way of storing and accessing data. Traditionally storage has been stored using block and file structures. Just to summarise all of these:

Representation of traditional block 

based storage, with data stored 

in equal sized block on disk

Block storage can be thought of in terms of LUNs (Logical Unit Numbers) that are presented to servers and applications as discrete disks that act as local devices on the server - essentially buckets containing blocks of data. SAN infrastructure is the mechanism by which the data is presented to servers and applications using the standard scsi protocols over fibre channel, iscsi and FCoE etc. Access to block data is controlled through the storage array through host mapping and further through fibre channel switches through zoning.

File based storage, using heirarchical files and folders 

common on NAS devices.

File storage is presented as folders and/or directories over the network using the NFS and CIFS protocols from NAS devices (“filers”). The structure can be thought of as nested drawers of files that are shared out to devices on a network for access by anyone with the relevant credentials for authentication. NFS has been experiencing something of a revival in the virtualisation space, being highly proficient in presentation of datastores to virtual environments.

Object based storage can be represented 

as a lookup table containing information 

needed to retrieve the object from storage.

Object Based Storage introduces the concept of a flat structure with unique identities for data objects. Each object has its’ own hash signature which is created according to the contents of the object.  Additionally, the metadata (or descriptive header) of an object can contain a great deal more than a traditional file. The data is accessed through the http protocol using the established authentication methods associated with it (such as X-Auth). It is worth noting that block and object storage are not mutually exclusive. The two complement each other with block storage providing the back-end and the object storage software providing the features and functionality off-array.  

Is object-based storage new technology?

No. It existed for some time in some form as CAS (variably known as Content Addressed, Addressable or Aware Storage).  In fact, the concept has been around for nearly 20 years but, necessity driving innovation, companies have developed their object storage strategies at an accelerated pace over the last 6 years or so in response to the rising need to tame their vast data estates and make them as accessible, responsive, contained and cost-effective as possible. 

Why is object storage better than traditional storage?

It is not better in all cases, such as pure file-sharing environments and the frequent access and update requirements of a relational database, but there are some definite advantages in a number of areas. Object-based storage can certainly be a cheaper alternative to the expensive, high end storage products that use proprietary management and administration tools. Cheaper back-end block storage can be used with an object storage software system to provide all of the additional features and functionality. Another benefit comes with the object metadata. With traditional block and file storage, a content management system, often with a relational database, would have to be used to store additional information on a file that cannot be stored with it. The objects in object storage have this information contained within them and it is highly customisable.

How and where is object storage used?

Any organisation that has a requirement to create and/or store large amounts of relatively static data would find object-based storage extremely effective. Examples include

• Cloud providers make extensive use of object storage due to its’ scalability, cost efficiency and ease of interface with cloud portal development.
• Social networking sites with thousands and perhaps many millions of pictures, videos and texts (Facebook, Twitter, Linkedin).
• Publishing companies that produce and are required to keep periodical publications for use in stats gathering and Big Data analysis.
• Healthcare organisations that are required to maintain a repository of scan and x-ray images, which are often used once and then kept to be referred to infrequently, if at all.

The common factor is evident from this list. Vast quantities of “unstructured” data that needs to be stored at as low a cost and footprint as possible

Is object storage really “infinitely scalable”?

Well, “infinite” is an over-used expression in the IT world but one could, in theory, continue to add storage nodes until you run out of money, space or time to an object storage environment. The object storage management system can accommodate continued additions to the back-end storage as long as the metadata and ID references can be maintained on the object storage nodes.

How is availability and redundancy maintained in object storage?

A key attribute of object storage is that copies of the objects can be stored, accordingly to definable policies, across multiple locations locally, nationally and internationally making it truly resilient to local DR scenarios.

Which vendors have implemented it?

Several players have emerged with a head above the clouds (pun intended) in the object storage arena, notably:-

• EMC – ISOLON, ATMOS, Centera
• DirectData Networks – Web Object Scalar (WOS)
• NetApp - StorageGRID
• Scality – Software based “RING” scale-out storage solutions
• HDS – HCP (Hitachi Content Platform)

Others are coming along behind, and undoubtedly every storage vendor has their own agenda in this area.