Beginning Well is Great, But Maintaining Uptime Takes Vigilance

Author Isabella Poretsis states, “Starting something can be easy, it is finishing it that is the highest hurdle.” It is great to have a kickoff meeting.  It is invigorating, and exciting. Managers and leaders look out at the greenfield with excitement and optimism is high.  But, this moment of kickoff, and even the Champagne popping moment of a successful deployment are but just the beginning. Maintaining uptime requires ongoing vigilance.

High availability and the elusive four nines of uptime for your critical applications and databases aren’t momentary occurrences, but rather, a constant endeavor to end the little foxes that destroy the vineyard.  Staying abreast of threats, up-to-date on the updates, and properly trained and prepared is the work from which your team “is never entitled to take a vacation.”

For those who want to stay vigilant in maintaining uptime, here are five tips:

1. Monitor the Environment 

Very little in enterprise software still follows the “set it and forget it” mindset.  Everything, since the day you uncorked the grand opening champagne to now, has been moving toward a state of decline.  If you aren’t monitoring the servers, workloads, network traffic, and hardware (virtual or physical), you may lose uptime and stability.

2. Perform Maintenance

One thing that I have always noticed in over twenty plus years of software development and services is that all software comes with updates.  Apply them.  Remember to execute sound maintenance policies, including taking and verifying backups. One tech writer suggested the only update you regret is the one you failed to make.

3. Learn Continuously

My first introduction to high availability came when I unplugged one end of the Token Ring for a server in our lab as an intern, fresh from the CE-211 lab.  The administrator was in my face in minutes.  After an earful, he gave me an education.  Ideally, you and your team want to learn without taking down your network, but you do absolutely want to keep learning.  Look into paid courses on existing technology, new releases, emerging infrastructure.  Check your vendors for courses and items related to your process, environment, software deployments and company enterprise.  Free courses for many things also exist if money is an issue.

4. Multiply the learning

In addition to continuous learning, make a plan to multiply the learning.  As the VP of Customer Experience at SIOS we have seen the tremendous difference between teams who share their learning and those who don’t.  Teams that share their learning avoid gaps in knowledge that compromise downtime.  The best way to know that you learned something is to teach it to somebody else. As you learn, share the learning with team members to reduce the risk of downtime due to error, and for that matter vacation.

5. End well . . .before the next beginning

All projects, servers, and software have an ending.  End well.  Decommission correctly.  Begin the next phase, deployment, software relationship, etc well by closing up loose ends, documenting what went well, what did not, and what to do next.  Treat your existing vendors well.  You just may need them again later.  Understand the existing systems and high availability solutions before proceeding with a new deployment.  This proper ending helps you begin again from a better starting place headed towards a stronger outcome.

Keeping the system highly available is a continuous process.  Set it and forget it is a nice catch phrase, but the reality is that uptime takes vigilance, continual monitoring, proper maintenance, and constant.

-Cassius Rhue, VP, Customer Experience

Reproduced with permission from SIOS

Understanding and Avoiding Split Brain Scenarios

Split brain. Most readers of our blogs will have heard the term, in the computing context that is, yet we cannot help but to sympathize with those whose first mental image is of the chaos that would result if someone had two brains, both equally in control at the same time.

What is a Failover Cluster Split Brain Scenario?

In a failover cluster split brain scenario, neither node can communicate with the other, and the standby server may promote itself to become an active server because it believes the active node has failed. This results in both nodes becoming ‘active’ as each would see the other as being failed. As a result, data integrity and consistency is compromised as data on both nodes would be changing. This is referred to as split brain.

There are two types of split-brain scenarios which may occur for an SAP HANA resource hierarchy if appropriate steps are not taken to avoid them.

• HANA Resource Split Brain: The HANA resource is Active (ISP) on multiple cluster nodes. This situation is typically caused by a temporary network outage affecting the communication paths between cluster nodes.
• SAP HANA System Replication Split Brain: The HANA resource is Active (ISP) on the primary node and Standby (OSU) on the backup node, but the database is running and registered as the primary replication site on both nodes. This situation is typically caused by either a failure to stop the database on the previous primary node during failover, having Autostart enabled for the database, or a database administrator manually running “hdbnsutil -sr_takeover” on the secondary replication site outside of the clustering software environment.

Avoiding Split Brain Issues

Recommendations for avoiding or resolving each type of split-brain scenario in the SIOS Protection Suite clustering environment are given below.

While in a split-brain scenario, a message similar to the following is logged and broadcast to all open consoles every quickCheck interval (default 2 minutes) until the issue is resolved.

EMERG:hana:quickCheck:HANA-SPS_HDB00:136363:WARNING: 
A temporary communication failure has occurred between servers 
hana2-1 and hana2-2. 
Manual intervention is required in order to minimize the risk of 
data loss. 
To resolve this situation, please take one of the following resource 
hierarchies out of service: HANA-SPS_HDB00 on hana2-1 
or HANA-SPS_HDB00 on hana2-2. 
The server that the resource hierarchy is taken out of service on 
will become the secondary SAP HANA System Replication site.

Recommendations for resolution:

1. Investigate the database on each cluster node to determine which instance contains the most up-to-date or relevant data. This determination must be made by a qualified database administrator who is familiar with the data.
2. The HANA resource on the node containing the data that needs to be retained will remain Active (ISP) in LifeKeeper, and the HANA resource hierarchy on the node that will be re-registered as the secondary replication site will be taken entirely out of service in LifeKeeper. Right-click on each leaf resource in the HANA resource hierarchy on the node where the hierarchy should be taken out of service and click Out of Service …
3. Once the SAP HANA resource hierarchy has been successfully taken out of service, LifeKeeper will re-register the Standby node as the secondary replication site during the next quickCheck interval (default 2 minutes). Once replication resumes, any data on the Standby node which is not present on the Active node will be lost. Once the Standby node has been re-registered as the secondary replication site, the SAP HANA hierarchy has returned to a highly available state.

SAP HANA System Replication Split Brain Resolution

While in this split-brain scenario, a message similar to the following is logged and broadcast to all open consoles every quick. Check interval (default 2 minutes) until the issue is resolved.

EMERG:hana:quickCheck:HANA-SPS_HDB00:136364:WARNING: 
SAP HANA database HDB00 is running and registered as 
primary master on both hana2-1 and hana2-2. 
Manual intervention is required in order to 
minimize the risk of data loss. To resolve this situation, 
please stop database instance 
HDB00 on hana2-2 by running the command ‘su – spsadm -c 
“sapcontrol -nr 00 -function Stop”’ 
on that server. Once stopped, 
it will become the secondary SAP HANA System Replication site.

Recommendations for resolution:

1. Investigate the database on each cluster node to determine whether important data exists on the Standby node which does not exist on the Active node. If important data has been committed to the database on the Standby node while in the split-brain state, the data will need to be manually copied to the Active node. This determination must be made by a qualified database administrator who is familiar with the data.
2. Once any missing data has been copied from the database on the Standby node to the Active node, stop the database on the Standby node by running the command given in the LifeKeeper warning message:
   

   su – adm -c “sapcontrol -nr <Inst#> -function Stop”

   where is the lower-case SAP System ID for the HANA installation and <Inst#> is the instance number for the HDB instance (e.g., the instance number, for instance, HDB00 is 00)

3. Once the database has been successfully stopped, LifeKeeper will re-register the Standby node as the secondary replication site during the next quickCheck interval (default 2 minutes). Once replication resumes, any data on the Standby node which is not present on the Active node will be lost. Once the Standby node has been re-registered as the secondary replication site, the SAP HANA hierarchy has returned to a highly available state.

Being aware of common split-brain scenarios and taking these steps to mitigate them can save you time and protect data integrity.

Reproduced with permission from SIOS

High Availability Architecture and Best Practices

13 Little Known Facts about High Availability

1. Hypervisor HA is not the Same as Application HA

A key misconception is that I have high availability because I have redundancy in my hardware or hypervisor. However, hardware and hypervisor redundancy does not guarantee high availability for applications. It is also not a guarantee that orchestration of applications will be properly executed in a failure.

2. In High Availability, Bigger Does Not Equal Better

If you are a powerlifter, bigger weights are better and smaller reps are better. Or, if we are talking about hugs. (You remember hugs are the things that we used to do when we saw a friend from a different town, that we hadn’t seen in a while.) But, bigger doesn’t always mean better. A bigger kidney stone, for example, is definitely not better. In higher availability, creating a bigger, more complex solution doesn’t always mean that you’ll have increased your high availability. It might mean you have the same availability or less. It may also mean that you have a bigger, more complex system with a lot of moving pieces to sort through in an outage.

3. Everything fails… sometimes

Application programming languages date back to the 1950’s. And while the languages, processors, IDEs, and quality of the code has improved, the reality is “all applications fail at some point.” Failures due to exceptions, bugs, unhandled terminations, accidental terminations, resource exhaustion, and more happen. Having an active/active, or active/passive application availability strategy is still necessary.

4. Focus on ‘why’ as much as ‘how’

Our natural tendency to jump into task completion mode is a necessary asset, but it needs to be tempered and guided by the answer to our questions of why. Adding a solution to an environment without understanding the business, application, database, and stakeholder requirements will lead to either a:

• Failure
• Over expenditures
• Underperformance
• Confusion and over architecture
• All of the above

Instead of focusing solely on getting availability implemented, spend the necessary resources and effort to understand the business needs and answers to “why”

5. Unpatched issues are a common source of regret

If you do or you don’t you will have consequences. The consequence of all unpatched issues is regret. As VP of Customer Experience I have seen firsthand the downtime caused by customers failing to address known issues in a timely manner.

6. Undocumented issues cause downtime too

Picture the scene. A new admin is looking into servers on the network. The usage reports indicate the server is not active and no clients are connected. Not recognizing the server and finding no “tags”, documentation or other identifiers, the new admin believes that it should be shut down. Unfortunately the undocumented and uncommunicated instance is actually a standby server whose removal will cause downtime when the primary crashes unexpectedly. This isn’t a fictional story, this is the true story of a new admin who incorrectly identified a server as an idle QA system and shut it down prior to a patching exercise.

7. Complacency is also an enemy

We’d all love it if availability on premises or in the cloud, or anywhere in between was something that we can “set and forget.” But, few, if anything in life is really as simple as “set it and forget it.” One of the biggest enemies of your availability in the future, is your success with high availability now. When disasters are few and far between, and teams feel confident that they have realized sustained stability, complacency can step in. Success tempts us to think nothing is going to change, and complacency in respect to high availability therefore is an enemy to high availability. Things around your enterprise and within your enterprise are changing. The cloud is changing, your business needs are changing, and the applications and Operating Systems are also changing.

8. Change is hard

Change is hard. Just ask anyone with a sweet tooth who’s been trying to give up that second slice of cake before bedtime. Similar resistance occurs even in high availability. Teams, even those who experience disasters, are often reluctant to change even if the change is good. They need a vision, an understanding of why, and support. Other teams, those with solutions in place, are reluctant to improve high availability with fear of introducing instability or exposing themselves to new risk.

9. All change is not good change

Change is good, when change is good. When considering a change to the higher availability solution and architecture it is critical that changes are analyzed against the goals, the requirements, and within the scope of increasing availability. Changes that increase stability, add protection for critical components, eliminate workarounds, optimize the availability of services and are thoroughly tested are good changes.

10. Cheaper is not always better

Cheaper is not always better. While cheaper solutions typically have a lower price tag, they may also come with a number of limitations that make them less than ideal. When there is a lower price tag, beware of missing features such as a lack of application awareness, limited orchestration, hidden complexity, manual recovery and failover, and limited to no user validation. Cheaper solutions may also fail to include customer support. Be sure to understand whether your cheaper solution includes support, or if the support is an additional, and substantial addon cost.

The same applies to cheaper deployments with reduced compute, disk or storage. While the price tag and monthly cost might be lower, your solution may also be functioning at a less than ideal capacity.

11. Loud does not equal effective

Ever heard the story of the boy that cried wolf. An application monitoring solution that produces an alert storm is sooner than later a solution that gets ignored. Having a solution that provides alerts is great, but if that solution triggers critical alerts in error or in excess, it is ineffective.

12. High Availability is a culture and a mindset, not just a product or hardware solution

Software, hardware, processes, solutions and services are all a part of high availability. However, without a buy-in across IT functions and business units, it will be fraught with frustration and constantly the source of budget discussions instead of discussions on value, business stability, increased customer satisfaction, and diminished risk.

13. Now is not too late

Hope is not a strategy for high availability, nor does hoping that you will not have a critical disaster or application failure need to be a strategy. Designing and architecting a highly available enterprise architecture can be made possible now, even if it has been weeks or months since the last disaster.

Contact SIOS to learn more about high availability solutions for your application.

– Cassius Rhue, VP, Customer Experience

Reproduced from SIOS