How COVID-19 Impacts High Availability

Compared to friends, family, and those who have required treatment, hospitalization, or intensive care, my COVID symptoms have been mild. This is likely the result of reasonably good health, both doses of the vaccine, a booster shot, and early detection and treatment.  And, my heart goes out to every family who has lost a loved one to any aspect of this pandemic, and to all those who have lost opportunities and special moments.  As I and several members of our SIOS team recover from COVId-19, we wanted to share five things that your IT Team may be dealing with as they fight COVID and enterprise downtime, and five things you can do to help them.

Five COVID Concerns Facing IT Teams

1. Personal and Family Concerns and Fears

   Initially, my symptoms were barely noticeable, a slight irritation in my throat, and a little sinus drainage, which I self-diagnosed as seasonal allergies.  But when the issues worsened, accompanied by a bad cough I became worried.  Of course, we’d all like to think that our work performance and responsibilities remain unchanged, but the reality may be a little harder to assess. Despite initial negative tests, I continued to develop symptoms that eventually impacted my ability to work, increased my personal health concerns, and raised a number of fears. If your team has been directly affected by COVID-19, understand that they are likely dealing with personal concerns, fears, and worries in addition to the real health challenges that may impact their schedules, tasks, and activities.In the midst of their personal concerns each team member is likely also dealing with larger concerns, namely concerns about family.  During my illness, thankfully, my children all remained well.  However, my wife was not so lucky.  She became ill three days after my symptoms and remained ill longer and with more severe symptoms and setbacks.  While we have the benefits of a large family unit, a licensed teenage driver, and an extra car not driven by COVID-positive parents, your team may not have these luxuries.  And even if they do, it does not give them freedom from concern or reduce the amount of time and mental energy they need to apply to sanitize the home, keep their children in school and healthy, and deal with regulations, mandates, and close contact issues.  Not to mention concerns over income and expenses.  Team members facing personal and family concerns may experience difficulty concentrating, short-tempers, and difficulty meeting deadlines and schedules.

2. FODO – Fear of Disappointing Others

   Even without COVID-19 illness, businesses worldwide are feeling the impact of a smaller workforce.  The events aptly described as the “Great Shift”, “Great Resignation”, or “Great Shuffle” have already dramatically reshaped workforces, including those dealing with HA, leaving teams with fewer people to carry on critical tasks. This deficit in team members can lead those with COVID to battle a Fear of Disappointing Others (FODO).  Sick team members may continue to try to work out of loyalty to the team or a fear of disappointing bosses, peers, or stakeholders.  This FODO often leads to workers who are already functioning in a stressed environment (see #1 and 2 above) to attempt to maintain pre-COVID levels of activity. While heroic, it is also counterproductive to personal and professional recovery.

3. Fatigue

   As I continue to deal with COVID-19 symptoms, one of the biggest issues I continue to face is fatigue.  Initially, that fatigue, which was driven by FODO, prevented me from getting adequate rest and recovery.  Because I had seen how shorthanded our team was and witnessed others try to brave their illness to keep up with demand, I tried to do the same.  But, without warning I found myself drained, not at the end of the day, but for periods of time throughout the day.  For me, starting the day before 5am and continuing to focus on work, tasks, strategy, and personnel matters for 8 to 12 hours was normal.  (We can debate later if that was ever healthy). Now some felt like climbing Everest before 8 AM.  The best advice I received was from a friend and co-worker who said, “Don’t fight it. When your body says rest, rest!”

4. Brain Fog

   Around the same time that I started feeling sick, a colleague shared that they felt like they were in a fog following their bout with COVID symptoms. Like me, they were fully vaccinated and their symptoms and duration were mild. In fact, they actually never tested positive.  Nevertheless, they spent days with what we both termed “brain fog.”  An experience that we describe as slowness to recall details, a sense of knowing the answer, but lacking some mental sharpness that is somehow different from the physical fatigue and mental fatigue.  In some instances, it appears as a slower response to a question, a pause in the keystrokes, or a delay before the light comes on in the room.

5. Failed Recovery

   Five days into COVID, I woke up from an early night’s rest feeling better than ever. I jumped into my regular routine and by noon discovered that I had not fully recovered.  Instead I was exhausting a small store of energy gained by sleeping well the night before.  Trying to fight through this exhaustion created a new setback in my recovery.  The following day I felt worse than before.  The agony of a failed recovery and a concern about how to avoid more setbacks was added to my fatigue and fog.

So, what should IT team leads, stakeholders and managers do when their teams experience an issue with COVID-19.

Five Ways to Help IT Teams Battling COVID

1. Practice Empathy

   Be mindful that COVID affects each person and family differently.  Some of your coworkers and administrators will have minor issues, no symptoms, and no complications.  While others, single parents, multi-generational families, or families with children or vulnerable persons will have many more issues and concerns.  Know that the virus also impacts each person uniquely.  Even within my own family my symptoms and those of my wife were different.  While I experienced greater fatigue, she experienced more headaches. Have patience for coworkers who may be dealing with brain fog, juggling work schedules, caring for sick loved ones, or dealing with myriad issues related to COVID.

2. Assess needs

   Unlike the flu or common cold, COVID recovery is irregular. A team member may show up at work one day feeling much improved and stay home sick the next. Your business still has technical needs and requirements for high availability and disaster recovery.  However, with persons in and out of availability due to illness, be sure to understand the current roles and responsibilities required within the team.  When an individual is out sick, be sure to assess their role, their impact to the team, their level of responsibility to the infrastructure,etc.  You may also need to assess who within the team or organization can provide coverage in the event of a critical downtime event.

3. Prioritize issues

   Help your team by prioritizing key issues.  Under normal circumstances, your IT team is balancing dozens of requests ranging from the trivial (USB keyboard) to the critical (issues related to downtime, security threats, or storage issues).  While it may be obvious to you and the team, other stakeholders may need to understand the status of the IT team and how operations will be handled until a return to more “normal” staffing occurs.

4. Be sure your Processes are up-to-date

   As team members swap in and out, it is critical that IT maintenance and management processes are kept up to date.  These processes will help each member of the team service your enterprise effectively and efficiently when performing a task that is not their normal responsibility. It will also reduce the amount of time each team member needs to spend researching the status of the systems they are covering while a coworker recuperates.

5. Give People Time

   I’ve rushed back into the routine more than I should have, only to suffer the consequences of setbacks and greater fatigue on the following day.  As a leader or individual contributor on a team, be sure to give yourself and your team time to “get back to normal.”

As the pandemic continues, we all hope for a future that greatly resembles normalcy, including less illness, fear and worry.  In the meantime, being more aware of the concerns your team members are facing during COVID illness and recovery will greatly help you proactively prepare and weather the current storm.  In addition, key lessons learned from this pandemic can be applied across a number of other organizational, employee life, and global concerns.

Reproduced with permission from SIOS

Improving Your Cloud Adoption Journey

Highly Available or Highly Vulnerable? A Checklist for High Availability

It’s no secret that businesses of all sizes have an ever-growing need for IT systems. But IT systems are only effective for these businesses and their clients if they are operational, resilient and highly available. As enterprises look to build out their enterprise availability, having a baseline for weighing and assessing your vulnerability can be the difference that produces a successful merger of infrastructure, software, services and support that increases your success.

Sometimes, the most basic of checklists can help you sort through whether or not your solution is highly available or highly vulnerable?

Does your organization have the proper infrastructure to support high availability?

• Do your data centers have environmental sensors in place to measure building systems?
• Do your data centers have 24x7x365 operations?
• Does your data center include redundant power and network connectivity from diverse sources?
• Does your data center include multiple layers of host and storage services?
• As VP of Customer Experience, I have seen customers attempt to create a highly available solution without addressing fundamental foundational issues within their infrastructure.

They deploy software but have instability within the network infrastructure, servers, and datacenter itself. Cloud addresses a lot of the infrastructure issues, but not all cloud platforms are architected the same. Be sure to understand your datacenter, on-premises or cloud.

Does your organization have a runbook (or playbook) in place that covers design, architecture, and process?

• Is your runbook well documented, publicized and easily accessible?
• Are routine parts of your runbook sufficiently automated?
• Who has access to your enterprise runbook?
• Is it current and currently maintained?
• Is there version control for your runbook and any automation tools therein?

If you answered, what is a runbook or playbook then your first step is to find or create one. A runbook (or playbook) helps your organization maintain systems and processes with respect to the highly available system architecture. Some companies use automated tools to create scripts that deploy and configure servers, others use a version-controlled document to outline how all things work together to provide resilience and success. Your team needs to have a place that newcomers and existing team members can go to to understand the environment, the process, and the tools being used.

Does your organization have resources dedicated to maintaining high availability best practices?

• Does your organization give these employees and contractors support and training?
• Does your organization give these teams autonomy to adapt and create better best practices?

“I didn’t set these systems up,” the IT Admin stated, “I just inherited these systems with some other servers.” The lament was an honest and often observed phenomenon in organizations. Whether it is the result of mergers and acquisitions, cost reductions, outsourcing, or general staff turnover, a key component of a highly available enterprise is sufficient staffing. A key to a highly vulnerable enterprise is a lack of staffing, undertrained or undersupported staffing.

Does your organization have proper change management controls in place?

• Do you have a regular update policy and schedule?
• Do you have a defined process on patch maintenance?
• Do you have a review process in place for patches (vulnerabilities, threats, etc)?

Change management is important. Change management controls and polices are an absolute must in reducing risk and making sure that your systems are available. A user without proper restraints can add packages or updates that destroy stability, or make changes that disrupt the organization for hours. In addition, not having a defined policy often creates drift between what is expected (documented) and the actual (what is in place). Change management is also critical to ensure that your standby cluster is at the same patch and software levels as the primary/source system, and that QA (or Pre-Production) are not grossly deviating from Production.

Does your organization have proper access controls in place?

• Do you have account management tiers for server administration?
• Do you have controls to prevent accidental downtime?

Our Services team joined a customer call and waited, and waited, and waited for the administrator with permissions to run a set of elevated commands to join the session to configure and update their software. Weeks later, our team joined a different customer call and watched in horror as multiple users, all with administrative privileges, ran a bevy of commands on the same cluster. The difference in the two calls pointed out with stunning clarity that access controls are important. A highly available enterprise needs to ensure that proper access controls are in place that prevents users from running elevated commands that could damage the configuration or diminish its operation. Be sure that users have limits on what they can do based on their roles, needs, and even experience.

Does your company have a regular test process?

• Does your organization test in a pre-production or QA environment prior to production?
• Does your organization perform regular backups and backup testing?
• Does your organization practice disaster recovery scenarios and chaos testing for continuous improvement?

Testing takes time, but in my role of assisting customers with their cloud migrations and high availability deployments, the time has always been well spent. Often, the difference between the highly available and the highly vulnerable can come down to the customer or partner’s test process. As solutions become more complex, testing and validation are becoming more and more essential to reducing risk and vulnerabilities. If everything goes from design to production, you’re running a highly vulnerable system. But, if you’ve got tests and checkpoints, a process to verify changes before they make it into production your risks are significantly reduced. As VP of Customer Experience, our services team worked with a banner customer who deployed their systems for an entire year in QA before completing their go-live migration. Over that year they simulated outages, disasters, customer loads, downtime, maintenance, patching strategies, backups, recovery from backup, and a bevy of other test suites. Consequently, they’ve had remarkable results in performance, process adherence, high availability, and enterprise success.

While no checklist will be able to cover every potential vulnerability in high availability, answering these questions will give you a strong foundation for understanding if your enterprise is highly available or highly vulnerable.

Reproduced with permission from SIOS