SIOS SANless clusters

SIOS SANless clusters High-availability Machine Learning monitoring

Fix Your SQL Server Multi-Subnet Failover Cluster

by Leave a Comment

Fix-Your-SQL-Server-Multi-Subnet-Failover-Cluster

Help! I Can’t Connect To My SQL Server Multi-Subnet Failover Cluster

Many of my customers have encountered this problem. To solve it, I usually tell them the following

The link would have everything you need to know.

They don’t go into great detail about what to do if your connection does not support multisubnetfailover=true. If your connection does NOT support that parameter, then set registerallprovidersip to false and cleanup DNS. That procedure is described best here.

How Does SQL Server Multi-Subnet Failover Cluster Actually Work?

In general people aren’t aware of how SQL Server Multi-Subnet Failover Cluster work. Multi-subnet failover clustering support was added in Windows Server 2012 with the addition of the “OR” technology when defining cluster resource dependencies. This allowed people to allow a Cluster Name resource to be dependent upon IP Address x.x.x.x OR IP Address y.y.y.y.
x.x.x.x would be an a cluster IP resource valid in Subnet A. y.y.y.y would be a cluster IP address valid in Subnet B. Only one address will be online at any given time, whichever address was valid for the subnet the resource was currently running on.
Microsoft SQL Server started supporting this concept starting with SQL Server 2012 with both failover cluster instances (FCI) using 3-party SANless clustering solutions like SIOS DataKeeper and SQL Server Always On Availability Groups.
By default if you create a SQL Server Multi-Subnet Failover Cluster, the cluster should be automatically configured optimally. This includes setting up the two IP addresses, adding two A records to DNS and setting the registerallprovidersIP to true. However, on the client end you need to tell it that you are connecting to a multi-subnet failover cluster, otherwise the connection won’t be made.

Configuring The Client

Configuring the client is done by adding multisubnetfailover=true to the connection string. This Microsoft documentation is a great resource. But if you just search for multisubnetfailover=true you will find a lot of information about that setting.
Do not that not every application will support adding that to the connection string. If you find yourself in that situation you should ask your application vendor to add support for that or show you how to do it.
Fret not. All is not lost if you find yourself in that situation. You will want to change the behavior of the cluster so that upon failover DNS is update so that the single A record associated with the cluster client access point is updated with the new IP address. This is in lieu of having two A records in DNS, one with each cluster IP address, which is the default behavior in an multi-subnet cluster.

Help Is On The Way

This article reference SharePoint, you can ignore that, the rest of the article is pretty well written to describe the process you should follow.
The highlights of that article are as follows…
Get-ClusterResource “[Network Name]” | Set-ClusterParameter RegisterAllProvidersIP 0
After restarting the cluster-name-object (basically restarting the role) & cleaning up all “A” records manually (clean-up isn’t done automatically) we can see our old A-records are still in DNS so we’ll need to delete those manually.
In addition to those steps I’d advise you to reduce the TTL on the HostRecordTTL as described in this article.
The highlight of that article is as follows.
PS C:\> Get-ClusterResource -Name cluster1FS | Set-ClusterParameter -Name HostRecordTTL -Value 300
With a Value of 300 you could potentially be waiting up to 5 minutes for your clients to reconnect after a failover, or even longer if if have a large Active Directory infrastructure and AD replication takes some time to update all the DNS servers across your infrastructure.
You are going to want to figure out what the optimal TTL is to facilitate quick client reconnections without over burdening your DNS servers with a bunch of DNS Lookup requests.
This type of configuration is common in disaster recovery configurations where your DR site is in a different subnet. It is also very common in HA deployments in AWS because different Availability Zones are in different subnets.
Let me know if you have any questions about SQL Server Multi-Subnet Failover Cluster. You can always reach me on Twitter @daveberm

Can I Put My File Share Witness On A DFS Share?

by Leave a Comment

Can I Put My File Share Witness On A DFS Share?

Can I Put My File Share Witness On A DFS Share?

I get asked this question all the time – Just where can put my File Share Witness On A DFS Share. People are concerned about losing their file share witness. Hence like many of their other shares, they want to leverage DFS for some additional availability. This is a very bad idea and is not supported.

Microsoft recently publish a great blog article that describes exactly why File Share Witness On A DFS Share is not supported.

https://blogs.msdn.microsoft.com/clustering/2018/04/13/failover-cluster-file-share-witness-and-dfs/

Much of this article would also apply to people who ask if they can use a DataKeeper replicated volume resource as a Disk Share. It makes sense. You can use a DataKeeper volume resource in place of a Physical Disk resource for any other workload, so why not a Disk Witness?

This issue is the same as the DFS issue. In the event of a loss of communication between the two servers, there is nothing to guarantee that the volume wouldn’t come online on both servers. It would result in a potential split-brain condition. The Physical Disk resource overcomes this issue by using SCSI reservations. This would ensure the disk is only accessible by one cluster node at a time.

The good news is that Microsoft already blocks you from trying to use a replicated DataKeeper Volume resource. And coming in Windows Server 2019, it looks like they will also block you from using a DFS share as a File Share Witness.

Can I Put My File Share Witness On A DFS Share?
Taken from the Failover Clustering and Network Load Balancing Team Blog Post “Failover Cluster File Share Witness and DFS

Have questions like this about putting File Share Witness On A DFS Share? Read through our blog or contact us!
Reproduced with permission from ClusteringForMereMortals.com

Network Speed Between Azure Regions Connected With Virtual Network Peering

by Leave a Comment

What Is The Network Speed Between Azure Regions Connected With Virtual Network Peering?

This is the question I asked myself today. Of course I couldn’t find the reason behind Network Speed Between Azure Regions Connected With Virtual Network Peering documented anywhere. I’m assuming there is no guarantee. It probably depends on current utilization, etc. If I’m wrong, someone please point me to the documentation that states the available speed. I primarily looked here and here.

So I set up two Windows 2016 D4s v3 instances – one in Central US and one in East US 2. Both are paired regions.

If you don’t know what peering is, it essentially lets you to easily connect two different Azure virtual networks. Peering is very easy to setup. Just make sure you configure it from both Virtual Networks. Once it is configured properly it will look something like this.

Doing Tests

What Is The Network Speed Between Azure Regions Connected With Virtual Network Peering?
A properly functioning peered network in Azure

I then downloaded iPerf3 on each of the servers and began my testing. At first I had some pretty disappointing results.

But then upon doing some research, I found that running multiple threads and increasing the window size reports a more accurate measurement of the available bandwidth. I tried a few different setting. It seemed to max at at just about 1.9 Gbps on average, much better than 45 Mbps!

I used the client parameters and produced the best results. See as follows:

iperf3.exe -c 10.0.3.4 -w32M -P 4 -t 30

A sample of that output looks something like this.

- - - - - - - - - - - - - - - - - - - - - - - - -
 [ 4] 2.00-3.00 sec 34.1 MBytes 286 Mbits/sec
 [ 6] 2.00-3.00 sec 39.2 MBytes 329 Mbits/sec
 [ 8] 2.00-3.00 sec 56.1 MBytes 471 Mbits/sec
 [ 10] 2.00-3.00 sec 73.2 MBytes 615 Mbits/sec
 [SUM] 2.00-3.00 sec 203 MBytes 1.70 Gbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ 4] 3.00-4.00 sec 37.5 MBytes 315 Mbits/sec
 [ 6] 3.00-4.00 sec 19.9 MBytes 167 Mbits/sec
 [ 8] 3.00-4.00 sec 97.0 MBytes 814 Mbits/sec
 [ 10] 3.00-4.00 sec 96.8 MBytes 812 Mbits/sec
 [SUM] 3.00-4.00 sec 251 MBytes 2.11 Gbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ 4] 4.00-5.00 sec 34.6 MBytes 290 Mbits/sec
 [ 6] 4.00-5.00 sec 24.6 MBytes 207 Mbits/sec
 [ 8] 4.00-5.00 sec 70.1 MBytes 588 Mbits/sec
 [ 10] 4.00-5.00 sec 97.8 MBytes 820 Mbits/sec
 [SUM] 4.00-5.00 sec 227 MBytes 1.91 Gbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ 4] 5.00-6.00 sec 34.5 MBytes 289 Mbits/sec
 [ 6] 5.00-6.00 sec 31.9 MBytes 267 Mbits/sec
 [ 8] 5.00-6.00 sec 73.9 MBytes 620 Mbits/sec
 [ 10] 5.00-6.00 sec 86.4 MBytes 724 Mbits/sec
 [SUM] 5.00-6.00 sec 227 MBytes 1.90 Gbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ 4] 6.00-7.00 sec 35.4 MBytes 297 Mbits/sec
 [ 6] 6.00-7.00 sec 32.1 MBytes 269 Mbits/sec
 [ 8] 6.00-7.00 sec 80.9 MBytes 678 Mbits/sec
 [ 10] 6.00-7.00 sec 78.5 MBytes 658 Mbits/sec
 [SUM] 6.00-7.00 sec 227 MBytes 1.90 Gbits/sec

I saw spikes as high as 2.5 Gbps and lows as low as 1.3 Gbps.

Update From Twitter

So I received some feedback from @jvallery that I must try out.

What Is The Network Speed Between Azure Regions Connected With Virtual Network Peering?

First thing I did was bump up my existing instances to D64sv3 and used -P 64. I saw a significant increase

iperf3.exe -c 10.0.3.4 -w32M -P 64 -t 30

[SUM] 0.00-1.00 sec 2.55 GBytes 21.8 Gbits/sec

I then spun up some F72v2 instances as suggested and I saw even better results.

iperf3.exe -c 10.0.2.5 -w32M -P 72 -t 30

[SUM] 0.00-1.00 sec 2.86 GBytes 24.5 Gbits/sec

 

I’m not well versed enough in Linux. Bu there seems to be a reasonable amount of bandwidth available between Azure regions when using peered networks.

If someone wanted to repeat this test using Linux as @jvallery suggested, I’ll be glad to post your results here! Seems that there is indeed to possible to vary the Network Speed Between Azure Regions Connected With Virtual Network Peering.

Using SIOS DataKeeper For Disaster Recovery

For one of my clients, I chose to use these two peered networks to address SQL Server disaster recovery using SIOS DataKeeper to asynchronously replicate SQL data between regions for disaster recovery.

What Is The Network Speed Between Azure Regions Connected With Virtual Network Peering?
SIOS DataKeeper replicating data from Azure EAST US 2 to CENTRAL US

In this particular scenario, we were measuring a RPO measured in milliseconds. As you’ll see in the video below, during a DISKSPD test meant to simulate a typical SQL Server workload the RPO was <1 second.

I’d love to hear from you regarding your experience regarding any network speed you measure in Azure and how you are using peered networks in Azure.

Have questions about Network Speed Between Azure Regions Connected With Virtual Network Peering? Read through our blog or contact us!
Reproduced with permission from ClusteringForMereMortals.com

Convert Azure Clusters To Managed Disks

by Leave a Comment

Why you should convert azure clusters to managed disks

Why You Should Convert Azure Clusters To Managed Disks

You may have heard about the recent storage outage that impacted some instances in the US East region back on March 16th. A root cause analysis of the outage is posted here. March 16th US East Storage Outage

Customer Impact

A subset of customers using Storage in the East US region may have experienced errors and timeouts while accessing their storage account in a single Storage scale unit.

You might be asking, “What is a single Storage scale unit”. Well, you can think of it as a single storage cluster, or single SAN, or however you want to think about it. I don’t think Azure publishes their exact infrastructure. Although you can probably assume that behind the scenes they are using Scale Out File Servers for backend storage.

Survive The Outage With Minimal Downtime

So the question is, how could I have survived this outage with minimal downtime? If you read further down that root cause analysis you come across this little nugget.

Virtual Machines using Managed Disks in an Availability Set would have maintained availability during this incident.

Hence, it is time to Convert Azure Clusters To Managed Disks

What’s Managed Disks?

On February 8th Corey Sanders announced the GA of Managed Disks.

Managed Disks would have helped in this outage. Because by leveraging an Availability Set combined with Managed Disks, each of the instances in your Availability Set are connected to a different “Storage scale unit”. So in this particular case, only one of the cluster nodes would have failed, leaving the remaining nodes to take over the workload.

Prior to Managed Disks being available (anything deployed before 2/8/2016), there was no way to ensure that the storage attached to your servers resided on different Storage scale units. Sure, you could use different storage accounts for each instances. But in reality that did not guarantee that those Storage Accounts provisioned storage on different Storage scale units. More reasons to Convert Azure Clusters To Managed Disks.

So while an Availability Set ensured that your instances reside in different Fault Domains and Update Domains to ensure the availability of the instance itself, the additional storage attached to each instance really represented a single point of failure. Although the storage itself is highly resilient, with three copies of your data and geo-redundant options available, in this case with a power failure the entire Storage scale unit went down along with all the servers attached to it.

So long story short… Convert Azure Clusters To Managed Disks as soon as possible in order to help minimize downtime

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-migrate-to-managed-disks

And if you really want to minimize downtime you should consider Hybrid Cloud Deployments that span cloud providers or on-prem to cloud!

Reproduced with permission from Clusteringformeremortals.com

Cloud Witness To Build Multi-Instance SQL Server Failover Cluster In Azure

by Leave a Comment

New Azure ILB Feature Allows You To Build A Multi-Instance SQL Server Failover Cluster In Azure

New Azure ILB Feature Allows You To Build A Multi-Instance SQL Server Failover Cluster In Azure

The new feature, Cloud Witness is my favourite at the moment. Before we look at the new quorum features in Windows Server 2016, I think it is important to know where we came from. In my previous post Understanding the Windows Server Failover Cluster Quorum in Windows Server 2012 R2 I went into some great detail regarding the history and evolution of the cluster quorum. I suggest you review that post to understand how the quorum works in Windows Server 2012 R2. Also, how the new features of Windows Server 2016 are going to make your cluster deployments even more resilient.

Cloud Witness

A Cloud Witness allows you to leverage Azure Blob Storage to act as a witness for your cluster. This witness would be in place of a Disk Witness or File Share Witness. The configuration of a Cloud Witness is extremely easy. From my experience costs next to nothing to host in Azure. The only downside is that the cluster nodes will need to be able to communicate over the internet to with your Azure Blob Storage. Very often cluster nodes are forbidden to communicate over to the public internet. So you will need to coordinate with your security team if you want to enable a Cloud Witness.

There are many compelling reasons for using a Cloud Witness to build the Multi-Instance SQL Server Failover Cluster In Azure. But for me it makes most sense in three very specific environments: Failover Cluster in Azure, Branch Office Clusters, and Multisite Clusters.

On A Closer Look

Let’s take a look at each of these scenarios to see how a Cloud Witness can help.

New ILB Feature For Multi-Instance SQL Server Failover Cluster In Azure
Figure 1 – When you’re trying to build Multi-Instance SQL Server Failover Cluster In Azure, the cloud witness storage account should always be configured Locally Redundant Storage (LRS)

Highly Available Deployments

If you are moving to Azure (or really any cloud provider), you will want to make sure your deployments are highly available. If you are taking about SQL Server, File Servers, SAP or other workloads traditionally clustered with Windows Server Failover Clustering, you will need to use either a File Share Witness or a Cloud Witness, since a Disk Witness is not possible in Azure. With Windows Server 2012 R2 or Windows Server 2008 R2, you will need to use a File Share Witness. Windows Server 2016 makes it possible to use a Cloud Witness. The advantage of a Cloud Witness is that you don’t have to maintain another Windows instance in Azure to host the File Share. Instead, Microsoft allows you to leverage Blob Storage.  This gives you a less expensive solution, one that is much easier to manage, and more resilient.

Location

When looking at cluster deployments in branch offices, cost and maintenance is always a consideration. For a retail chain with hundreds or thousands of locations, having a SAN in each location can be cost prohibitive. Each location might to run a two node Hyper-V cluster on a S2D Hyper-converged configuration or a 3rd party replication solution to host a number of virtual machines. Now what a Cloud Witness can do is to help the business avoid the cost of adding an additional physical server in each location to act as a File Share Witness or the cost of adding a SAN to each location.

Eliminates The Need For A 3rd Data Center

And finally, when deploying a multisite cluster, the Cloud Witness eliminates the need for a 3rd data center to host the File Share Witness. Before the introduction of the Cloud Witness, best practice would dictate that the File Share Witness reside in a 3rd location. Access to a 3rd datacenter just to host a file share witness was not always feasibly and certainly introduced another layer of complexity. By using a Cloud Witness you eliminate the need to maintain a 3rd location and access to the witness is done over the public internet, minimizing the network requirements as well.

Site Awareness

When building a multisite cluster, there has always been another common problem. Controlling the failover to always prefer the local site was not possible. While you could specify Preferred Owners, the Preferred Owners setting is commonly misunderstood. Administrators may not have realized this. But do you know even if they didn’t list a server as Preferred Owner, the server is automatically appended to the end of the Preferred Owners list maintained by the cluster. The result of this misunderstanding is that although you may have only listed the local servers as Preferred Owners, you could potentially have a cluster resource failover to the DR site. And this is even when there is a perfectly good node available in the local site. Obviously this is not what you expect and using Site Awareness will eliminate this problem moving forward.

Site Awareness fixes this problem by always preferring the local site when deciding which node to bring online. So in a normal circumstance a clustered workload will always failover to a local node unless you have a complete site outage. In which case one of the DR nodes will come online. The same holds true once you are running in the DR site. The cluster will recover the workload on a server in the DR site if it was previously running on a node in the DR site. Site Awareness will always prefer a local node.

Fault Domains

Building upon site awareness is Fault Domains. Fault Domains goes a step further and lets you define Node, Chasse, and Rack locations in addition to Site. Fault Domains have three benefits: Storage Affinity in a Stretch Cluster, increases Storage Spaces resiliency. It enhances the Health Services alerts by including meta data about the location of the associated resources raising the alarm. Storage Affinity will help ensure that your cluster workloads and storage are running in the same location. You certainly wouldn’t want your VM reading and writing data that is sitting on a CSV in a different city.

However, I think the biggest winner here is the Storage Spaces Direct (S2D) scenario. SD2 will leverage the information you provide about your cluster nodes location (Site, Rack, Chassis) to ensure that the multiple copies of data that is written for redundancy all live in different Fault Domains. This helps ensure that data placement is optimized so that the failure of a single Node, Chassis, Rack or Site does not bring down your entire S2D deployment.  Cosmos Darwin has an excellent video on Channel 9 that explains this concept in great detail.

Summary

Windows Server 2016 adds several new enhancements to the cluster quorum that will provide some immediate benefits to your cluster deployments. In addition, check out some of the other great new cluster enhancements like rolling system upgrade, Virtual Machine Resiliency, Workgroup and Multi-Domain Clusters and others.

To read about other tips such as building a new Multi-Instance SQL Server Failover Cluster In Azure with Cloud Witness, have a read at our posts.

Reproduced with permission from Clusteringformeremortals.com