Uh oh.. I was recently contacted with an ‘SOS’ for a severe case of AX 7 blues. In this case, everything was extremely slow. Let’s walk through the issue and how to fix it.
Case: A new AX7 implementation reported general slowness all around, not just in one form but multiple forms. Also, the slowness wasn’t limited to a particular area like financials or supply chain. It was everywhere. The company was small, less than 100 employees. The database also contained a small amount of data.
WHAT COULD IT BE: The Company is small so that rules out high volume issues unless they have a huge amount of integrations They are also a new company so that rules out an overly large database, and it usually eliminates a good percentage of detectable code performance issues (that comes when an implementation gets further along).
ANSWER: Hardware. The client had made the mistake of purchasing AX 7 without having gone through the hardware options within Azure. They just assumed that because they were in the cloud, this part would be ignored. You had a pretty strong hint that it was hardware because the implementation was slow from almost the very beginning – that is usually outside configurations or hardware.
To discover the problem, I pulled out my free IOMeter to measure hard disk performance and ran a test on the Azure machine.
Azure comes with a base speed expectation of up to 500 IOPS per a second. In reality, this means little without the latency and can be considered slow. So, I ran a read-only ideal scenario to see how much speed we could expect. The results are posted here. While we did beat the 500 IOPS in an idealized scenario, our response time was still problematic. We would like to see under 10 for a server hosting a database. And idealized scenarios are often very unrealistic – hence the notorious 1 million IOPS advertisements frequently being used today.
Thankfully, IO meter is awesome and comes with real-world work load settings with mixtures of read and writes. So, I ran a more realistic scenario. Now, watch what happens to the latency and response time.
Oh no.. Microsoft advertises the base virtual machines as being up to 500 IOPS per a second which is still slow. But, notice how we are only getting 186.36 IOPS??? In other words, the drive is so slow that no amount of super programming, voodoo, or whatever will make this implementation run faster. They simply need to upgrade the hardware. We strive for an average response time of 10ms or less but we are at 309.0445. We then ran a comparison against the client’s on premise hardware and found that the speeds were 5 times faster than what we were using in Azure. As I spoke to the IT director, he just choked up as he reviewed the results. The hardware and network planning for a cloud solution had not come up because he had been told (by several sales people) that he would never have to worry about scaling hardware or making decisions. This isn’t something that Microsoft advertises, but it’s become a common misconception. You need to do an extensive amount of hardware planning when migrating to the cloud.
THE SOLUTION: Know thy Hardware and sign up for Premium Storage. To begin the fix, I pointed the client out to the premium storage options of Azure. Since no one had bothered to take the time to explain the throttling behavior of Azure and how to increase disk speed, I had to start. We discussed packages, redundancy, and increasing IOPS. As a test, I had the client download the offline development machine to their on premise environment with faster storage where they noticed a rapid increase in speed. The client is now undergoing budget approval for higher performance Azure packages. After the disk, we’ll also be addressing the layout of the data centers, redundancy strategy, endpoints, maintenance schedule, network throttling, and other options that need to be there for an implementation.
Lesson learned, do not just blindly accept a hardware configuration for your Azure AX Virtual Machines. Know the packages. There are several ways to configure the premium disk speed for optimal performance. No one configuration will fit all environments.
While this case didn’t have the instant happy ending – the solution was ultimately to spend more money on a better storage configuration – it was a case that underscores the importance of architecture planning when going to the cloud. We have an obligation to know the configurations in the cloud which will meet our needs. And make sure you sign up for premium storage while including it as part of your budget when going to AX 7. Till next time..