It was my fifth implementation. Everything looked so innocent. I had been brought in to fix a performance issue. Little did I know that I was about to walk into the hardest troubleshooting case that I had encountered (at least at that time) in my AX career. Since then, I’ve faced the scenario countless times but have never seen it written about so hopefully my story saves someone out there from having a slow implementation if they are searching for the answer and this is the cause. It is actually a very common problem, I find. And people are always shocked when I point it out on an implementation.
CASE: A client reported that their AX System was slow. Yet, none of the performance counters correlated with this information. Memory, Processor, Hard Disk, Network Utilization were all fine. Volume was extremely low. Only 22 users were on the system. Code customization was minimal. Yet, AX was extremely slow. The problem was baffling to everyone. Go-Live was approaching and the company was seriously considering calling off their entire AX implementation over this (and firing the consulting firm along with all the executives who brought in the consulting firm). The pressure was on.
Can you guess what the problem is? Code was written to best practices as it was Microsoft’s default code, pretty much. The application was slow everywhere but none of the performance metrics correlated with what we were experiencing.
For one week, I looked everywhere.. desperately.. I couldn’t figure out what was wrong.
And then one day, I brought up process monitor. (note: always use process monitor when you can’t figure out what is happening. And little did I know, I noticed that the AX file directories were blocked. The security software was actually blocking the AX from generating CIL. I had the security software temporarily turned off as a test, and performance increased significantly. Suddenly, where it took 6 minutes to load AX after an AOS restart, it now took 45 seconds. All of the users began to report that performance was good after the antimalware was turned off. I was surprised. Quickly, I found every single Dynamics AX folder that was being locked and impacted by security software and had it added as an exclusion.
The term to watch out for here is File System Filter Drivers. These are drivers that modify the behavior of some of the built-in system drivers such as loading information to a disk. Antivirus and antimalware software typically install these sorts of drivers to scan incoming and outgoing files on the disk. The scan operation can cause significant delays in response times because the application has to first stop the data in transit to scan it before adding or removing or reading(with some filters) it from the disk. This is why it is so devastating to not exclude directories like the AX bin directories and SQL Server directories and other directories involved in performance. At the very least, I’ve seen File System Drivers cause a 25% hit in AX performance when AX directories are not excluded. At worst, I’ve seen the anti-malware applications cause AX to be completely unstable.
In order to detect this, here is the powershell that you need to run:
1st, we run the fltmc application. Notice that I have 4 file System Drivers on my personal machine. Now, what I’m looking for is the antimalware applications in particular though I have seen other file system filter drivers cause problems – ones related to encryption and compression.
2nd, To figure out the name of our application, so that we can trace it in procmon and make sure that it isn’t blocking anything with our AX application, we run this powershell command. Notice that the filter name goes in the quotations.
Gwmi –Query ‘SELECT * FROM Win32_SystemDriver WHERE Name=”luafv”
Finally, open up Process Monitor from the SysInternals. Put a filter so you can only see the antimalware application and watch it. Look for AX directories, SQL Server directories, and user cache directories used by AX that it is locking. Note them down and lock them. Here is an example in Process Monitor. AX is not installed on my home laptop (it is on my 30 virtual machines though), but you can see what I’m looking at:
If you just understand how to spot a case of a filter severely impacting AX performance, you will be able to often make significant performance impacts fast. I use it on every implementation where I serve as the lead Architect. It never fails me. Hope my experience served someone well out there. Till next time.