A company that had been using and upgrading Dynamics for a long time, since version 4.0 suddenly found themselves in the midst of upgrade slowness. See, from a performance perspective, upgrades often present very special circumstances which will ordinarily lead to slowness. Believe it or not, this isn’t too unusual and can normally be fixed pretty fast once you get used to seeing the same issues present themselves again and again. However, there are a certain subset of performance issues which take on a whole different meaning and get really complicated. Those are hard as hell to solve and will even give very experienced performance tuners a run for their money. Let’s take an example of this now.

An experienced Dynamics AX implementation with highly experienced developers who had been using Dynamics forever suddenly experienced extreme slowness after carrying out an upgrade. The case was baffling with multiple large partners struggling and Microsoft support tickets that had went unresolved. In this case, one of the developers had stumbled upon my blog. Hence, the story begins..

It definitely wasn’t the SQL

On these more complicated cases, it seldom is the SQL. This proved to be the case here. Notice that the SQL time is zero, but something really strange is happening. Notice the run times for the Customer form taking 5 and a half minutes to load. Ouuuuch! Even more interesting, there is no sql time meaning that all of our database analysis tools are pretty useless here. The difficulty of this case just went up by a factor of 10. Hence why it the case had slipped by so many advanced people.

Hmmm.. It is on the AOS Tier but where?

So, I knew it was on the AOS tier but where.. I got a hint, the custtable issue mainly arose when a regular user tries to log in. Someone in the SysAdmin role never experiences this behavior. This was a dead giveaway for something related to permissions.

A further analysis showed that the behavior was caused by reference controls. Reference controls (the ones used to do a lookup like seeing all customer names and letting someone pick from them) were the cause. While we can see numerous hangs on the reference controls, the behavior completely goes away if someone has the custtablemaintain permission.

Getting even further, notice how the redraw event is the real cause of the behavior. Somehow, the form is not able to properly lock for a redraw on reference controls unless the maintain permission is there. But when the maintain permission is there, everything just works. Notice the time.

Ah ha, let’s just add a permission on the problematic controls since nothing else worked

Basically, I did a standard check. There was no record level security, no XDS, no additional security configurations. The behavior simply applies to an out of the box upgrade when someone tries to access a reference control with controls equivalent to read-only. No code or server permissons are attached which cause this behavior. I even tried adding the controls directly to the form security (old school) and that didn’t work. Finally, it became pretty clear that this was EMBEDDED in the kernel.

Done, I explained what the options where to the client.

This was a pretty advanced issue. Basically, I explained what the options where to the client. They could choose XDS, but there was a disadvantage in that this query would attract to anything that wanted the custtable form degrading performance across the application. XDS does not work well when thrown on heavily used tables like custtable – too many processes use it and it tanks the performance when this happens. It’s better on small, customized tables. And unfortunately, a read-only version of the custtable form was incompatible. So the client had two options in this case:

1. Microsoft ticket to fix the read only problem on reference controls preventing a redraw from working well
2. Create a duplicate of CustTable form, remove the reference controls on the customized version, add logic when someone clicks on the menu item to choose which form loads based on a security role.

The customer opted to go for a Microsoft ticket first as that carries no cost with the partner of record. This is often normal. In the performance tuning situation where a client has spent a lot of money, they’ve often already paid multiple people to solve the issue that you’ve been brought in to solve. Sometimes, just finding the answer is enough for those who’ve already been paid for the issue to solve it from there. A lot of clients elect for that option to bring it back to those who’ve already been paid and let them finish. Case closed on my end.

In Summary, this was a perfect case of some advanced issue resolution

The advanced issues are always the interesting ones. The ones at the kernel offer a very difficult analysis because you must use a lot of inference and experience to get to the answer. The answer wasn’t directly there in this trace, but by looking at a number of issues combined with trial and elimination allowed us to arrive at the correct result. Sometimes, it is like that. Upgrade performance issues are usually pretty easy to resolve, but there are times when it can get tricky. Hope you enjoyed this one and stay tuned for the next post!

Note: I’d like to thank an incredibly advanced development shop and wonderful implementation. You all were phenomenal to work with. I was honored that such a high quality development shop could recall cases of finding value in my blog. This post is for you. J