Just finished one of those marathon weekends that every person whose job involves Support dreads.. You know, the one where you cancel all your great weekend plans because something unexpected and bad happened. Such was the case when a client of mine innocently downloaded the new version of the tax engine, tested it, and then deployed it to production. I write this post in hopes that anyone who has gone through the same thing – especially with it being the tax engine and the time of year when most implementations are updating it. Most hotfixes that Microsoft introduces are stable and usually a good thing, but there are exceptions from time to time. This is software after all.

THE SYMPTOMS:

Around Thursday, one of my clients called me up and told me that they had run into a real strange error. A team of developers were working on it and having a lot of trouble on the diagnosis. I was working on something else for the client but they asked me if I could take a look at it on Friday if it wasn’t solved. They then read the error to me, which went like this.

“ErrorTextCannot create a record in Tax code parameter value (PayrollWorkerTaxCodeParameterValue). Insert not supported with the values specified for ‘Effective’ and ‘Expiration’. New record overlaps with multiple existing records.”

Simple enough, I thought… It sounded like just your run of the mill ValidTimeState error with a record violating the “nogap” condition. See, here is what usually happens when you see ValidTimeState conditions.

Somebody has an existing record in a table. Then they try to insert a row with a ValidFrom date earlier than an existing row in a table and a validto with an equal expiration – usually the maximum value for date in a table. The logic correctly fails because the entire purpose of valid time state is to ensure that rows have continuity of times. There is a setting called “no gap” which is supposed to enforce the contiguous behavior.

The thing that usually gets people into trouble is that they don’t understand just what “no gap” means. Clients are given a choice of whether to enforce the contiguous time behavior at the date level or at the more granular seconds (what is actually dependable – not the microseconds as it looks) level. The inherent assumption is that you use date when updates are frequent and are spread apart. You use utcdatetime on the property settings when you expect to have many frequent updates – brings up entire other range of issues. Then, you plan the heck out of it because ValidTimeState because it will slow down your transactions – it forces row by row transactions so you want to be careful where you use it.

There are a series of rules which handle the auto-incrementing behavior for you, which is a nice luxury for keeping your time periods aligned with data. None of them are hard to manually code but it is still a nice luxury.

So, it was a seemingly straight forward Issue

It didn’t look so bad. Just find the problematic record with the noncontiguous dates acting badly, and correct it by probably going back to the data entry and possibly adding a validation to handle the condition in the future or user education.. and Wallah.. Weekend should be saved. The client reported that an integration was broken also. So, I would need to debug through Visual Studio – that is a pain in the tail for R2 and R3 because you sometimes have to recompile the CIL between changes making your iteration cycles take longer.

And now it gets Funky…..

So, I started debugging and I was surprised to see records that should have gone through. The validfrom and ValidTo’s were just fine. Now, watch what happened, and you can see from the screenshot that this was right in the Tax Engine. The code was no longer picking up null values. Something got in the Kernel that messed this up and caused the “is null” checks to fail for certain parts of the tax engine. You can see below that taxcodelist is null. Yet, it still moves to the next line of code and executes the statement. Downstream, this manifested itself by trying to insert blank records into the validtimestate tables.

OH HELL, A TRUE KERNEL ERROR IN PRODUCTION!!!!!!!!!!!!!!!!

Daaaarn.. a Kernel Error in Production.. and it broke a whole lot of stuff for this client. There goes the weekend. Now, there are two concerns – handle this error plus any data integrity damage to existing tables. And the condition is happening on every TimeState table integrated with that tax engine. Furthermore, while very senior AX troubleshooters can detect Kernel errors, that code is locked down. You have to submit an emergency ticket to Microsoft and it can take days or months to fix. When a client is down, this isn’t always an option.

Light at the end of the Tunnel – QUIETLY Microsoft Released another version of the Tax Engine One Week Later:

My client and I were discussing it and he went back to LifeCycle Services and noticed that there was a completely new version of the Tax Engine released just one week later. Lol…….. That was a close one. Usually, tax engine versions are released quarterly. But on occasion, we’ve all seen “no comment” versions of the certain hotfixes released immediately afterwards with new versions or even the same version but different. This looks like one of those cases. In this case, we didn’t have a choice. We immediately had to test the new version of the Tax engine and get things working for an emergency deployment to production to happen as quickly as possible. .

WHAT WERE THE RESULTS:

In the new version of the tax engine, the funky error went away and duplicate error behavior stopped. But the TimeState tables still didn’t follow their rules as before. Looking at the table, you could see where some of the values were out of wack. So, I wrote up a code fix to handle the gap conditions and got things going again. I also wrote the time state rules into a method that I attached to all inserts on the table as a safeguard for now to allow for production operations to resume normally, but give us more time to test the hotfix. Your primary concern is to always get the client up and running – even with a band aid.

WHAT DID WE ALL LEARN?

Make sure you double check and get the emergency version of the tax engine that was released a week later. Microsoft is an excellent company but they are human. Everybody makes mistakes. It seems like they caught the bad version of the tax engine pretty quickly, but sending a warning out to customers who downloaded it might have helped. Just make sure you get the latest version of the tax engine or if you are experiencing the same problem, download the version that came out a week later.