In our continuing mission to get the real-world upgrade experience, let’s resolve close to 5000 AX2012 to Dynamics 365 code upgrade errors for a billion dollar, multinational company in one blog post.  Note, the company will remain extremely anonymous as always, but this really happened.  While we are at it, let’s learn how setup our projects up for speed, some reporting, be able to give back estimated hours, fix more unexpected lifecycle services broken functionality, and learn of some real-world scenarios that will affect your code progress. All the emails and comments may have got me too excited.  We took it really far on this one, but I’m excited about all the people who’ve written me of late, and I deeply appreciate the positive encouragement.   Get ready because I don’t think that I’ve ever read an upgrade post like this, and I read a lot of blogs!   Let’s get to it.

1st, gain an understanding of the upgrade tasks and the amount of work that Dynamics has estimated as part of the upgrade:

This is the work item view. It’s good in that it shows you the amount of work and what tasks need to be done.

Let’s make the query a big more useful. I know all about points and love agile definitions. But the issue we have in Dynamics is that customers are often billed by hours. So, they will often ask for a set number of hours on an estimate. Let’s configure Visual Studio online to give us that.

Click Work à Queries à New Query

Then click the column options button

Then, choose the AreaID and the RemainingWork. The AreaID isn’t unique and needs to be the same for all columns in this case, so you can see the Remaining work for the entire upgrade project. The chart tool requires it. Be sure to click OK when done.

Save your query and give it whatever name you like and then click on Charts:

Configure Visual Studio Team Foundation Services by changes the values to the following. This is the goofy workaround way to be able to sum the estimate in hours. Notice how I changed aggregation to Sum, selected Remaining Work as the column, and grouped it by AREA ID. Since there was only one AreaID, I got a total sum.

Okay, so it’s really 738.15 hours for an upgrade.

Now, just add the tasks to your dashboard and you are in business. You can track the remaining work for the course of your upgrade project if you choose to go through the task list. My only problem is that the task list adds way too much overhead to coding in my opinion. It’s like reading a novel and just adds too much time. So, I’ll show you the method that I use to go through the upgrade in this post.

Second, gain an understanding of the upgrade intentions

First, you should have Source Control Explorer Open (ViewàOther Windows à Source Control Explorer) in case it isn’t open.

When you browse through the 3 project types that the upgrade creates for you, it will look confusing as hell. This has led to a very popular tactic of just ignoring these projects altogether. But hang on… at least let me explain “the method behind the madness” before you just create a model that references everything, create your own projects, and only use that model and those projects.

Since these aren’t defined anywhere, I’ll go ahead and give them my definitions:

This almost makes perfect sense except that you will notice something. These components of the projects are not mutually exclusive. For example, if a form contains a method that hasn’t been touched but was upgraded and another method that needs some code merging, it will appear in both projects as of today. That leads to confusion. In the old way of merging, we would just take care of everything right there and only had to worry about an element needing work being located in one place.

Okay, so knowing what you know now, which solution would you open first? I’d personally start with CodeMergeSolution. I want to see which custom code needs some helping human decision-making to make it compatible. Some very good Technical Architects disagree with me and strongly prefer the dump everything into one model approach over using these projects. I’m not saying that I think they are wrong, it’s just that Microsoft’s code project organization approach makes sense to me here… personal preference.

Third, so right click on Code Merge Solution and select View to open it.

If it gives you a drama message about not being configured for integrated source control but configured for source control, it just means that someone opened up the source control in visual studio while it was offline at some point. Simply choose the bind all option when the dialog pops up or you can go through the file menu to stop the irritating message. I won’t talk about that here much because plenty of excellent bloggers have already explained and demonstrated how to resolve it in depth on their blogs if you search for it.

Okay,, now it’s time to start merging.

Disclaimer: I have to be very careful. There really was no practical way that I could reproduce the real upgrade experience from demo bait. I needed something real, like a form that had undergone 5 years of code development. And we also need a real environment, like this one that has over 30 ISV’s and tons of modifications.

Fourth, go to your Visual Studio Team System Link and look up something that needs to be upgraded

Here I did the SalesTable form because it’s such a good example. It has all kinds of conflicts – in all kinds of layers and even amongst multiple ISV’s in this environment. Fun.. Fun.. fun..

Wait… 38 hours to upgrade a form if you completely convert it to extensions?????????????????????????? Hmmmm.. Well, estimators are usually off pretty badly. Anyway, let’s just see.

Now, at the top of your solution, type in “SalesTable” to filter it. Notice that we got a lot of work to do. Notice, the “c” right next to the name for the SalesTable form. That means that we have a conflict. Okay, so that’s familiar. Let’s go into a little more detail.

Fifth, Open the SalesTable form to get the “designer” view and see the conflicts

So, just go to the salestable.

Sixth, it is “about to get hot now” as we are finally there. Look at that beautiful SalesTable form waiting for us to upgrade it!

Lots of information here. The form pattern and the form designer are displaying and notice how you can see the conflicts here. It’s looks like we are almost ready to get our development going, but not quite yet.

Something is still missing. How can we logically go through a list of tasks without having to open up TFS workspaces and read through a bunch of verbiage. I appreciate the work item tasks, but going back and forth with the work items, and reading through all that stuff for each and every task is just too much. We are trying to do an upgrade not reading through an encyclopedia.

Seventh, my preferred way of progressing through the upgrade and getting a more trustworthy estimate

I’ve seen a number of approaches here but this is how I like to do it, I don’t trust the code upgrade estimator. What I trust the most is the compiler – over the years that has been the single biggest indicator of upgrade work for me. I’m not saying that it is perfect. Every person who has done a code move before knows that there are many ways to make something compile and yet be completely wrong. But compiling represents “Point A” for me. It’s a milestone because now you can do your data upgrade, start your real testing, and do your real compile. Then, I typically take the warnings next and fix those but progress seems to really start once it is compiling and the testers can get started with real data after the data upgrade. Sometimes, you may be the tester and you need real data.

So, what I do first is just compile the solution:

Now, what that does is generate an error list – in other words what is keeping the solution from compiling. So, first notice how I have a much more realistic idea of what I need to work on. In this project, I have 6,923 errors to address and 5,947 warnings – Ouch.. The estimator showed 845 conflicts needing to be addressed. This is a heck of a lot more work. But wait.. We are human beings and the estimator isn’t. Let’s take a closer look at what is going on.

Eighth, another time saving tip on how to attack the download: the filter is your friend.

My personal Plan of attack for an upgrade is to take on project by project until I finish one of the big 3 major solutions (CodeMerge, Unparasable, etc) than I take on the next solution and do the same strategy of project by project, ect. So, for example, you see here where I clicked, which highlighted the ApplicationSuite.CusModel project to begin with here by just left clicking on it in Solution Explorer. Now, I can either build the solution by itself or go through each element listed in the project

Now, what I do is I take a starting element and type it’s name to the far right of the project in the filter. I set the filter equal to “current project” and I type the name of the element that I am fixing to the right so I can knock out the errors one by one on that element.

Ninth, double-click on the first Error to get started:

So, I double-click on the first error, and it takes me right to the bad code in Visual Studio. It’s telling me that no table, class, or element exists called SalesTable except as a human I know that nothing is wrong with this code. It gives a message saying that the Salestable can not be found, but it is right there. This is a very common upgrade error in D365. You will want to watch what happens next very closely as this doesn’t just happen in AX2012 to D365 Upgrade projects. It happens in regular upgrades and even some code moves as well.

What it tells us is that the references are messed up. That’s strange. Why is this project not seeing the salestable? There is no documentation detailing what to do when D365 doesn’t see a code component that you know is there. Oooouch.. Microsoft guidance quickly says that you will need to create a ticket with premier support if you run into one of these undocumented problems as long as you are a customer who has purchased a premier support plan. But I think there is a better solution than buying a premier support contract to try out the upgrade. So, let’s go further.

Experimenting, I went to each project and tried to build the project seperately. Yes, I got plenty of errors but one project in particular gave me a strange message that differed from all the others. It was a project called “Cus Model”. I right-clicked on the Cus Model, and hit reload the project, and got another error. Uh oh, more problems with the code upgrade tool.

And here you see the error below. This is the kind of stuff that hurts because it results in a developer wasting hours trying to figure out how to get a configuration that doesn’t work, working. It also undermines confidence in Microsoft’s testing procedures. Anyway, to save time, let me show you how to fix it.

Tenth, first let’s fix the broken LCS Code Upgrade tool again with a CUS Model reference this time

Let’s look inside of our packages store first. Notice that a model with the name of “Cus Model” was never created. That’s why it is complaining.

Hmmmm.. So, it means that a model with the name of Cus Model doesn’t exist. What you will notice is that there are models like “Application Suite Cus Model” and “Application Foundation Cus Model”, etc. So basically, the cus layer got distributed into multiple models.

Thus, we learn a very critical lesson.

In past versions of AX, the layer was everything with many places confining development to various layers. With the new version of AX, the layer is almost meaningless. Your customization strategy is strictly dependent on which models house the elements that your ax layer contains and what Microsoft allows you to do with them. The answer, varies, depending upon the model (Application Platform, Application Suite, etc) and which customizations Microsoft allows for it.

So, we get that our upgrade tool got broken.

Now read very carefully on what I do here. This is strictly due to the fact that overlayering isn’t allowed.

So, I go to the project, and right click on it and click “Edit CusModel.rnproj”.

What I’m doing here is a cost versus benefit analysis. If there are a significant number of elements like this, I will go back to the 2012 AX Modelstore, create a model, and transfer those elements in that model. I’ll then rerun the lifecycle services tool Code Upgrade tool until it gets it right. I have no choice. Something tells me this bug is messing up the upgrade, and it probably has to do with the “Cus Model” being a layer generated model (though I can’t say for sure without seeing Microsoft’s code). Anyway before going through the painful process of generating another LCS project by editing the AX 2012 modelstore and resubmitting, I’ll see if it’s faster for me to just fix the issue manually. If it is a small number of elements, I’ll just put a TFS note to manually recreate them myself and delete the project.

Thankfully, it turned out to be just one security role in this case. That’s an easy one to solve. I can just manually put the security role in at the end by recoding it. That was a close one!

To make sure that this isn’t forgotten, I click View à Team Explorer à and then click on Work Items à New Work Item. I choose the task option though I’ve created some sort of meaningful classification scheme that involves some sort of agreement between the project manager and I so that management gets some real-time reports that satisfy their financial reporting, cost tracking, and performance objectives for the code move. Cost tracking is very critical during an upgrade, so configuring TFS for it at the beginning is essential. Plus, it’s one of those things. It’s so much easier to do when the templates are put in at the beginning rather than later.

As you see here, I then add a note on the work item and save it so that no one forgets about it, making sure to manually include some notes. Yes, this was caused by a bug in the LCS Code Upgrade tool, but there are components of every upgrade that you will have to manually rewrite.

Finally, I go back to the solution and delete the project. Some TFS policies don’t allow deletes, so you will have to do whatever is accepted in your organization’s configuration. Also removing the solution may not remove it from TFS depending on how your source control is configured and if the solution isn’t checked out.

If removing doesn’t take the solution out of source control, manually go to Source Control Explorer, right-click on Cus Model and hit delete.

All done for how to handle a bad project. You could either create a new one and make sure that it is in the packages directory, go back to AX 2012 and create a new model and move the elements over there and then run lifecycle services, or just delete the bad project and manually code the functionality in.

Next up… watch what happens next as we fix the bad references to the salestable.

Eleventh, sometimes it really is that simple as you will see here

So, I wondered. Was it the bad LCS generated project that was really the cause for the bad references. See, in the new framework, cross-reference (how code finds other objects) has changed, and the cross-reference is a lot more important than it was previously (and it was important previously but the behavior has changed in this version). You could successfully compile, for example, without a cross-reference unless your code explicitly used the cross-reference. In the new framework, your compile will be messed up without a valid cross-reference and the cross-reference takes place on the code build. So, if your code can’t build enough or skips cross-reference updates, you will have what looks like a reference problem with stuff complaining about other stuff not being there.

Was it somehow doing the impossible and blocking the other components from compiling enough to get access to the cross-reference data that they needed for the compiler to function correctly? If so, then my theory is that if I recompile the project with the references, we should see the problem go away because the compile is not really being blocked. Even though the projects can’t fully compile, if I can get enough of the project to compile, it should be able to update the cross-references enough to proceed with development.

Putting it to the test… I started rebuilding the projects in the solution.

I went to the Dynamics 365 menu  à Build Models

I selected model by model, not all of them at once. So, I ran the build for each model by checking it.

See notice how I only ran one model at a time:

And notice here how I ensured the cross reference was on:

Twelve, See the results of fixing the wacky Lifecycle Services Code

And check it out. We went down to 1003 errors. 800 of the errors could simply be fixed if the ISV’s had updated models.

Interesting. So this client was on 2012R3 and on a recent CU version. The more recent the CU version the more amenable the upgrade. What this basically means is that this multi-national, huge organization is close to getting the first initial code compile and data upgrade done within 3 weeks. It’s amazing what a human eye can do. By fixing the Microsoft generated code, we radically cut down on the time for this upgrade. Yes, I still have to show how to resolve conflicts, we need to go through all the warnings that are todos also, and testing is always time intensive, but a really positive start as opposed to the first fake initial compile with thousands of errors or going through and reading a gigantic task list.

HALLELUJAH!!!!!!!!!!!!

So, I shocked my client. I told them I could do the first initial code run for this upgrade in 4 to 6 (underpromise/overdeliver) weeks. They have some very dedicated and intelligent development expertise on staff for AX2012R3. As good as a consultant is, there is also an inherent advantage to having a developer who has been working with a company for several years, learning the business process, go through your code. I had that advantage here. So, my initial development thing was really a speed up sort of thing. 20 percent of the language is new but other than that, Dynamics Operations is remarkably similar. So, we decided that I would train the employees on how to do an upgrade, and they would take over from there.

But and there is always a “but”, there are certain Universal things to be aware of about this upgrade for AX to Dynamics 365 for Operations Upgraders:

Just a couple of things. If you have to convert every overlayered element into extensions, add about 6 months of time to the upgrade.

And now, we are ready for the final part of this series.. but I should submit it in the next blog post. We merge the code and get it ready. All errors have disappeared now for our dear old sales form.

So, what have we learned:

Geeeze.. we definitely “kept it real” on this blog post. We took a customer from 6,937 errors down to just over a 1000 resulting in massive amounts of saved time with a little human intuition once we identified a red herring, undocumented, LCS code upgrade bug problem causing delays and over-inflated estimates.

Alright, just one more post to go for this series. By now upgrades should start to click. Everything is setup and we resolved almost 5000 errors, radically speeding up the upgrade.. But the important point was to learn by doing. Till the next time..

Sad and Happy Note: it is that time of the year where the technical architects are actually appreciated and people want things like fast systems (strangely, this behavior is very seasonal). If I’m slow to answer emails, it’s probably because somebody has me on something at some location in the world. Rest assured, I’ll answer your emails as soon as I can.