A data migration to Google Web Services (Google Premium - as Cloud) could run in an application architecture that includes the following technologies: Dynamics CRM (On-Premise), Windows Services, Microsoft Message Queue and Google Premium (Google Cloud Service). I'd like to describe how we could resolve an issue in a context of time contraints.
Introduction and context
Accounts and contacts data had to be retrieved from CRM and placed in the Microsoft Message Queue (MSMQ) data storage directory. Data was exported in XML format and stored in a directory created for this purpose. The service (Windows Service - .NET application developed in C#, deployed in Windows Server 2003) behaved as a "Listener" in order to retrieve the data stored in the MSMQ directory.
Once retrieved, the "Listener" had to authenticate to Google Premium Web Services in order to import the data (CRM accounts and contacts) into the online service directories (Google Premium).
Once the data was extracted from the MSMQ directory to the Google Premium directories, the data was cleaned from the MSMQ directory.
However, in the event of an error or exception generated by Google’s Web services, the data was stored in an MSMQ directory created for this type of situation.
Finally, the "Listener" had to establish communication with Google’s Web services and for that purpose, a security token ("Token") had to be generated and its refresh was also generated during the transaction.
For further information, Google API Web service uses the OAuth 2.0 "Framework" which specifies several types of authorization (Refresh Token, Client Credentials, etc.).
Problem : Refreshing the token during data migration
The security token could no longer be refreshed according to the contents of the error generated by the .NET library "Google.Api.Ads.dll":
Preliminary solution validation process
Log file of the listener
First, the necessary information for the authentication process had to be regenerated from the tool "OAuth2TokenGenerator.exe. I therefore regenerated the value of the “refresh_token”.
Then, assuming that the authentication settings had changed on the Google web services side, we ran a series of tests by extracting and adding the settings mentioned on the Google website. However, according to Google’s website, the parameter that seemed to be missing was "grant_type", but adding it didn’t work.
Event viewer of the server
After much research, this error 2164 (associated with the listener) led us to apply the solution proposed by Microsoft [2], but that did not work. However, error 2164 no longer appeared.
Other issues
The lack of space on the server's hard drive has led us to increase its space;
The "Firewall" of the 2003 server was disabled;
McAfee anti-virus software was disabled.
Isolation process
Windows Service (.exe)
First isolation test :
An application that was not a "Windows Service" (Listener) type was deployed in this environment and run from the same environment. However, the validation test generates the same error as the above mentioned one "Failed to refresh access token".
Second isolation test :
The application ("Windows Service" / Listener) was deployed in and runs from the same environment. However, the second validation test gets the same error as the above mentioned one "Failed to refresh access token".
Third-Party and Google Web Services
The purpose is to know if the issue was directly related to the .NET libraries of the Google API. To do so, two validation methods were possible: Unit Testing and Remote Debugging.
Third isolation test : Unit tests were performed on the "Third-Party" used by the Google Web service that the lsitener used when requesting DFP Publisher. The aim was to test the methods of the DLL "Google.Dfp.dll" in practice. In concrete terms, I retrieved the code from the "GitHub" site and performed unit tests on this solution (Visual Studio).
This test allowed me to assess the classes and methods that were mentioned in the error (see the appendix page which presents the details of the exception). Also, the annex presents a series of images showing the execution of unit tests (recovery of web services, initialization of services, generation of security token and refresh of security token).
Fourth isolation test : With the "Remote debugger" of Visual Studio 2010, I debugged the listener code to recover more information. I have only retrieved the information already mentioned.
Phases of testing and debugging process
1- Get the Web Services
2- Initialize the services
3- Generate the security token
4- Refresh the security token
Physical environment of the listener
The purpose is to run the Listener as a .NET executable in different Windows Server operating systems, considering the different « Domain Controller ». The latter is a server responsible for network security and acts as a guardian for user authentication and authorization. The results were mixed, both positive and negative.
Logged-in user in servers and domain controller
As before, the goal is to run the Listener as a .NET executable in different Windows Server operating systems, considering the different Domain Controllers. However, this time we take into account the user connected via the "Domain Controller".
Resolution process : Provision a new listener in the architecture
The idea was to create a bridge between the two "Message Queues" of 2003 and 2008/2012 servers while maintaining the Dynamics CRM platform as it is and the current "Listener" code.
Once the new listener (Listener A) has been developed and deployed in the environment (source), it will be able to transfer the data to the other listener (Listener B). The last one will be able to transfer the data to the Cloud (Google Web Service). Here’s below the steps of the resolution process :
Step 1 - The CRM data is transferred in XML format to a private queue of the MSMQ of the 2003 server (Windows Server 2003).
Step 2 - The new bridge that is a Windows "ListenerBridge" service (Listener A) retrieves the message from the MSMQ server 2003.
Step 3 - The new "ListenerBridge" bridge transfers to the MSMQ private queue of another server (Windows Server 2008/2012).
Step 4 - Once the transfer is complete, the "ListenerBridge" clears the message sent from the MSMQ on the 2003 server and processes the other messages in the queue.
Step 5 - The old Windows service "MSMQ Listener" (Listener B), which is in the 2008/2012 server, retrieves the new message from the private queue of the same name.
Step 6 - The "MSMQ Listener" service connects to the Google[4] service and retrieves all objects (CompanyService, etc.).
Step 7 - Once the specific service is retrieved (CompanyService, etc.), the Listener executes a “GET” request to validate if the contact and account mentioned in the message already exist in the "Cloud" directory.
Step 8 - Once the validation of step 7 has been completed, the service creates or modifies the contact or account in Google's "Cloud" universe (a “POST” type of query).
Step 9 - Then, the "Listener" deletes the message from the private queue and moves to the next message. If an error occurs in step 8, the message is transferred to a private queue created for that purpose.
Conclusion
The purpose was to demonstrate how we could solve a problem of refreshing related to a securiy token, how we could implement a process of resolution in a context of time constraints.
You can download my white paper : "Lessons from a Cloud data migration journey with Dynamics CRM on-premise".