Please navigate to the bottom of the page for Table of Contents

Friday, March 15, 2013

Data migration strategies and design patterns

Data migration is an extremely common operation in software design and development. Whenever a new system is introduced or a legacy system is redesigned, existing data has to be moved from the legacy system to the new target system.

From an interview perspective, you should know and understand the following well understood design patterns and guidelines that make the process of data migration smoother, faster and accurate with least amount of surprises during go-live.

  1. Develop with Production Data: Use real data from the production system for tests during the development of the migration code.
  2. Migrate along Domain Partitions: Divide and conquer the migration effort by migrating largely independent parts of the domain model one after another.
  3. Measure Migration Quality: Implement code that collects and stores all sorts of information about the outcome of the migration during every run.
  4. Periodic Quality Reports: Generate detailed reports about the measured quality of the migrated data and make it available to all affected stake holders.
  5. Robust Processing: To prevent the migration process to halt from unexpected failure, apply extensive exception handling to cope with all kinds of problematic input data.
  6. Data Cleansing: To prevent the new application from being swamped with useless data right from the start, enhance your transformation processes with data cleansing mechanisms.
  7. Incremental Transformation: Perform an initial data migration before the new application goes live. Migrate data that has changed since then immediately before the new application is launched. 
 A few other salient points to keep in mind are:
  1. During migration (extraction and load), the processes should log the following items to facilitate the monitoring, debugging and verification process:
    • Global and per entity start time
    • Global and per entity end time
    • Number of entities to process
    • Entity
    • Source table(s) and schema
    • Destination schema
    • Number of records processed
    • Errors/Warnings encountered
  2.  Data attributes should be anonymized before being processed for testing purposes. The following data elements should be anonymized before they are processed by “load” systems. Anonymization process should use well defined values that the test team is aware of.
    • Email
    • Phone numbers
    • Passwords
    • IP Addresses 
  3.  Build automated tools for analyzing the logs generated during Migration. The advantages of logging and analyzing the logs have huge advantages:
    • Potential migration issues are discovered earlier and can be fixed thereby reducing the cost and effort
    • Data inconsistencies issues can be fixed sooner
    • Facilitates the coordination between factories and provides a common language for analyzing and debugging
Lastly, you might be required to draw your thoughts on a whiteboard on how you would model an ETL (Extract-Transform-Load) strategy. Here is one model that has worked for many common data migration scenarios. Since data migration is a one-time operation, the GOAL should be to reduce the number of transformation (and load) processes. C can be selectively used based on if multiple passes are required to transform the data.

What other strategies have you used? What has worked well for you in the past? What did not go well? The readers and would love to hear your thoughts and experiences.


  1. Thank you. this is great information. I am going through a data migration project and this helped a lot.

  2. This is a good article & good site.Thank you for sharing this article. It is help us following categorize:
    healthcare, e commerce, programming, multi platform,inventory management, cloud-based solutions, it consulting, retail, manufacturing, CRM, technology means, digital supply chain management, Delivering high-quality service for your business applications,
    Solutions for all Industries,packaged applications,business applications, Web services, data migration
    Business intelligence, Business Development, Software Development etc.

    Our address:
    2002 Timberloch Place, Suite 200
    The Woodlands, TX 77380


  3. Being flagship too for Microsoft, Asp.Net is loaded with various essential tools that allow developers to build robust web applications. PHP Training in Chennai

  4. Your post about web design and development is really interesting. When comes to content management system, wordpress is widely used tool. It loaded with various stunning features that make website creation lot simpler. Wordpress Training in Chennai | Wordpress Course in Chennai | Wordpress Training Institutes in Chennai

  5. Contact us for Website designs and service and also ios applications development


  6. This is one of the cult game now, a lot of people enjoy playing them . Also you can refer to the game :
    animal jam 2 | five nights at freddys 2 | hotmail login

  7. Thank you for this valuable information. I have got some important suggestions from it. I'm working in Brave Technologies Private Limited. We provides lowest price of ERP Software for our clients. Contact us on
    Manufacturing ERP software Chennai | Automotive ERP in Chennai

  8. Really Nice Blog. Thank you for Sharing. We are the best erp software providers in chennai. For more details call +91 9677025199 or email us on
    erp in chennai

  9. Nice information about data migeration strategies My sincere thanks for sharing this post
    Hadoop Training in Chennai

  10. really you have posted an informative blog. before i read this blog i didn't have any knowledge about this but now i got some knowledge. so keep on sharing such kind of an interesting blogs.
    hadoop training in chennai

  11. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work. Cloud Computing Training in Chennai | Salesforce Training in Chennai

  12. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post..Big Data Hadoop Training in Bangalore | Data Science Training in Bangalore

  13. Your Blog is nice and informative,thanks for sharing this helpful blog.keep updating such an valuable information sharing here..
    Mat Lab Project Center in Chennai | Best Mat Lab Project Center in Chennai | No.1 Mat Lab Project Center in Chennai.

  14. I found a lot of interesting information here. A really good post, very thankful and hopeful that you will write many more posts like this one.

    Webdesign Lüdenscheid

  15. I found a lot of interesting information here. A really good post
    office 2010 professional plus key deutsch

  16. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts, have a nice weekend!

    Data Science Training in Chennai

  17. Thank you for the writing a good article and it helps me a lot. Buy the Cold Pressed Oil in India.

  18. Your good knowledge and kindness in playing with all the pieces were
    very useful. I don’t know what I would have done if I had not
    encountered such a step like this.

    AWS Training in Bangalore

    AWS Training in Bangalore