Please navigate to the bottom of the page for Table of Contents

Friday, March 15, 2013

Data migration strategies and design patterns

Data migration is an extremely common operation in software design and development. Whenever a new system is introduced or a legacy system is redesigned, existing data has to be moved from the legacy system to the new target system.

From an interview perspective, you should know and understand the following well understood design patterns and guidelines that make the process of data migration smoother, faster and accurate with least amount of surprises during go-live.



  1. Develop with Production Data: Use real data from the production system for tests during the development of the migration code.
  2. Migrate along Domain Partitions: Divide and conquer the migration effort by migrating largely independent parts of the domain model one after another.
  3. Measure Migration Quality: Implement code that collects and stores all sorts of information about the outcome of the migration during every run.
  4. Periodic Quality Reports: Generate detailed reports about the measured quality of the migrated data and make it available to all affected stake holders.
  5. Robust Processing: To prevent the migration process to halt from unexpected failure, apply extensive exception handling to cope with all kinds of problematic input data.
  6. Data Cleansing: To prevent the new application from being swamped with useless data right from the start, enhance your transformation processes with data cleansing mechanisms.
  7. Incremental Transformation: Perform an initial data migration before the new application goes live. Migrate data that has changed since then immediately before the new application is launched. 
 A few other salient points to keep in mind are:
  1. During migration (extraction and load), the processes should log the following items to facilitate the monitoring, debugging and verification process:
    • Global and per entity start time
    • Global and per entity end time
    • Number of entities to process
    • Entity
    • Source table(s) and schema
    • Destination schema
    • Number of records processed
    • Errors/Warnings encountered
  2.  Data attributes should be anonymized before being processed for testing purposes. The following data elements should be anonymized before they are processed by “load” systems. Anonymization process should use well defined values that the test team is aware of.
    • Email
    • Phone numbers
    • Passwords
    • IP Addresses 
  3.  Build automated tools for analyzing the logs generated during Migration. The advantages of logging and analyzing the logs have huge advantages:
    • Potential migration issues are discovered earlier and can be fixed thereby reducing the cost and effort
    • Data inconsistencies issues can be fixed sooner
    • Facilitates the coordination between factories and provides a common language for analyzing and debugging
Lastly, you might be required to draw your thoughts on a whiteboard on how you would model an ETL (Extract-Transform-Load) strategy. Here is one model that has worked for many common data migration scenarios. Since data migration is a one-time operation, the GOAL should be to reduce the number of transformation (and load) processes. C can be selectively used based on if multiple passes are required to transform the data.



What other strategies have you used? What has worked well for you in the past? What did not go well? The readers and would love to hear your thoughts and experiences.

71 comments:

  1. Thank you. this is great information. I am going through a data migration project and this helped a lot.

    ReplyDelete
  2. This is a good article & good site.Thank you for sharing this article. It is help us following categorize:
    healthcare, e commerce, programming, multi platform,inventory management, cloud-based solutions, it consulting, retail, manufacturing, CRM, technology means, digital supply chain management, Delivering high-quality service for your business applications,
    Solutions for all Industries,packaged applications,business applications, Web services, data migration
    Business intelligence, Business Development, Software Development etc.


    Our address:
    2002 Timberloch Place, Suite 200
    The Woodlands, TX 77380
    281-364-1799

    prologic-corp

    ReplyDelete
    Replies
    1. Selamat Petang,


      What a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this.


      I google so many articles and blog to know what exactly Proxy Pattern is and it's examples. every where I say example related to banking or Images.

      But now I am confuse and just want to know "Can we say that Server side pagination comes under Proxy Pattern".

      From Blogs and Articles I read, I come to know that proxy pattern create object having original object to interface its functionality to outer world. In other words, provide desire information from the real object as and when required.

      please correct me if I am wrong and help me to clear my confusion.





      But nice Article Mate! Great Information! Keep up the
      good
      work!


      Thanks a heaps,

      Delete
  3. Being flagship too for Microsoft, Asp.Net is loaded with various essential tools that allow developers to build robust web applications. PHP Training in Chennai

    ReplyDelete
  4. Your post about web design and development is really interesting. When comes to content management system, wordpress is widely used tool. It loaded with various stunning features that make website creation lot simpler. Wordpress Training in Chennai | Wordpress Course in Chennai | Wordpress Training Institutes in Chennai

    ReplyDelete
  5. Contact us for Website designs and service and also ios applications development

    ReplyDelete

  6. This is one of the cult game now, a lot of people enjoy playing them . Also you can refer to the game :
    animal jam 2 | five nights at freddys 2 | hotmail login

    ReplyDelete
  7. Thank you for this valuable information. I have got some important suggestions from it. I'm working in Brave Technologies Private Limited. We provides lowest price of ERP Software for our clients. Contact us on info@bravetechnologies.in
    Manufacturing ERP software Chennai | Automotive ERP in Chennai

    ReplyDelete
  8. Really Nice Blog. Thank you for Sharing. We are the best erp software providers in chennai. For more details call +91 9677025199 or email us on info@bravetechnologies.in
    erp in chennai

    ReplyDelete
  9. Nice information about data migeration strategies My sincere thanks for sharing this post
    Hadoop Training in Chennai

    ReplyDelete
  10. really you have posted an informative blog. before i read this blog i didn't have any knowledge about this but now i got some knowledge. so keep on sharing such kind of an interesting blogs.
    hadoop training in chennai

    ReplyDelete
  11. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work. Cloud Computing Training in Chennai | Salesforce Training in Chennai

    ReplyDelete
  12. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post..Big Data Hadoop Training in Bangalore | Data Science Training in Bangalore

    ReplyDelete
  13. Your Blog is nice and informative,thanks for sharing this helpful blog.keep updating such an valuable information sharing here..
    Mat Lab Project Center in Chennai | Best Mat Lab Project Center in Chennai | No.1 Mat Lab Project Center in Chennai.

    ReplyDelete
  14. I found a lot of interesting information here. A really good post, very thankful and hopeful that you will write many more posts like this one.

    Webdesign Lüdenscheid

    ReplyDelete
  15. I found a lot of interesting information here. A really good post
    office 2010 professional plus key deutsch

    ReplyDelete
  16. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts, have a nice weekend!

    Data Science Training in Chennai

    ReplyDelete
  17. Thank you for the writing a good article and it helps me a lot. Buy the Cold Pressed Oil in India.

    ReplyDelete
  18. Your good knowledge and kindness in playing with all the pieces were
    very useful. I don’t know what I would have done if I had not
    encountered such a step like this.


    AWS Training in Bangalore


    AWS Training in Bangalore

    ReplyDelete
  19. Interesting post! This is really helpful for me.I like it! Thanks for sharing...Java Projects Center in Chennai | Java Projects Center in Velachery

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Your very own commitment to getting the message throughout came
    to be rather powerful and have consistently enabled employees just
    like me to arrive at their desired goals.


    aws training in chennai

    ReplyDelete
  22. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    python training in bangalore

    ReplyDelete
  23. Thanks for one marvelous posting! I enjoyed reading it;Great post.The information was very useful.Keep the good work goin on!!
    Hadoop training in chennai | Mainframe training in chennai | SAP SD training in chennai

    ReplyDelete
  24. Thanks for sharing such informative guide on Data Migration Strategies and design. This post gives me detailed information about the Data Migration. Thanks for sharing. RPA Training Institute in Chennai | UI Path Training Institute in Chennai | Blue Prism Training Institute in Chennai

    ReplyDelete
  25. Very impressive and informative article with cute content..Thanks for sharing this nice blog.
    Best Data Mining Project Center in Chennai | No.1 Data Mining Project Center in Velachery

    ReplyDelete
  26. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    Austere Technologies | Best Cloud Solution services

    ReplyDelete
  27. REALLY VERY EXCELLENT INFORMATION. I AM VERY GLAD TO SEE YOUR BLOG FOR THIS INFORMATION. THANKS FOR SHARING. KEEP UPDATING.

    NO.1 Mobile APPilication DEVELOPMENT SERVICES | MASSIL TECHNOLOGIES

    ReplyDelete
  28. Superb blog with great explanation Data Migration. Thanks for sharing, keep updating.

    NO.1 CLOUD SERVICES | Oracle Cloud PAAS | MASSIL TECHNOLOGIES

    ReplyDelete
  29. Thanks a lot very much for the high your blog post quality and results-oriented help. I won’t think twice to endorse to anybody who wants and needs support about this area.
    rpa training in chennai

    ReplyDelete
  30. wowwww nice post, It was great experience for me. I have learned a lot of new methods.
    Robotics Project Center in Chennai | Robotics Training Institute in Chennai

    ReplyDelete
  31. Thanks for sharing this kind of information. iam so happy to read your blog. Thanks you so much.
    Mobile Computing Project Center in Chennai | Mobile Computing Projects in Velachery

    ReplyDelete
  32. Nice post.It was really effective..Thank you for sharingNice post.It was really effective..Thank you for sharingData Mining Project Center in Chennai | Data Mining Projects in Velachery

    ReplyDelete
  33. Excellent post..Thanks for sharing your amazing blog with useful content.keep updating
    Cloud Computing Project Center in Chennai | Cloud Computing Projects in Velachery

    ReplyDelete
  34. Excellent information you made in this blog, very helpful information. Thanks for sharing.

    Software Testing | Austere Technology

    ReplyDelete
  35. nice post, Thanks for sharing this information and keep updating us.This content is really helpful to me.
    Electrical Project Center in Chennai | Electrical Projects in Velachery

    ReplyDelete
  36. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...

    Image Processing Project Center in Chennai | Image Processing Projects in Velachery

    ReplyDelete
  37. This information is impressive; I am inspired by your post writing style & how continuously you describe this topic.
    Cloud Computing Project Center in Chennai | Cloud Computing Projects in Velachery

    ReplyDelete
  38. Thanks for sharing this kind of information. iam so happy to read your blog. Thanks you so much.
    Final Year Projects for BE in Chennai | Final Year Projects for BE in Velachery

    ReplyDelete
  39. Great article, really very helpful content you made. Thank you, keep sharing.

    chartered accountant | Avinash college of commerce

    ReplyDelete
  40. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work.
    Image Processing Project Center in Chennai | Image Processing Projects in Velachery

    ReplyDelete
  41. Your blog is really amazing with useful information you are shared.Thank you so much for sharing such an wonderful article with us..keep updating....
    Mobile Computing Project Center in Chennai | Mobile Computing Projects in Velachery

    ReplyDelete
  42. Impressive blog with lovely information. really very useful article for us thanks for sharing such a wonderful blog...keep updating..
    Power System Project Center in Chennai | Power System Project Center in Velachery

    ReplyDelete