Please navigate to the bottom of the page for Table of Contents

Friday, March 15, 2013

Data migration strategies and design patterns

Data migration is an extremely common operation in software design and development. Whenever a new system is introduced or a legacy system is redesigned, existing data has to be moved from the legacy system to the new target system.

From an interview perspective, you should know and understand the following well understood design patterns and guidelines that make the process of data migration smoother, faster and accurate with least amount of surprises during go-live.



  1. Develop with Production Data: Use real data from the production system for tests during the development of the migration code.
  2. Migrate along Domain Partitions: Divide and conquer the migration effort by migrating largely independent parts of the domain model one after another.
  3. Measure Migration Quality: Implement code that collects and stores all sorts of information about the outcome of the migration during every run.
  4. Periodic Quality Reports: Generate detailed reports about the measured quality of the migrated data and make it available to all affected stake holders.
  5. Robust Processing: To prevent the migration process to halt from unexpected failure, apply extensive exception handling to cope with all kinds of problematic input data.
  6. Data Cleansing: To prevent the new application from being swamped with useless data right from the start, enhance your transformation processes with data cleansing mechanisms.
  7. Incremental Transformation: Perform an initial data migration before the new application goes live. Migrate data that has changed since then immediately before the new application is launched. 
 A few other salient points to keep in mind are:
  1. During migration (extraction and load), the processes should log the following items to facilitate the monitoring, debugging and verification process:
    • Global and per entity start time
    • Global and per entity end time
    • Number of entities to process
    • Entity
    • Source table(s) and schema
    • Destination schema
    • Number of records processed
    • Errors/Warnings encountered
  2.  Data attributes should be anonymized before being processed for testing purposes. The following data elements should be anonymized before they are processed by “load” systems. Anonymization process should use well defined values that the test team is aware of.
    • Email
    • Phone numbers
    • Passwords
    • IP Addresses 
  3.  Build automated tools for analyzing the logs generated during Migration. The advantages of logging and analyzing the logs have huge advantages:
    • Potential migration issues are discovered earlier and can be fixed thereby reducing the cost and effort
    • Data inconsistencies issues can be fixed sooner
    • Facilitates the coordination between factories and provides a common language for analyzing and debugging
Lastly, you might be required to draw your thoughts on a whiteboard on how you would model an ETL (Extract-Transform-Load) strategy. Here is one model that has worked for many common data migration scenarios. Since data migration is a one-time operation, the GOAL should be to reduce the number of transformation (and load) processes. C can be selectively used based on if multiple passes are required to transform the data.



What other strategies have you used? What has worked well for you in the past? What did not go well? The readers and would love to hear your thoughts and experiences.

129 comments:

  1. Thank you. this is great information. I am going through a data migration project and this helped a lot.

    ReplyDelete
  2. This is a good article & good site.Thank you for sharing this article. It is help us following categorize:
    healthcare, e commerce, programming, multi platform,inventory management, cloud-based solutions, it consulting, retail, manufacturing, CRM, technology means, digital supply chain management, Delivering high-quality service for your business applications,
    Solutions for all Industries,packaged applications,business applications, Web services, data migration
    Business intelligence, Business Development, Software Development etc.


    Our address:
    2002 Timberloch Place, Suite 200
    The Woodlands, TX 77380
    281-364-1799

    prologic-corp

    ReplyDelete
    Replies
    1. Selamat Petang,


      What a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this.


      I google so many articles and blog to know what exactly Proxy Pattern is and it's examples. every where I say example related to banking or Images.

      But now I am confuse and just want to know "Can we say that Server side pagination comes under Proxy Pattern".

      From Blogs and Articles I read, I come to know that proxy pattern create object having original object to interface its functionality to outer world. In other words, provide desire information from the real object as and when required.

      please correct me if I am wrong and help me to clear my confusion.





      But nice Article Mate! Great Information! Keep up the
      good
      work!


      Thanks a heaps,

      Delete
  3. Being flagship too for Microsoft, Asp.Net is loaded with various essential tools that allow developers to build robust web applications. PHP Training in Chennai

    ReplyDelete
  4. Your post about web design and development is really interesting. When comes to content management system, wordpress is widely used tool. It loaded with various stunning features that make website creation lot simpler. Wordpress Training in Chennai | Wordpress Course in Chennai | Wordpress Training Institutes in Chennai

    ReplyDelete
  5. Contact us for Website designs and service and also ios applications development

    ReplyDelete

  6. This is one of the cult game now, a lot of people enjoy playing them . Also you can refer to the game :
    animal jam 2 | five nights at freddys 2 | hotmail login

    ReplyDelete
  7. Thank you for this valuable information. I have got some important suggestions from it. I'm working in Brave Technologies Private Limited. We provides lowest price of ERP Software for our clients. Contact us on info@bravetechnologies.in
    Manufacturing ERP software Chennai | Automotive ERP in Chennai

    ReplyDelete
  8. Really Nice Blog. Thank you for Sharing. We are the best erp software providers in chennai. For more details call +91 9677025199 or email us on info@bravetechnologies.in
    erp in chennai

    ReplyDelete
  9. Nice information about data migeration strategies My sincere thanks for sharing this post
    Hadoop Training in Chennai

    ReplyDelete
  10. really you have posted an informative blog. before i read this blog i didn't have any knowledge about this but now i got some knowledge. so keep on sharing such kind of an interesting blogs.
    hadoop training in chennai

    ReplyDelete
  11. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work. Cloud Computing Training in Chennai | Salesforce Training in Chennai

    ReplyDelete
  12. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post..Big Data Hadoop Training in Bangalore | Data Science Training in Bangalore

    ReplyDelete
  13. Your Blog is nice and informative,thanks for sharing this helpful blog.keep updating such an valuable information sharing here..
    Mat Lab Project Center in Chennai | Best Mat Lab Project Center in Chennai | No.1 Mat Lab Project Center in Chennai.

    ReplyDelete
  14. I found a lot of interesting information here. A really good post, very thankful and hopeful that you will write many more posts like this one.

    Webdesign Lüdenscheid

    ReplyDelete
  15. I found a lot of interesting information here. A really good post
    office 2010 professional plus key deutsch

    ReplyDelete
  16. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts, have a nice weekend!

    Data Science Training in Chennai

    ReplyDelete
  17. Thank you for the writing a good article and it helps me a lot. Buy the Cold Pressed Oil in India.

    ReplyDelete
  18. Your good knowledge and kindness in playing with all the pieces were
    very useful. I don’t know what I would have done if I had not
    encountered such a step like this.


    AWS Training in Bangalore


    AWS Training in Bangalore

    ReplyDelete
  19. Interesting post! This is really helpful for me.I like it! Thanks for sharing...Java Projects Center in Chennai | Java Projects Center in Velachery

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Your very own commitment to getting the message throughout came
    to be rather powerful and have consistently enabled employees just
    like me to arrive at their desired goals.


    aws training in chennai

    ReplyDelete
  22. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    python training in bangalore

    ReplyDelete
  23. Thanks for one marvelous posting! I enjoyed reading it;Great post.The information was very useful.Keep the good work goin on!!
    Hadoop training in chennai | Mainframe training in chennai | SAP SD training in chennai

    ReplyDelete
  24. Thanks for sharing such informative guide on Data Migration Strategies and design. This post gives me detailed information about the Data Migration. Thanks for sharing. RPA Training Institute in Chennai | UI Path Training Institute in Chennai | Blue Prism Training Institute in Chennai

    ReplyDelete
  25. Very impressive and informative article with cute content..Thanks for sharing this nice blog.
    Best Data Mining Project Center in Chennai | No.1 Data Mining Project Center in Velachery

    ReplyDelete
  26. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    Austere Technologies | Best Cloud Solution services

    ReplyDelete
  27. REALLY VERY EXCELLENT INFORMATION. I AM VERY GLAD TO SEE YOUR BLOG FOR THIS INFORMATION. THANKS FOR SHARING. KEEP UPDATING.

    NO.1 Mobile APPilication DEVELOPMENT SERVICES | MASSIL TECHNOLOGIES

    ReplyDelete
  28. Superb blog with great explanation Data Migration. Thanks for sharing, keep updating.

    NO.1 CLOUD SERVICES | Oracle Cloud PAAS | MASSIL TECHNOLOGIES

    ReplyDelete
  29. Thanks a lot very much for the high your blog post quality and results-oriented help. I won’t think twice to endorse to anybody who wants and needs support about this area.
    rpa training in chennai

    ReplyDelete
  30. wowwww nice post, It was great experience for me. I have learned a lot of new methods.
    Robotics Project Center in Chennai | Robotics Training Institute in Chennai

    ReplyDelete
  31. Thanks for sharing this kind of information. iam so happy to read your blog. Thanks you so much.
    Mobile Computing Project Center in Chennai | Mobile Computing Projects in Velachery

    ReplyDelete
  32. Nice post.It was really effective..Thank you for sharingNice post.It was really effective..Thank you for sharingData Mining Project Center in Chennai | Data Mining Projects in Velachery

    ReplyDelete
  33. Excellent post..Thanks for sharing your amazing blog with useful content.keep updating
    Cloud Computing Project Center in Chennai | Cloud Computing Projects in Velachery

    ReplyDelete
  34. Excellent information you made in this blog, very helpful information. Thanks for sharing.

    Software Testing | Austere Technology

    ReplyDelete
  35. nice post, Thanks for sharing this information and keep updating us.This content is really helpful to me.
    Electrical Project Center in Chennai | Electrical Projects in Velachery

    ReplyDelete
  36. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...

    Image Processing Project Center in Chennai | Image Processing Projects in Velachery

    ReplyDelete
  37. This information is impressive; I am inspired by your post writing style & how continuously you describe this topic.
    Cloud Computing Project Center in Chennai | Cloud Computing Projects in Velachery

    ReplyDelete
  38. Thanks for sharing this kind of information. iam so happy to read your blog. Thanks you so much.
    Final Year Projects for BE in Chennai | Final Year Projects for BE in Velachery

    ReplyDelete
  39. Great article, really very helpful content you made. Thank you, keep sharing.

    chartered accountant | Avinash college of commerce

    ReplyDelete
  40. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work.
    Image Processing Project Center in Chennai | Image Processing Projects in Velachery

    ReplyDelete
  41. Your blog is really amazing with useful information you are shared.Thank you so much for sharing such an wonderful article with us..keep updating....
    Mobile Computing Project Center in Chennai | Mobile Computing Projects in Velachery

    ReplyDelete
  42. Impressive blog with lovely information. really very useful article for us thanks for sharing such a wonderful blog...keep updating..
    Power System Project Center in Chennai | Power System Project Center in Velachery

    ReplyDelete
  43. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Cloud Computing Project Center in Chennai | Cloud Computing Project Center in Velachery

    ReplyDelete
  44. This an awesome blog with helpful content, I m really impressed with your article, such great & useful knowledge post..
    Embedded Project Center in Chennai | Embedded Project Center in Velachery

    ReplyDelete
  45. Your Blog is really Nice and Informative..Thanks for sharing such a interesting article..keep updating..I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work.

    Java Project Center in Chennai | Java Project Center in Velachery

    ReplyDelete
  46. Hi Thanks for the nice information its very useful to read your blog. We provide best Block Chain Services

    ReplyDelete
  47. Excellent post. I have read your blog it's very interesting and informative.Thanks for sharing such nice article, keep on up dating such good articles.
    Android Project Center in Chennai | Android Project Center in Velachery

    ReplyDelete
  48. Nice and good article.. it is very useful for me to learn and understand easily.. thanks for sharing your valuable information and time.. please keep updating.
    Cloud Computing Project Center in Chennai | Cloud Computing Project Center in Velachery

    ReplyDelete
  49. Thank you for sharing this valuable information. But get out of this busy life and find some peace with a beautiful trip book Andaman family tour packages

    ReplyDelete
  50. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    Final Year Project Center for BE in Chennai | Final Year Projects for BE in Velachery

    ReplyDelete
  51. Thank you for sharing this valuable information. But get out this busy life and find some peace with a beautiful trip. book ANDAMAN BUDGET PACKAGES @ 4999/-

    ReplyDelete
  52. Awesome blog. Your articles really impressed for me, because of all information so nice and unique.
    Android Project Center in Chennai | Android Project Center in Velachery

    ReplyDelete
  53. Thank you for sharing this valuable information. But get out this busy life and find some peace with a beautiful trip. book Andaman Tourism

    ReplyDelete
  54. Thank you for sharing this valuable information. But get out this busy life and find some peace with a beautiful trip. book Best Travel Agency In India

    ReplyDelete
  55. Your Blog is really amazing with useful information you are shared here...Thank you for sharing such an informative and wonderful post with us. Keep updating.
    AWS Training Institute in Chennai | AWS Training Institute in Velachery

    ReplyDelete
  56. I have read your blog,its really useful for me..Thank you for sharing such an creative article..
    Linux Training Institute in Chennai | Linux Training in Velachery

    ReplyDelete
  57. Your Blog is really amazing with informative content,thank you so much for sharing such an wonderful Post with us.keep updating...

    Best Python Training Institute in Chennai | Best Python Training in Velachery

    ReplyDelete
  58. Thank you so much for sharing your informative post.Its really useful for me.keep updating such a wonderful post with us..
    No.1 Java Training Institute in Chennai | No.1 Java Training Center in Velachery

    ReplyDelete
  59. Hi Thanks for the nice information its very useful to read your blog. We provide best Find All Isfs Courses

    ReplyDelete
  60. Hi Thanks for the nice information its very useful to read your blog. We provide best Massil Technologies

    ReplyDelete
  61. Your article is really an amazing with useful content, thank you so much for sharing such an informative information. keep updating..
    Best AWS Training Institute in Chennai | No.1 AWS Training Institute in Velachery

    ReplyDelete
  62. Thank you so much for sharing such an informative information with us.its really useful for everyone,keep updating such an wonderful blog you are shared.
    Best Linux Training Institute in Kanchipuram | No.1 Linux Training Institute in Kanchipuram

    ReplyDelete
  63. Your Blog is really awesome with useful content,its very helpful for us.Thanks for sharing such an wonderful article here..keep updating
    Best CCNA Training Institute in Kanchipuram | CCNA Training Center in Kanchipuram

    ReplyDelete
  64. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us. Do check Six Sigma Training in Bangalore | Six Sigma Training in Dubai & Get trained by an expert who will enrich you with the latest trends.

    ReplyDelete
  65. Great post and informative blog.it was awesome to read, thanks for sharing this great content to my vision. This is a great inspiring article.I am pretty much pleased with your good work. You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post..

    Best Python Training in Chennai | Python Certification Training Centers in Chennai | Python Exams in Chennai | Python Exam Center in Chennai

    ReplyDelete
  66. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…This is a great inspiring article.I am pretty much pleased with your good work. You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post..


    Best Python Training in Chennai | Python Certification Training Centers in Chennai | Python Exams in Chennai | Python Exam Center in Chennai

    ReplyDelete
  67. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
    Embedded System Training Institute in Kanchipuram | Embedded Training Center in Kanchipuram

    ReplyDelete
  68. Great post....Thank you for posting the great content……I found it quiet interesting, hopefully you will keep posting such blogs…Keep updating your creative and awesome blog.

    Best Python Exam Center in Chennai | Python Certification Training in Chennai | Python Training in Chennai | Python Exams in Chennai

    ReplyDelete
  69. This is a great inspiring article.I am pretty much pleased with your good work. You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post,.Thank You For Sharing This Information..
    Best CCNA Training Institute in Chennai | CCNA Training Center in Velachery

    ReplyDelete
  70. Great post and informative blog.it was awesome to read, thanks for sharing this great content to my vision. This is a great inspiring article.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post..
    Best Python Exam Center in Chennai | Python Certification Training in Chennai | Python Training in Chennai | Python Exams in Chennai

    ReplyDelete
  71. Your article is really amazing with useful information you are shared,its very helpful for everyone.Thanks a lot for sharing such an informative information.keep updating..
    Best Python Exam Center in Chennai | Python Exams in Chennai | Python Exams in Velachery

    ReplyDelete
  72. Very informative post and useful post so happy to read this article share it for everyone.Pretty post. Whatever we accumulated data from the web journals, we ought to actualize that in basically then no one but we can comprehend that precise thing obviously.

    Linux Training Institute in Chennai | Linux Training Institute in Velachery

    ReplyDelete
  73. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
    Best Java Training Institute in Chennai | Java Training in Velachery

    ReplyDelete
  74. It’s really a nice and helpful piece of information. I’m satisfied that you just shared this helpful information with us. Please stay us informed like this. Thanks for sharing.
    MatLab Training Institute in Chennai | MatLab Training in Velachery

    ReplyDelete
  75. Nice Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one,keep updating..

    Best AWS Training Institute in Taramani | No.1 AWS Training Center in Taramani

    ReplyDelete
  76. Your article is really amazing with informative information,you are shared.Thanks a lot for sharing this wonderful blog.keep updating such a excellent post with us.
    Embedded System Training in Tambaram | Embedded Training in Tambaram

    ReplyDelete
  77. Thanks for sharing such an informative blog to us, keep on updating.
    ACCA training institute in Hyderabad | ISFS

    ReplyDelete
  78. Very informative blog. Helps to gain knowledge about new concepts and techniques. Thanks a lot for sharing this wonderful blog.keep updating such a excellent post with us.
    Best MatLab Training Institute in OMR | No.1 MatLab Training Center in OMR

    ReplyDelete
  79. I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details.
    Best CCNA Training Institute in Guindy | No.1 CCNA Training Institute in Guindy

    ReplyDelete
  80. Very informative blog. Helps to gain knowledge about new concepts and techniques. Thanks a lot for sharing this wonderful blog.keep updating such a excellent post with us.
    AWS Exam Center in Chennai | AWS Certification Exams in Chennai | AWS Exams in Velachery

    ReplyDelete
  81. Very informative blog. Helps to gain knowledge about new concepts and techniques. Thanks a lot for sharing this wonderful blog.keep updating such a excellent post with us.
    AWS Exam Center in Chennai | AWS Certification Exams in Chennai | AWS Exams in Velachery

    ReplyDelete
  82. Thank you for your post. This was really an appreciating one. You done a good job. Keep on blogging like this unique information with us.
    Best Embedded Training Institute in Thiruvanmiyur | No.1 Embedded Training Institute in Thiruvanmiyur

    ReplyDelete
  83. Thank you for your post. This was really an appreciating one. You done a good job. Keep on blogging like this unique information with us.
    Best Embedded Training Institute in Thiruvanmiyur | No.1 Embedded Training Institute in Thiruvanmiyur

    ReplyDelete