Please navigate to the bottom of the page for Table of Contents

Monday, January 21, 2013

Distributed vs Parallel computing

In this post I will provide a very high level overview of Distributed versus Parallel computing.

Distributed computing refers to the study of distributed systems to solve complex or time consuming problems, broken down to small tasks, across multiple computers (nodes) each of which has its own memory and disk.
In addition, the distributed system has additional constraints such as fault tolerance (individual nodes may fail), unknown structure (the network topology, etc.  may not be known or well defined) and decoupled (individual nodes may not have knowledge of entire system). The key to distributed computing is that there are many small nodes processing and executing tasks without knowing the broader system.

Parallel computing, on the other hand, is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel").

The same system may be characterized both as "parallel" and "distributed"; the processors in a typical distributed system run concurrently in parallel. The key difference between the two is that in parallel computing, all processors may have access to a shared memory to exchange information between processors and in distributed computing, each processor has its own private memory and information is exchanged by passing messages between the processors.


  1. This is very incorrect of course, confusing parallel processing with presence or absence of shared memory. Most and best parallel computing use MPI which is exactly passing messages. While shared-memory techniques like OpenMP had limited success.

  2. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    Best Digital Transformation Services | DM Services | Austere Technologies

  3. Hi Nikhil,

    In total awe…. So much respect and gratitude to you folks for pulling off such amazing blogs without missing any points on the Distributed vs Parallel computing. Kudos!

    I know that you may answer as already in such case that you noticed that I open case 4914182641 in support center and that I need to wait their answer. But when I opened it, I start to search such problems with overdue payments on forums. And I found that everytime people said that they not get answer from support (from 48 hours to 1 week). I'm very affraid that in my case support can not react same as I read it from forums. And that my account can be closed. It's really little sum (approx. 8 EUR for february) and very big time of use (many years). I create domain's there and buckets. And it's very affraid to lost account such way. Can you, please, help me?

    Amazon S3 provides access to reliable and inexpensive data storage infrastructure. It is designed to make web-scale computing easier by enabling you to store and retrieve any amount of data, at any time, from within Amazon EC2 or anywhere on the web.

    But nice Article Mate! Great Information! Keep up the good work!

    Kind Regards,

  4. Hi Mate,

    11/10!! Your blog is such a complete read. I like your approach with Distributed vs Parallel computing . Clearly, you wrote it to make learning a cake walk for me.

    We're currently in an infinite loop between sales and support, neither of whom seem to be able to understand a basic issue.

    We want to purchase some sizeable reserved instances but are told that the only way to pay is all at once with a credit card. No split payments, no offer to pay by check, no offer to pay by ACH, no offer to pay by wire. AWS Training

    Can someone explain to me how AWS serves enterprises if they only accept consumer methods of payment?

    Super likes !!! for this amazing post. I thinks everyone should bookmark this.

    Kind Regards,

    Irene Hynes

  5. Merhaba,

    Three cheers to you ! Hooray!!! I feel like I hit the jackpot on Distributed vs Parallel computing

    I logged into today and now every few seconds a popup says I need to reload (and re-login). This has happened before after a long time of inactivity, but now it's happening so often I can't look at anything without having to reload and login. AWS Tutorial

    THANK YOU!! This saved my butt today, I’m immensely grateful.


  6. I am very impressed after reading such a amazing post about cloud computing. Thanks for sharing such post. Cloud Computing Solutions Companies.

  7. This was nice and amazing and the given contents were very useful and the precision has given here is good.

    Apache Spark Training in Pune
    Spark Training Institute in Pune

  8. I was extremely dazzled by this post, this site has consistently been wonderful information. you particularly for such a fascinating post. oracle fusion finance training india

  9. Great job, You are providing the best knowledge. It is really helpful and factual information for us and everyone to increase knowledge. Continue sharing your data. Thank you. microsoft dynamics houston tx

  10. It’s my fortune to go to at this blog and realize out my required stuff that is also in the quality.
    interaction design agency

  11. I was searching for the finest article on DPU. Because one of my friends told me about fungible that they are providing the best dpu service at a cheap rate. So i think i should visit their website and learn more about DPU.

  12. I’m trampled by your contents carry on the wonderful work.
    logo creation services

  13. The United Kingdom, with its many waves of immigration, is extremely culturally diverse. The UK is not only one of the leading financial centers in the world, but many other strong industries, which together form the fifth largest economy in the world and the second largest economy in Europe after Germany.

  14. This is important information which is shared by you. This info is meaningful and important for everyone to increase our knowledge about it. Always keep sharing this kind of information. Thank you. Read more info about Cloud PBX Baton Rouge

  15. I was thinking to starting a business. But i don't have any idea on this. Then i found drops of knowledge. Here i got all the professional mentors who helped me so much. If you are also a new bee then you can seek from them.

  16. It is a proficient article that you have shared here.Devops Company I got some different kind of information from your article which I will be sharing with my friends who need this info. Thankful to you for sharing an article like this.

  17. Valuable info, this information is excellent and essential for everyone. I am very very thankful to you for providing this kind of information. Massey Ferguson Tractor

  18. oh good! you bring us good content I'm glad to read. keep sharing it. - eicher 485

  19. Thanks for publishing such excellent information. You are doing such a great job. This information is very helpful for everyone. Keep it up. Thanks for sharing it once again. John Deere Tractor