Please navigate to the bottom of the page for Table of Contents

Monday, January 21, 2013

Distributed vs Parallel computing

In this post I will provide a very high level overview of Distributed versus Parallel computing.

Distributed computing refers to the study of distributed systems to solve complex or time consuming problems, broken down to small tasks, across multiple computers (nodes) each of which has its own memory and disk.
In addition, the distributed system has additional constraints such as fault tolerance (individual nodes may fail), unknown structure (the network topology, etc.  may not be known or well defined) and decoupled (individual nodes may not have knowledge of entire system). The key to distributed computing is that there are many small nodes processing and executing tasks without knowing the broader system.

Parallel computing, on the other hand, is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel").

The same system may be characterized both as "parallel" and "distributed"; the processors in a typical distributed system run concurrently in parallel. The key difference between the two is that in parallel computing, all processors may have access to a shared memory to exchange information between processors and in distributed computing, each processor has its own private memory and information is exchanged by passing messages between the processors.


  1. This is very incorrect of course, confusing parallel processing with presence or absence of shared memory. Most and best parallel computing use MPI which is exactly passing messages. While shared-memory techniques like OpenMP had limited success.

    1. I am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts. Python Projects for Students Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account. Project Center in Chennai

  2. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    Best Digital Transformation Services | DM Services | Austere Technologies

  3. Hi Nikhil,

    In total awe…. So much respect and gratitude to you folks for pulling off such amazing blogs without missing any points on the Distributed vs Parallel computing. Kudos!

    I know that you may answer as already in such case that you noticed that I open case 4914182641 in support center and that I need to wait their answer. But when I opened it, I start to search such problems with overdue payments on forums. And I found that everytime people said that they not get answer from support (from 48 hours to 1 week). I'm very affraid that in my case support can not react same as I read it from forums. And that my account can be closed. It's really little sum (approx. 8 EUR for february) and very big time of use (many years). I create domain's there and buckets. And it's very affraid to lost account such way. Can you, please, help me?

    Amazon S3 provides access to reliable and inexpensive data storage infrastructure. It is designed to make web-scale computing easier by enabling you to store and retrieve any amount of data, at any time, from within Amazon EC2 or anywhere on the web.

    But nice Article Mate! Great Information! Keep up the good work!

    Kind Regards,

  4. Hi Mate,

    11/10!! Your blog is such a complete read. I like your approach with Distributed vs Parallel computing . Clearly, you wrote it to make learning a cake walk for me.

    We're currently in an infinite loop between sales and support, neither of whom seem to be able to understand a basic issue.

    We want to purchase some sizeable reserved instances but are told that the only way to pay is all at once with a credit card. No split payments, no offer to pay by check, no offer to pay by ACH, no offer to pay by wire. AWS Training

    Can someone explain to me how AWS serves enterprises if they only accept consumer methods of payment?

    Super likes !!! for this amazing post. I thinks everyone should bookmark this.

    Kind Regards,

    Irene Hynes

  5. Merhaba,

    Three cheers to you ! Hooray!!! I feel like I hit the jackpot on Distributed vs Parallel computing

    I logged into today and now every few seconds a popup says I need to reload (and re-login). This has happened before after a long time of inactivity, but now it's happening so often I can't look at anything without having to reload and login. AWS Tutorial

    THANK YOU!! This saved my butt today, I’m immensely grateful.


  6. I am very impressed after reading such a amazing post about cloud computing. Thanks for sharing such post. Cloud Computing Solutions Companies.

  7. This was nice and amazing and the given contents were very useful and the precision has given here is good.

    Apache Spark Training in Pune
    Spark Training Institute in Pune

  8. I was extremely dazzled by this post, this site has consistently been wonderful information. you particularly for such a fascinating post. oracle fusion finance training india

  9. Great job, You are providing the best knowledge. It is really helpful and factual information for us and everyone to increase knowledge. Continue sharing your data. Thank you. microsoft dynamics houston tx

  10. Excellent job, this is necessary information which is shared by you. This information is meaningful and very important for us to increase our knowledge about it. Always keep sharing this type of information. Thanks. small business phone service

  11. It’s my fortune to go to at this blog and realize out my required stuff that is also in the quality.
    interaction design agency

  12. I was searching for the finest article on DPU. Because one of my friends told me about fungible that they are providing the best dpu service at a cheap rate. So i think i should visit their website and learn more about DPU.

  13. I’m trampled by your contents carry on the wonderful work.
    logo creation services

  14. The United Kingdom, with its many waves of immigration, is extremely culturally diverse. The UK is not only one of the leading financial centers in the world, but many other strong industries, which together form the fifth largest economy in the world and the second largest economy in Europe after Germany.

  15. This is important information which is shared by you. This info is meaningful and important for everyone to increase our knowledge about it. Always keep sharing this kind of information. Thank you. Read more info about Cloud PBX Baton Rouge