Please navigate to the bottom of the page for Table of Contents

Monday, January 21, 2013

Distributed vs Parallel computing


In this post I will provide a very high level overview of Distributed versus Parallel computing.

Distributed computing refers to the study of distributed systems to solve complex or time consuming problems, broken down to small tasks, across multiple computers (nodes) each of which has its own memory and disk.
In addition, the distributed system has additional constraints such as fault tolerance (individual nodes may fail), unknown structure (the network topology, etc.  may not be known or well defined) and decoupled (individual nodes may not have knowledge of entire system). The key to distributed computing is that there are many small nodes processing and executing tasks without knowing the broader system.

Parallel computing, on the other hand, is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel").

The same system may be characterized both as "parallel" and "distributed"; the processors in a typical distributed system run concurrently in parallel. The key difference between the two is that in parallel computing, all processors may have access to a shared memory to exchange information between processors and in distributed computing, each processor has its own private memory and information is exchanged by passing messages between the processors.

6 comments:

  1. This is very incorrect of course, confusing parallel processing with presence or absence of shared memory. Most and best parallel computing use MPI which is exactly passing messages. While shared-memory techniques like OpenMP had limited success.

    ReplyDelete
  2. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.

    Best Digital Transformation Services | DM Services | Austere Technologies

    ReplyDelete
  3. Hi Nikhil,

    In total awe…. So much respect and gratitude to you folks for pulling off such amazing blogs without missing any points on the Distributed vs Parallel computing. Kudos!


    I know that you may answer as already in such case that you noticed that I open case 4914182641 in support center and that I need to wait their answer. But when I opened it, I start to search such problems with overdue payments on forums. And I found that everytime people said that they not get answer from support (from 48 hours to 1 week). I'm very affraid that in my case support can not react same as I read it from forums. And that my account can be closed. It's really little sum (approx. 8 EUR for february) and very big time of use (many years). I create domain's there and buckets. And it's very affraid to lost account such way. Can you, please, help me?

    Amazon S3 provides access to reliable and inexpensive data storage infrastructure. It is designed to make web-scale computing easier by enabling you to store and retrieve any amount of data, at any time, from within Amazon EC2 or anywhere on the web.

    But nice Article Mate! Great Information! Keep up the good work!

    Kind Regards,
    Kevin

    ReplyDelete
  4. Hi Mate,

    11/10!! Your blog is such a complete read. I like your approach with Distributed vs Parallel computing . Clearly, you wrote it to make learning a cake walk for me.

    We're currently in an infinite loop between sales and support, neither of whom seem to be able to understand a basic issue.

    We want to purchase some sizeable reserved instances but are told that the only way to pay is all at once with a credit card. No split payments, no offer to pay by check, no offer to pay by ACH, no offer to pay by wire. AWS Training

    Can someone explain to me how AWS serves enterprises if they only accept consumer methods of payment?

    Super likes !!! for this amazing post. I thinks everyone should bookmark this.

    Kind Regards,

    Irene Hynes

    ReplyDelete
  5. Merhaba,


    Three cheers to you ! Hooray!!! I feel like I hit the jackpot on Distributed vs Parallel computing


    I logged into console.aws.amazon.com today and now every few seconds a popup says I need to reload (and re-login). This has happened before after a long time of inactivity, but now it's happening so often I can't look at anything without having to reload and login. AWS Tutorial





    THANK YOU!! This saved my butt today, I’m immensely grateful.


    Obrigado,
    Ajeeth

    ReplyDelete