1.3. HARDWARE CONCEPTS Even though

1.3. HARDWARE CONCEPTS

Even though all distributed systems consist of multiple CPUs, there are several different ways the hardware can be organized, especially in terms of how they are interconnected and how they communicate. In this section we will take a brief look at distributed system hardware, in particular, how the machines are connected together. In the next section we will examine some of the software issues related to distributed systems.
Various classification schemes for multiple CPU computer systems have been proposed over the years, but none of them have really caught on and been widely adopted. Probably the most frequently cited taxonomy is Flynn's (1972), although it is fairly rudimentary. Flynn picked two characteristics that he con-
sidered essential: the number of instruction streams and the number of data streams. A computer with a single instruction stream and a single data stream is called SISD. All traditional uniprocessor computers (i.e., those having only one CPU) fall in this category, from personal computers to large mainframes.
The next category is SIMD, single instruction stream, multiple data stream. This type refers to array processors with one instruction unit that fetches an instruction, and then commands many data units to carry it out in parallel, each with its own data. These machines are useful for computations that repeat the same calculation on many sets of data, for example, adding up all the elements of 64 independent vectors. Some supercomputers are SIMD.
The next category is MISD, multiple instruction stream, single data stream. No known computers fit this model. Finally, comes MIMD, which essentially means a group of independent computers, each with its own program counter, program, and data. All distributed systems are MIMD, so this classification system is not tremendously useful for our purposes.
Although Flynn stopped here, we will go further. In Fig. 1-4, we divide all MIMD computers into two groups: those that have shared memory, usually called multiprocessors, and those that do not, sometimes called multicomput-
ers. The essential difference is this: in a multiprocessor, there is a single virtual







SEC. 1.3 HARDWARE CONCEPTS 9
address space that is shared by all CPUs. If any CPU writes, for example, the value 44 to address 1000, any other CPU subsequently reading from its address 1000 will get the value 44. All the machines share the same memory.


MIMD

Parallel and distributed computers

Tightly Loosely
coupled coupled



Multiprocessors Multicomputers
(shared Memory) (private memory)






Bus Switched Bus Switched

Sequent, Ultracomputer, Workstations Hypercube,
Encore RP3 on a LAN Transputer

Fig. 1-4. A taxonomy of parallel and distributed computer systems.

In contrast, in a multicomputer, every machine has its own private memory. If one CPU writes the value 44 to address 1000, when another CPU reads address 1000 it will get whatever value was there before. The write of 44 does not affect its memory at all. A common example of a multicomputer is a collection of personal computers connected by a network.
Each of these categories can be further divided based on the architecture of the interconnection network. In Fig. 1-4 we describe these two categories as bus and switched. By bus we mean that there is a single network, backplane, bus, cable, or other medium that connects all the machines. Cable television uses a scheme like this: the cable company runs a wire down the street, and all the subscribers have taps running to it from their television sets.
Switched systems do not have a single backbone like cable television. Instead, there are individual wires from machine to machine, with many dif-
ferent wiring patterns in use. Messages move along the wires, with an explicit switching decision made at each step to route the message along one of the out-
going wires. The worldwide public telephone system is organized in this way.
Another dimension to our taxonomy is that in some systems the machines are tightly coupled and in others they are loosely coupled. In a tightly-coupled






10 INTRODUCTION TO DISTRIBUTED SYSTEMS CHAP. 1
system, the delay experienced when a message is sent from one computer to another is short, and the data rate is high; that is, the number of bits per second that can be transferred is large. In a loosely-coupled system, the opposite is true: the intermachine message delay is large and the data rate is low. For example, two CPU chips on the same printed circuit board and connected by wires etched onto the board are likely to be tightly coupled, whereas two computers connected by a 2400 bit/sec modem over the telephone system are certain to be loosely coupled.
Tightly-coupled systems tend to be used more as parallel systems (working on a single problem) and loosely-coupled ones tend to be used as distributed systems (working on many unrelated problems), although this is not always true. One famous counterexample is a project in which hundreds of computers all over the world worked together trying to factor a huge number (about 100 digits). Each computer was assigned a different range of divisors to try, and they all worked on the problem in their spare time, reporting the results back by electronic mail when they finished.
On the whole, multiprocessors tend to be more tightly coupled than multicomputers, because they can exchange data at memory speeds, but some fiberoptic based multicomputers can also work at memory speeds. Despite the vagueness of the terms "tightly coupled" and "loosely coupled," they are useful concepts, just as saying "Jack is fat and Jill is thin" conveys information about girth even though one can get into a fair amount of discussion about the concepts of "fatness" and "thinness."
In the following four sections, we will look at the four categories of Fig. 1-4 in more detail, namely bus multiprocessors, switched multiprocessors, bus multicomputers, and switched multicomputers. Although these topics are not directly related to our main concern, distributed operating systems, they will shed some light on the subject because as we shall see, different categories of machines use different kinds of operating systems.

1.3.1. Bus-Based Multiprocessors

Bus-based multiprocessors consist of some number of CPUs all connected to a common bus, along with a memory module. A simple configuration is to have a high-speed backplane or motherboard into which CPU and memory cards can be inserted. A typical bus has 32 or 64 address lines, 32 or 64 data lines, and perhaps 32 or more control lines, all of which operate in parallel. To read a word of memory, a CPU puts the address of the word it wants on the bus address lines, then puts a signal on the appropriate control lines to indicate that it wants to read. The memory responds by putting the value of the word on the data lines to allow the requesting CPU to read it in. Writes work in a similar way.
Since there is only one memory, if CPU A writes a word to memory and







SEC. 1.3 HARDWARE CONCEPTS 11
then CPU B reads that word back a microsecond later, B will get the value just written. A memory that has this property is said to be coherent. Coherence plays an important role in distributed operating systems in a variety of ways that we will study later.
The problem with this scheme is that with as few as 4 or 5 CPUs, the bus will usually be overloaded and performance will drop drastically. The solution is to add a high-speed cache memory between the CPU and the bus, as shown in Fig. 1-5. The cache holds the most recently accessed words. All memory requests go through the cache. If the word requested is in the cache, the cache itself responds to the CPU, and no bus request is made. If the cache is large enough, the probability of success, called the hit rate, will be high, and the amount of bus traffic per CPU will drop dramatically, allowing many more CPUs in the system. Cache sizes of 64K to 1M are common, which often gives a hit rate of 90 percent or more.



CPU CPU CPU Memory
Cache Cache Cache




Bus

Fig. 1-5. A bus-based multiprocessor.
However, the introduction of caches also brings a serious problem with it. Suppose that two CPUs, A and B, each read the same word into their respective caches. Then A overwrites the word. When B next reads that word, it gets the old value from its cache, not the value A just wrote. The memory is now incoherent, and the system is difficult to program.
Many researchers have studied this problem, and various solutions are known. Below we will sketch one of them. Suppose that the cache memories are designed so that whenever a word is written to the cache, it is written through to memory as well. Such a cache is, not surprisingly, called a writethrough cache. In this design, cache hits for reads do not cause bus traffic, but cache misses for reads, and all writes, hits and misses, cause bus traffic.
In addition, all caches constantly monitor the bus. Whenever a cache sees a write occurring to a memory address present in its cache, it either removes that entry from its cache, or updates the cache entry with the new value. Such a cache is called a snoopy cache (or sometimes, a snooping cache) because it is always "snooping" (eavesdropping) on the bus. A design consisting of snoopy write-through caches is coherent and is invisible to the programmer. Nearly all bus-based multiprocessors use either this architecture or one closely related to it.






12 INTRODUCTION TO DISTRIBUTED SYSTEMS CHAP. 1
Using it, it is possible to put about 32 or possibly 64 CPUs on a single bus. For more about bus-based multiprocessors, see Lilja (1993).

1.3.2. Switched Multiprocessors

To build a multiprocessor with more than 64 processors, a different method is needed to connect the CPUs with the memory. One possibility is to divide the memory up into modules and connect them

Hasil (Bahasa Indonesia) 1:

1.3. HARDWARE KONSEP Meskipun semua sistem terdistribusi terdiri dari beberapa CPU, ada beberapa cara berbeda hardware dapat diatur, terutama dalam hal bagaimana mereka saling berhubungan dan bagaimana mereka berkomunikasi. Dalam bagian ini kita akan singkat melihat sistem terdistribusi hardware, khususnya, bagaimana mesin terhubung bersama-sama. Pada bagian berikutnya kita akan meneliti beberapa perangkat lunak isu-isu yang berkaitan dengan sistem terdistribusi. Berbagai skema klasifikasi untuk beberapa CPU komputer sistem telah diajukan selama bertahun-tahun, namun tidak satupun dari mereka benar-benar tertangkap dan telah diadopsi. Mungkin yang paling sering dikutip taksonomi adalah Flynn's (1972), meskipun cukup sederhana. Flynn mengambil dua karakteristik yang ia mem-sidered penting: jumlah instruksi sungai dan jumlah aliran data. Sebuah komputer dengan satu instruksi sungai dan aliran data tunggal yang disebut SISD. Semua komputer tradisional uniprocessor (yaitu, mereka memiliki hanya satu CPU) jatuh dalam kategori ini, dari komputer pribadi ke Mainframe besar. Kategori berikutnya adalah SIMD, satu instruksi stream, beberapa aliran data. Jenis ini merujuk kepada array prosesor dengan satu instruksi unit yang mengambil instruksi, dan kemudian memerintahkan banyak data unit untuk melaksanakan secara paralel, masing-masing dengan data sendiri. Mesin ini sangat berguna untuk perhitungan yang mengulang perhitungan yang sama pada banyak set data, misalnya, menambahkan semua elemen dari 64 independen vektor. Beberapa superkomputer adalah SIMD. Kategori berikutnya adalah MISD, beberapa instruksi stream, aliran data tunggal. Ada dikenal komputer cocok dengan model ini. Akhirnya, datang MIMD, yang pada dasarnya berarti sekelompok independen komputer, masing-masing dengan sendiri program counter, program dan data. Semua sistem terdistribusi yang MIMD, sehingga sistem klasifikasi ini tidak sangat berguna untuk tujuan kita. Meskipun Flynn berhenti di sini, kita akan pergi selanjutnya. Di gambar 1-4, kami membagi semua komputer MIMD menjadi dua kelompok: orang-orang yang telah berbagi memori, biasanya disebut multiprocessors, dan mereka yang tidak, kadang-kadang disebut multicomput -ERS. Perbedaan penting adalah ini: dalam multiprosesor, ada satu virtual SEC. 1.3 HARDWARE KONSEP 9ruang alamat yang dibagi oleh semua CPU. Jika CPU setiap menulis, misalnya, nilai 44 ke alamat 1000, CPU lain yang kemudian membaca dari alamat nya 1000 akan mendapatkan nilai 44. Semua mesin berbagi memori sama. MIMD Komputer paralel dan terdistribusi Erat longgarditambah digabungkanMultiprocessors Multicomputers(memori bersama) (pribadi memori)Bus beralih Bus beralihBerturut-turut, Ultracomputer, Hypercube workstation,Encore RP3 pada LAN TransputerGambar 1-4. Sebuah taksonomi dari sistem komputer paralel dan didistribusikan.Sebaliknya, dalam multicomputer, setiap mesin memiliki memori pribadi sendiri. Jika satu CPU menulis nilai 44 ke alamat 1000, Kapan lain CPU membaca alamat 1000 ini akan mendapatkan nilai berapa pun yang ada sebelum. Menulis 44 tidak mempengaruhi memori sama sekali. Contoh umum dari multicomputer adalah kumpulan terhubung dengan jaringan komputer. Setiap kategori dapat lebih lanjut dibagi berdasarkan pada arsitektur jaringan interkoneksi. Di gambar 1-4 kami menggambarkan kedua Kategori sebagai bus dan beralih. Dengan bus kami berarti bahwa ada dalam satu jaringan, backplane, bus, kabel, atau media lain yang menghubungkan semua mesin. Televisi kabel menggunakan skema seperti ini: perusahaan TV kabel berjalan kawat di jalan, dan semua pelanggan memiliki keran berjalan untuk itu dari televisi mereka. Beralih sistem tidak memiliki tulang punggung tunggal seperti televisi kabel. Sebaliknya, ada individu kabel dari mesin ke mesin, dengan banyak dif-ferent pola kabel digunakan. Pesan bergerak sepanjang kabel, dengan keputusan switching eksplisit yang dilakukan di setiap langkah untuk merutekan pesan sepanjang salah satu dari out-kabel pergi. Sistem telepon umum di seluruh dunia diselenggarakan dengan cara ini. Dimensi lain taksonomi kami adalah bahwa pada beberapa sistem mesin digabungkan erat dan orang lain yang mereka longgar ditambah. Di-digabungkan erat 10 PENGENALAN SISTEM TERDISTRIBUSI BAB 1sistem, keterlambatan mengalami ketika pesan dikirim dari komputer lain singkat, dan kecepatan data tinggi; itu adalah, jumlah bit per detik yang dapat ditransfer besar. Dalam sistem longgar-coupled, sebaliknya benar: penundaan pesan intermachine besar dan data tingkat rendah. Misalnya, dua chip CPU yang sama dicetak papan sirkuit dan terhubung dengan kabel terukir ke papan cenderung akan digabungkan erat, sedangkan dua komputer yang terhubung dengan modem 2400 bit/sec atas sistem telepon tertentu untuk longgar digabungkan. Sistem digabungkan erat cenderung digunakan lebih sebagai sistem paralel (bekerja pada satu masalah) dan longgar-coupled yang cenderung digunakan sebagai sistem yang terdistribusi (bekerja pada banyak masalah yang tidak terkait), meskipun hal ini tidak selalu benar. Satu terkenal balik adalah sebuah proyek di mana ratusan komputer seluruh dunia bekerja bersama-sama berusaha faktor sejumlah besar (sekitar 100 digit). Setiap komputer ditugaskan berbagai pembagi untuk mencoba, dan mereka semua bekerja pada masalah di waktu luang mereka, melaporkan hasil kembali melalui surat elektronik ketika mereka selesai. Secara keseluruhan, multiprocessors cenderung lebih erat daripada multicomputers, karena mereka dapat bertukar data pada kecepatan memori, tetapi beberapa fiberoptic berdasarkan multicomputers juga dapat bekerja pada kecepatan memori. Meskipun ketidakjelasan istilah "digabungkan erat" dan "digabungkan secara longgar", mereka adalah konsep-konsep yang berguna, seperti mengatakan "Jack adalah lemak dan Jill tipis" menyampaikan informasi mengenai ketebalan meskipun satu dapat menjadi cukup banyak diskusi tentang konsep "kegemukan" dan "thinness." Dalam empat bagian berikut, kita akan melihat empat kategori dari Fig. 1-4 secara lebih rinci, yaitu bus multiprocessors, beralih multiprocessors, bus multicomputers, dan beralih multicomputers. Meskipun topik ini tidak secara langsung berhubungan dengan perhatian utama kami, sistem operasi didistribusikan, mereka akan menumpahkan cahaya pada subjek karena seperti yang akan kita lihat, kategori yang berbeda dari mesin menggunakan jenis sistem operasi. 1.3.1. bus berbasis Multiprocessors Multiprocessors berbasis bus terdiri dari beberapa jumlah CPU semua terhubung ke sebuah bus umum, bersama dengan modul memori. Konfigurasi yang sederhana adalah untuk memiliki backplane berkecepatan tinggi atau motherboard ke mana CPU dan memori kartu dapat dimasukkan. Bus khas memiliki 32 atau 64 alamat baris, baris data 32 atau 64, dan mungkin 32 atau lebih kontrol jalur, yang beroperasi secara paralel. Untuk membaca kata-kata memori, CPU menempatkan alamat kata ini ingin pada baris alamat bus, kemudian menempatkan sinyal pada garis kontrol yang sesuai untuk menunjukkan bahwa mereka ingin membaca. Memori merespon dengan menempatkan nilai kata pada baris data agar meminta CPU untuk membacanya di. Menulis bekerja dengan cara yang sama. Karena ada hanya satu memori, jika CPU A menulis sebuah kata untuk memori dan SEC. 1.3 HARDWARE KONSEP 11kemudian CPU B membaca kata kembali satu mikrodetik kemudian tersebut, B akan mendapatkan nilai saja ditulis. Memori yang memiliki properti ini dikatakan koheren. Koherensi memainkan peran penting dalam sistem operasi didistribusikan dalam berbagai cara-cara yang akan kita pelajari nanti. Masalah dengan skema ini adalah bahwa dengan sedikit selama 4 atau 5 CPU, bus akan biasanya dipenuhi dan kinerja akan turun drastis. Solusinya adalah untuk menambah memori cache berkecepatan tinggi antara CPU dan bus, seperti yang ditunjukkan dalam gambar 1-5. Cache memegang yang paling baru-baru ini diakses kata-kata. Semua permintaan memori pergi melalui cache. Jika diminta kata dalam cache, cache sendiri menanggapi CPU, dan tidak ada bus permintaan dibuat. Jika cache cukup besar, probabilitas keberhasilan, disebut tingkat hit, akan menjadi tinggi, dan jumlah lalu lintas bus per CPU akan turun secara dramatis, memungkinkan banyak CPU yang lebih dalam sistem. Ukuran cache 64K 1 m umum, yang sering memberikan tingkat hit 90 persen atau lebih. CPU CPU CPU MemoryCache Cache CacheBus Gambar 1-5. Bus berbasis multiprosesor. Namun, pengenalan cache juga membawa masalah serius dengan itu. Anggaplah bahwa dua CPU, A dan B, masing-masing membaca kata yang sama ke cache masing-masing. Maka A akan menimpa kata. Ketika B berikutnya membaca kata itu, ia mendapat nilai lama dari cache, bukan nilai A hanya menulis. Memori sekarang tidak koheren, dan sistem ini sulit untuk program. Many researchers have studied this problem, and various solutions are known. Below we will sketch one of them. Suppose that the cache memories are designed so that whenever a word is written to the cache, it is written through to memory as well. Such a cache is, not surprisingly, called a writethrough cache. In this design, cache hits for reads do not cause bus traffic, but cache misses for reads, and all writes, hits and misses, cause bus traffic. In addition, all caches constantly monitor the bus. Whenever a cache sees a write occurring to a memory address present in its cache, it either removes that entry from its cache, or updates the cache entry with the new value. Such a cache is called a snoopy cache (or sometimes, a snooping cache) because it is always "snooping" (eavesdropping) on the bus. A design consisting of snoopy write-through caches is coherent and is invisible to the programmer. Nearly all bus-based multiprocessors use either this architecture or one closely related to it. 12 INTRODUCTION TO DISTRIBUTED SYSTEMS CHAP. 1Using it, it is possible to put about 32 or possibly 64 CPUs on a single bus. For more about bus-based multiprocessors, see Lilja (1993). 1.3.2. Switched Multiprocessors To build a multiprocessor with more than 64 processors, a different method is needed to connect the CPUs with the memory. One possibility is to divide the memory up into modules and connect them