Development of a Distributed Network Block Device for Unicast Networks

Albert-Ludwigs-University Freiburg Faculty of Applied Sciences Chair of Communication Systems Prof. Dr. Gerhard Schneider Dirk von Suchodoletz, advisor

Development of a Distributed Network Block Device for Unicast Networks

Bachelor Thesis by Vito Di Leo Gremmelspacher

Abstract

Diskless clients are computers that do not have an operating system installed locally on a hard-disk but rather import it from a server through the network. E\ufb03ciently providing diskless clients with an operating system is crucial to the overall performance and scalability of networks containing many such clients. The Distributed Network Block Device (DNBD) is a solution for this task under multicast networks, specially in wireless environments. In this thesis we develop, based on DNBD, another network block device for GNU/Linux operating systems suited for unicast networks. The result should ideally provide a simple, robust, scalable and high-performance infrastructure to deliver operating systems to many diskless clients on big networks such as found on universities.

June 2007

Trusted by over 1 million members

Try Scribd FREE for 30 days to access over 125 million titles without ads or interruptions! Start Free Trial Cancel Anytime.



























arung Erkl \u00a8

Hiermit erkl\u00a8are ich, dass ich diese Abschlussarbeit selbst\u00a8andig verfasst ha uber hinaus erkl\u00a8are ich, dass diese Abschlussarbeit nicht, auch nicht auszugsweise,ur bereits eine andere f \u00a8ufung Pr \u00a8 angefertigt wurde.





























Contents 1

Introduction

1.1

1.2

1.3 2

1

ComputingPlatforms . . . . . . . . . . . . . . . . . . . . . . . . . 1 . 1.1.1

PersonalComputers . . . . . . . . . . . . . . . . . . . . . 1.

1.1.2

DisklessClients . . . . . . . . . . . . . . . . . . . . . . . . 2 .

1.1.3

ThinClients . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.4

SmartTerminals . . . . . . . . . . . . . . . . . . . . . . . .2

1.1.5

Notes on Computing Platforms . . . . . . . . . . . . . .3. .

RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.1

The Distributed Network Block Device . . . . . . . . . 4. . .

1.2.2

Other Network Block Devices . . . . . . . . . . . . . . .6. .

Goals and Paper Structure . . . . . . . . . . . . . . . . . . . . . 7 . .

Design

9

2.1

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

2.2

Network Storage and Write Semantics

. . . . . . . . . . . . .9. . .

2.6.1 RequestProcessing . . . . . . . . . . . . . . . . . . . . . . . 17

3

2.7

Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.8

DeferredFeatures . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Implementation

21 22

23

3.1

Protocol and Global Data Structures . . . . . . . . . . . . . . . . .

3.2

ServerApplication . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

3.3 ClientDriver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

23















4

Experiments

29

4.1 TheSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5

Conclusion

37

A. Setup Example

39

List of Figures

41

References

43

















1

1

Introduction

Introduction

1.1

Computing Platforms

There are several paradigms to provide a computing platform to users. By this

All platforms we introduce shortly are equipped with the interface mentioned

The differences between platforms result from the capabilities and limitations

1.1.1

Personal Computers

To put it concisely, a personal computer is a device that has a microprocessor

















1

Introduction

1.1.2

Diskless Clients

These are personal computers with no persistent storage. Just like PCs, and

Diskless clients usually boot from a boot-server using Etherboot or PXE. Other

1.1.3

Thin Clients

A thin client is a small device with the same architecture of a PC but only a

The idea behind thin clients is to have a small operating system and lightweight,

Thin Clients offer the ergonomics of smart terminals and some advantages of

















1

Introduction

An example of smart terminals are X-Terminals, which run an X-Server and can display X Window System applications. Another example are Sun Microsystems SunRays, which use the Appliance Link Protocol (ALP) instead of the X Display Protocol. Since smart terminals require relatively little hardware and processing power

Despite the responsiveness penalty incurred by transporting all user input from

1.1.5

Notes on Computing Platforms

There is no single platform that is suited for every field of application. Let’s

What would it take, on the other hand, to run 20 X-Terminals? We would

one operating system and one host provide a computing platform for 20 users to

















1

Introduction

OS Apps

OS Personal Computer

Thin Client

OS

Diskless Client

Apps Homes

OS Smart Terminal

Figure 1: Special attention must be given to Diskless Clients. They allow to

centralize storage and distribute processing power.

networking will become standard, and storing operating systems and applications

1.2

Related Work

Our project is mainly based on Thorsten Zitterell’s Distributed Network Block Device (DNBD), which in turn borrows some concepts from other NBDs. Below

















1

Introduction

Personal

Diskless

Thin

Smart

Computer

Client

Client

Terminal

general purpose processing

very strong

strong

weak

none

local persistent storage

big

none

small

none

peripheral devices

full support

full supportlimited

very limited

multimedia playback

fast

fast

moderate

very slow

stand-alone solution

yes

no

no

no

Table 1: Characteristics of different computing platforms.

As opposed to switched, wired networks, wireless networks are based on a lossy

1 times. This would lead to bandwidth consumption in the order of O(nm). We would experience this problem in a much larger scale if we where to use a n

−

To make use of the intrinsic communications overhead of wireless networks,

Under certain circumstances pre-caching leads to bursts – synchronized requests



















1

Introduction

a)

b) (t1) firs t request

(t1 ) firs t request

t (t2) firs reply

Client 1

t (t2) firs reply

Client 1

c on d (t 3) se t s reque

(t4 ) se

cond reply

Server

Server (t3) pre-cache

Client 2

Client 2

c)

d) (t1 ) firs t request

t (t2) firs reply

Client 1

(t2) buffer

d secon (t 1+ ) st reque

(t3 ) fi rst reply

Client 1

d secon (t 1+ ) st reque ε

Client 2

(t1) firs t request

(t3 ) se

cond reply

Server

ε

Server

Client 2

Figure 2: a) Client 2 does not pre-cache the server’s reply and sends its own

request. b) Pre-caching avoids the superfluous request. c) Both clients send the



















1

•

Introduction

NBD: The original

Network Block Device, developed by Pavel Macheck, was the first NBD for the Linux kernel and still is the only one include in the

root@server# nbd-server 5000 /dev/sda1 root@client# nbd-client server 5000 /dev/nb0 •

•

•

ENBD: The Enhanced

Network Block Device, by Peter T. Breuer [18], developed to provide failure-mode and redundancy features. In contrast to

ANBD: Another

Network Block Device, from Louis D. Langholtz [19], pro vides additional features beyond the original NBD, including server-side copy-

GNBD: The Global

Network Block Device is part of the Global File Sys tem (GFS), an open source cluster file system for enterprise deployments. GFS was originally developed at the University of Minnesota. Today it is





































2

2

Design

Design This section is divided into eight subsections. In the first subsection we list

2.1 •

Requirements Simplicity: We

look for simplicity both in the realization of the project well as in the mode of operation of the final product. Setting up a DNBD2



















2

Design

Server or Network Storage

Applications

Network

Client

ftp, rsync

Applications user space

Virtual File System

SM B D NB

kernel space

Virtual File System

D

NFS

AoE, FCIP, iSCSI

Figure 3: Data exchange at different layers.

As show in figure 3, there are different layers at which clients can bind data from





















2

a)

Ext3 (read-write) /sbin/init

b)

UnionFS (read-write) /sbin/init /dev/tty1 /var/run/utmp

/sbin/init

cowloop /var/run/utmp

TmpFS (read-write)

Ext3 or SquashFS (read-only)

Design





















2

Design

thus created at boot-time and lost at shutdown. Being so, it makes sense to move

Write semantics on the client can be provided by copy-on-write (COW) either

2.3

ExportedDatasets Dataset3





















2

2.4

Design

Networking

DNBD2 will use UDP for data transfer even though it is unreliable by definition

Before blessing this design choice we ran some simple tests in a real-world





















2

Design

a dat S1

S2 dat a

Figure 6: In the initial setup clients only communicate with S1 . A second server,

S2 ,

is connected to the network. Without human intervention this constellation remains unchanged. Neither does S2 talk to the clients, nor do the clients try to discover S2 .

DNBD2 adopts these concepts from DNBD, but has to deal with the question





















2

a)

Design

b) S2 active

S1

se

r

r ve

s?

} 1 {S

se

{S rv er s?

1}

se

r

r ve

S1

s? 1 {S

re qu es

ts

,S

}

2

se rv e

{S

1

u

S

2}

rs ?

q re

,

t es

s























2

Design

is a reliable way to automatically find out if

S1

is really gone or only temporarily





















2

Design

of a driver is more difficult than that of an application. Our driver consists of a

2.6.1

RequestProcessing





















2

Design

2.6.2

Load Balancing and Emergency Mode step1 step2

start emergency mode no





















2

Design

When the driver is started, it marks an arbitrarily server on its list as active,





















2

Design

2.6.3

Sysfs Interface

Sysfs is a ram-based file system that provides means to export kernel data























2

Design





















2

Design





















3

Implementation





















3

Implementation





















3

Implementation





















int void dnbd2_rx_loop(void dnbd2_hb_timer(unsigned *data); long /*are receive-thread arg); */ Linux terface. at not the affect emergency time In other of addition this applications, mode writing. there (figure is We an reboot 10) programmed include cyimplemented. cfile, les are against kernel/dnbd2.h faster, Reading and, 2.6.18 there which kernel/core.h iscontains because no needis this the for half is a the voidused dnbd2_send_request(struct dnbd2_to_timer(unsigned long request *req, dnbd2_device_t *dev); version sacrificial declarations the way computer. of by tothe the understand data diskless Setting structures clients up theUML driver. that onarg); was which represent notwe easy would DNBD2 but well later devices worth test the the and driver. while the servers –Thanks we void dnbd2_end_request(struct request *req, success); to [10], must they communicate have making crashed thiswith. itdriver around Most work a functions 100 with times. therevolve latest The int problem Kernel around should with these UML data not is be structures. that difficult. it is slow. Our Let’s driver Once now describe we has got2042 tothe the lines components phase of where code, and DNBD’s speed show was has some a issue, 2331. of we their started functions to testprototypes: the driver













withactive smoothed the rency. spinlocks Inside RTT, server’s the (or send_queue_lock driver SRTT SRTTwe for gets have short) updated. and tofor pending_queue_lock protect each To solve server the pending this in )its problem because list. queue, This we timers thedecided value send areis queue, not to a usethe measurement heartbeat servers allowed list to replies sleep. and of a the to We server’s reset device mustthe take overall configuration SRTT. care speed. When to disable It (VID, anisHB used software RID, returns to etc). choose interrupts its RTT which iswhen used server as taking to thetalk a to, orSRTT. new spinlock when outside Totomake retransmit thethis timers value aorrequest. else morethey significant might interrupt the server usreads and deadlock a random byblock taking fromspinlock the the dataset a second before time. answering to an HB.













it socket. a from theBoth list. these The problem operations is that can the sleep driver so wenotices must protect this event the in servers the requeue list with a mutex. timer. WeThis muststructure take servers_mutex has its own tomutex delete(the servers_mutex server but ) because we are not the allowed driver 8 might to sleep need in the to timer. modifyThe it while solution holding is to enqueue config_mutex this. task into the kernel’s shared workqueue. The tasks in this workqueue are completed by a kernel thread in the













formed enjoy servers, them the which role withwe of great an will OpenSLX care call srv1, and248 server suspicion. srv2 and from srv3. The which performance Their the hardware diskless clients aconfigurations system booted. that sits The were • srv1: Dual AMD Opteron 2.2 GHz, 6 GB RAM and of gigabit Ethernet. on top the three following: servers of so many exported layers the of same abstraction dataset, and a 3524075520-byte uses so many components (3.3 GiB) squashfs can vary a comsrv2: Dual Intel Xeon 5160 kernel, 3.0 GHz, 8 GB and gigabit Ethernet. lot.• Inside pressed image and of outside a custom the Suse-10.2. Linux Onboth all three onRAM clientservers and this server-side, file was located there are in many things tmpfs, i.e., completely that could in RAM. influence Thisthe image performance served both oftoDNBD2: boot the clients and to srv3: Dual AMD Opteron 275 2.2 GHz, 8 GB RAM and gigabit Ethernet.













with theThese switch. diskless clients. squashfs four Unfortunately, image switches mentioned where as we connected above will see as rootfs later, to a fifth exported the gigabit maximum over switch DNBD2. throughput to which The was srv1 was also always average aboot connected. littletime less was than Srv2 29 1 Gbit/s. seconds and srv3 We from were only the located had moment physical on another theaccess kernel network, to was theloaded diskless two routers until a away, but clients graphical andlogin each the first was hadfour displayed. a 1 switches Gbit/s We link so repeated to wethe couldn’t last the switch test figure with mentioned out NBD where andthe got (figure problem exactly 13).the was. As a result same boot time. of thisThis we could was expected not show, because as was the our amount intention, of data that automatic transferred load during













33the of srv3 % respectively, othree f the CPU DNBD2 time. theservers. retransmission Having We a tree rebooted data rates structure all 2.78 clients %, 1.7 in (to the % clear and server 0.9 the only %.buffer-caches) The makes actual sense and performance if one configured is goingistoa them export littletohigher aspeak lot ofbecause to datasets srv1 and the and not measured needs to switch to times find tothem other onlyquickly. tell servers us when Otherwise (setting the it would be advisable to change it for a bi-dimensional array of VIDs and RIDs.













balancing on to measure works; theand maximum second,throughput that it increases of thethe client. overall Wetransfer went to rate the location to the pool of srv3 of diskless and connected clients. We a high-end rebootednotebook the clients directly and settoto_percent it using a topatch 30 and cable. to_jiffies The standard to 5 on alltest of them. dd /dev/null that a client showed switches a throughput to another of 106.8 server MiB/s. if it is at lease 30 % and 5 jiffies faster than the active server. Then we started the command













clients and 3 Gbit/s automatically as expected. provide Before a decent blaming loadDNBD2 balancing. we tried In figure to get 17 those we can numbers see by other that mostmeans. of the time We transferred each serverthe wasdatasets communicating simultaneously with at using most 6scp clients, from each the servers tobeing optimum a different 4 and diskless the worst client, beingeach 12. In of which figure was 16 we connected see that the to atotal different work switch. was alsoThe divided throughput quite good wasbetween again limited all servers. to 1 Gbit/s. The overall Sure we performance could have tried decreased to prove%our 11.5 in comparison hypothesis by to the limiting last experiment, the flow rates when on each all clients serverwere to 100 locked Mbit/s. to srv3, But this was but is neither bounded fun nor by the veryperformance relevant since ofgigabit the slowest networking server,is srv1. already becoming standard now. The answer to whether the throughput scales as expected, the second goal of this experiment, remains thus unknown.













30 seconds, but not whatto each the client detriment was doing of theduring overallthose performance. 90 seconds. WeFor conclude this purpose that the we have figureinterval heartbeat 18, which should shows contain to which a random server each component. client was talking to during that interval. What strikes the eye on this figure is that the clients switched servers more than once, in a very short interval, at around times 5, 35 and 65, before settling on one server. The interval between these events, about 30 seconds (matching the heartbeat request interval), makes us believe that this is not an oscillation phenomenon where clients switching to a new server displace other clients. We rather believe that this is a consequence of the simultaneous boot of the clients: If they are started at approximately the same time, they will also, on a somewhat synchronized manner, send heartbeats, reset the SRTTs associated with each server and very likely switch servers.













told to switch automatically to faster servers and then dd /dev/null was started on all clients simultaneously. We were not interested on when the clients finished reading /dev/vnbd0, but whether they ever did so. The purpose of this test was to show that a client could handle consecutive server failures, even to the point where all servers stop responding for some time. The order in which we stopped and started the servers is given in table 2. The test was successfully passed by all clients without any I/O errors.

























factory. We believe that we have fulfilled our goal of developing a network block device that suits diskless clients better than any other available alternative. It would have been nice, though, to implement caching on persistent storage, even in the advent of gigabit networking. But, alas, time was not on our side! Hopefully, one day, DNBD2 will prove itself in a large deployment of diskless clients.

























root@server1# root@server2# root@client# cp-a /etc/dnbd2 cd echo -a/media/cdrom/kernel 5001 5002 /media/cdrom/server >> . .how and export 192.168.0.1 the same cp dataset. and server2 Then IPserver.conf we 192.168.0.2, show to bind thatthat we dataset want theinapplications the client. We root@server1# root@server2# root@client# cd echo kernel server /data/bigfile /etc/dnbd2/dataset > 5002, dataset 192.168.0.2:5002 192.168.0.1:5001 >> server.conf assume listening that the oncdsource ports 5001 CD is and mounted respectively. on /media/cdrom on all computers. root@server1#make root@client# make; 17 echo make ; make >> install dataset install root@server1# echo 42 >> dataset













root@client# cd /sys/block/vnbd0 root@client# echo 192.168.0.1 5001 > server0/sock root@client# echo 192.168.0.2 5002 > server1/sock root@client# echo 17 > config/vid root@client# echo 42 > config/rid root@client# echo 1 > config/running













10

Flowcharts of the emergency start/stop functions . . . . . . . . . .

11

Flowchart of the server’s main part . . . . . . . . . . . . . . . . . .

19 21

12 DNBD2Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 13

Topology of the testing environment

. . . . . . . . . . . . . . . . .

30

14

Server maximum throughput . . . . . . . . . . . . . . . . . . . . . .

31

15

Call graph of dnbd2-server . . . . . . . . . . . . . . . . . . . . . .

16

Work distribution among servers . . . . . . . . . . . . . . . . . . . .

17

Monitoring client distribution among servers . . . . . . . . . . . . .

33

18

Monitoring the active server of each client . . . . . . . . . . . . . .

34

32 32





































Development of a Distributed Network Block Device for Unicast Networks

Recommend Documents