Lab overview
In this sequence of labs, you'll build a multi-server file system called Yet Another File System (yfs) in the spirit of Frangipani. At the end of all the labs, your file server architecture will look like this:
You'll write a file server process, labeled yfs above, using the FUSE toolkit. Each client host will run a copy of yfs. Each yfs will create a file system visible to applications on the same machine, and FUSE will forward application file system operations to yfs. All the yfs instances will store file system data in a single shared "extent" server, so that all client machines will see a single shared file system.
This architecture is appealing because (in principle) it shouldn't
slow down very much as you add client hosts. Most of the complexity is
in the per-client yfs program, so new clients make use of
their own CPUs. The extent server is shared, but hopefully it's simple
and fast enough to handle a large number of clients. In contrast, a
conventional NFS server is pretty complex (it has a complete file
system implementation) so it's more likely to be a bottleneck when
shared by many NFS clients.
Lab assignments
Lab 1 - Lock Server
Lab 2 - Basic File Server
Lab 3 - MKDIR, UNLINK, and Locking
Lab 4 - Caching Lock Server
Lab 5 - Caching Extent Server + Consistency
For the rest of the assignments, you may pick one of the following two options:
Option 1:
Lab 6 - Paxos
Lab 7 - Replicated lock server
Option 2:
Team project
Collaboration Policy
You must write all the code you hand in for the lab assignments, except for code that we give you as part of the assignment. You are not allowed to look at anyone else's solution, and you are not allowed to look at code from previous years. You may discuss the assignments with other students, but you may not look at or copy each others' code. Violations of academic integrity will result in serious disciplinary actions according to NYU GSAS policy.Programming Environment
You are required to do all labs on the class virtual machine (based on Ubuntu Linux). To get the virtual machine running on your personal desktop or laptop, take the following steps.
- Step 1. Download the Virtualbox virtual machine monitor here. Choose the right binary to download according to the type of operating system running on your laptop.
- Step 2. Download the class virtual machine image here. This file is fairly large (1.2GB), so you need to be patient.
- Step 3. Install and launch Virtualbox. On the Virtualbox application toolbar, under the Menu item "File", click on "Import appliance...", and choose the previously downloaded oneiric.ova file when prompted.
- Step 4. After importing, start the virtual machine named "oneiric" and log in using the username and password given in the first class (also available in Piazza). Change your password (optionally).
If you already have a Linux desktop or laptop, you may directly run programming labs there by setting up FUSE. To install FUSE, type "sudo aptitude install libfuse2 fuse-utils libfuse-dev ". Then add yourself to the FUSE group by typing "sudo adduser {your_user_name} fuse". Note that the official environment for the labs is still the virtual machine, and that's where we will be testing your code. If you are running on your own Linux machine, note that we don't have the energy or expertise to debug any problems you might have.
There has been at least one report that running the code in virtual machine is much slower than running it directly on hardware, and as a result, RPCs timeout more often even when RPC_LOSSY is unset. Be on the lookout for errors related to this.
Aids for working on labs
There are a number of resources available to help you with the lab portion of this course:- C++ Standard Template Library:
You will use C++ STL classes such as map and string a lot, so it's worth while to be familiar with the methods they offer; have a look here.
- pthreads:
All the labs use the POSIX threads API (pthreads). A comprehensive guide to programming with pthreads can be found here: http://www.llnl.gov/computing/tutorials/pthreads/
- FUSE:
The labs use the FUSE interface to plug the lab file system into the operating system. See the FUSE website for more information.
- Debugging and Core files:
printf statements are often the best way to debug. You may also want to use gdb, the GNU debugger. You may find this gdb reference useful. Below are a few tips.
If your program crashes and leaves a core dump file, you can see where the crash occured with gdb program core, where program is the name of the executable. Type bt to examine the call stack at the time of the crash.
If you know your program is likely to have a problem, you can run it with gdb from the beginning, using gdb program. Then type run.
If your program is already running (or it is hard to start with gdb due to complex start-up scripts), you can attach gdb to it by typing gdb program 1234, where 1234 is the process ID of your running program.
While in gdb, you can set breakpoints (use the gdb command b) to stop the execution at specific points, examine variable contents (print ...), get a list of threads (info threads), switch threads (thread X), etc.
To apply a given gdb command to all threads in your program, prepend thread apply all to your command. For example, thread apply all bt dumps the backtrace for all threads.
Look at the GDB manual for full documentation.