-- Lab 2: Basic File Server --

---

Introduction

In this lab, you will embark on the actual file system implementation. In particular, you will start by getting the following FUSE operations to work:

• CREATE/MKNOD, LOOKUP, and READDIR

Recall that YFS has the following architecture.

We provide you with skeleton code for both the YFS and extent server modules above.

The YFS module implements the core file system logic. This module runs as a single process called yfs_client that supports a mountpoint on the local host. The code skeleton for this module consists of two pieces:

• The FUSE interface. This code lies in fuse.cc, and serves to translate FUSE operations from the FUSE kernel modules into YFS client calls. We provide you with much of the code needed to register with FUSE and receive FUSE operations; you will be mostly responsible for calling the appropriate methods on the yfs_client class and replying back over the FUSE interface.
• The YFS client. This code lies in yfs_client.{cc,h}. Unlike a traditional network file system client, the YFS client actually implements the file system logic! For example, when creating a new file, the yfs_client must add directory entries in the directory block itself (In a traditional network file system, the server performs this task). To fetch and store data blocks that contain file data or directory entries, yfs_client communicates with the extent server. Therefore, yfs_client should know how to interpret and manipulate extents in order to perform the appropriate file system operations.

The extent server acts as a centralized storage location for all the data representing your filesystem, much like a hard disk would. In later labs, you will serve the same file system contents on multiple hosts, each running its own yfs_client. The only way they can share data is by reading and writing the extent server. The extent server code skeleton consists of two pieces:

• Extent client. This code lies in extent_client.{cc,h}. This is a wrapper class for communicationg with extent server using RPCs.
• Extent server. The code lies in extent_server.{cc,h} and extent_smain.cc. The extent server manages a simple key-value store. The extent server simply stores entire files as strings (std::string), without interpreting the contents of those strings. It also stores information about the attributes of files. More concretely, your extent server should support put(key,value), get(key), getattr(key), and remove(key) RPCs.

Getting started

First, use a computer with FUSE and the FUSE libraries installed (see the Lab Overview for details).

You will need to merge your solution to Lab 1 with the new code we have supplied for Lab 2. To do this, first commit your solution to Lab 1:

% cd ~/ds-class/lab
% git commit -am 'my solution to lab1'
Created commit 254dac5: my solution to lab1
 3 files changed, 31 insertions(+), 6 deletions(-)

(Before you commit, you will need to explicitly git add any new files you added. Run git status to show all the files that were added or changed.)

Next, fetch the latest version of the course repository, then create a local branch called lab2 based on our lab2 branch, origin/lab2:

% git pull
remote: Generating pack...
[...]
% git checkout -b lab2 origin/lab2
Branch lab2 set up to track remote branch refs/remotes/origin/lab2.
Switched to a new branch "lab2"
%

The git checkout -b command shown above actually does two things: it first creates a local branch lab2 that is based on the origin/lab2 branch provided by the course staff, and second, it changes the contents of your lab directory to reflect the files stored on the lab2 branch. Git allows switching between existing branches using git checkout branch-name, though you should commit any outstanding changes on one branch before switching to a different one.

You will now need to merge the changes you made in your lab1 branch into the lab2 branch, as follows:

% git merge lab1

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g., if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a. To use YFS in this lab, you'll need start the extent server and yfs_client(s). If you are using the class machines, choose a port number that other students aren't using. For example, to run the extent server on port 3772, type this:

% cd ~/ds-class/lab
% ./extent_server 3772 &

Next, start the yfs_client process using three parameters: a unique mountpoint (that other students aren't using), the port number for the extent server, and the port number for the running lock server, which is not used in this lab. The mountpoint must be an empty directory that already exists. To start the yfs_client using mountpoint ./myfs and extent_server that listens on port 3772, type this:

% cd ~/ds-class/lab
% mkdir myfs
% ./yfs_client ./myfs 3772 3762 &

On Linux, you may need to run yfs_client as root, since normal users don't usually have mount permission. On FreeBSD, you can run sysctl vfs.usermount=1 as root, and then you should be able to run yfs_client as a normal user.

We provide you with the script start.sh to automatically start extent_server and yfs_client and stop.sh to kill previously started processes. Actually, start.sh starts two yfs_clients with ./yfs1 and ./yfs2 mountpoints respectively. (In this lab, you should only be concerned with one yfs_client. The next lab will use both yfs_clients.) Thus, instead of typing in commands manually as before, you can simply do:

% cd ~/ds-class/lab
% ./start.sh
% ./test-lab-2.pl ./yfs1
% ./stop.sh

The skeleton code implements only the GETATTR and STATFS operations, and so the file system you just mounted will not be useful at all. However, once you finish this lab, you should be able to run the Lab 2 tests successfully, which tests creating empty files, looking up names in a directory, and listing directory contents. Note: testing this lab on the command line using commands like touch will not work until you implement the SETATTR operation, which is not required until the next lab. For now, you should do your testing via the creat/open, lookup, and readdir system calls in a language like Perl, or simply use the provided test script.

Your Job

Your job is to implement the extent server, then implement the LOOKUP, CREATE/MKNOD, and READDIR FUSE operations in YFS. You must store the file system's contents in the extent server, so that in future labs you can share one file system among multiple servers. For our labs, it is okay to implement a simple extent server that stores data only in memory; this means that if you restart it, all the data previously stored will be lost.

On some systems, FUSE uses the MKNOD operation to create files, and on others, it uses CREATE. The two interfaces have slight differences, but in order to spare you the details, we have given you wrappers for both that calls a single common routine called createhelper(). You should implement this routine.

As before, if your server passes our tester on the official class programming environment, you are done. If you have questions about whether you have to implement specific pieces of file system functionality, then you should be guided by the tester: if you can pass the tests without implementing something, then you do not have to implement it. For example, you don't need to implement the exclusive create semantics of the CREATE/MKNOD operation. You may modify or add any files you like, other than the tester script and the RPC library.

The Lab 2 tester is the test-lab-2.pl script. Run it with your YFS mountpoint as the argument. Here's what a successful run of test-lab-2.pl looks like:

% ./test-lab-2.pl ./yfs1
create file-yyuvjztagkprvmxjnzrbczmvmfhtyxhwloulhggy-18674-0
create file-hcmaxnljdgbpirprwtuxobeforippbndpjtcxywf-18674-1
...
Passed all tests!

The tester creates lots of files with names like file-XXX-YYY-Z and checks that they appear in directory listings.

Note that if you implemented at-most-once RPC correctly, the tests should pass with RPC_LOSSY set to 5 as well.

If test-lab-2.pl exits without printing "Passed all tests!", then it thinks something is wrong with your file server. For example, if you run test-lab-2.pl on the skeleton code we give you, you'll probably see an error message like this:

test-lab-2: cannot create /tmp/b/file-ddscdywqxzozdoabhztxexkvpaazvtmrmmvcoayp-21501-0 : No such file or directory

This error message appears because you have not yet assigned a method to handle the CREATE/MKNOD operation with FUSE. See the main() method in fuse.cc for examples on how to make this assignment.

Detailed Guidance

• Implementing the extent server: You will need to implement the extent server in extent_server.cc. There are four operations: put(key,value), get(key), getattr(key), and remove(key). The put and get RPCs are used to update and retrieve an extent's contents. The getattr RPC retrieves an extent's attributes. The attribute consists of the file size, last modification time (mtime), change time (ctime), and last access time (atime). Tracking this data in the extent server should be straightforward in the handlers for the put(key,value) and get(key) RPCs. Wikipedia has a succinct description of when these three times should be updated.
• Deciding on the file system representation:

  YFS should name all files and directories with a unique identifier (much like the i-node number in an on-disk UNIX file system). We have defined such an 64-bit identifier (called inum) in yfs_client.h. Since FUSE accesses each file and directory in the file system using a unique 32-bit identifier, we suggest you use the least significant 32-bits of inum as the corresponding FUSE identifier.

  When creating a new file or directory, you must assign a unique inum. The easiest thing to do is to pick a number at random, hoping that it will indeed be unique. (What's the collision probability as the number of files and directories grows?)

  YFS needs to tell whether a particular inum refers to a file or a directory. To do this, you should ensure that any 32-bit FUSE identifier (i.e. the lower 32-bit of inum) for a file has the most significant bit equal to one; likewise, that bit for a directory should be set to zero. The provided method yfs_client::isfile assumes this property holds for inum.

  Next, you must choose the format for storing and retrieving file system meta-data (i.e. file/directory attributes and directory contents). You do not need to store or retrieve file contents in this lab yet. A file or dir's attribute contains information such as the file's length, modification times. A directory's content contains a list of name to inum mappings. Thus, resolving a file's inum involves a series of lookups starting from the file system root (The root directory has a well-known FUSE id of 0x000000001).

  It is convenient to store and retrieve the entire dir content (or file content, for later labs) in the extent server using the file/dir's inum as the key to the extent server's put(key,value) and get(key) functions. However, since YFS must also be able to retrieve the file attribute information based on the file/dir's inum, we ask you to implement a separate function, called extent_server::getattr(key), to retrieve a file/dir's attribute based on its inum.

• Implementing a FUSE file system:

  For these labs, you will be interfacing with FUSE via its "lowlevel" API. We have provided you with lots of code in the main() method of fuse.cc that handles much of the lowlevel nastiness, along with skeleton code for the operations you will need to implement. You can find details on what the methods you implement need to do at fuse_lowlevel.h. Study our getattr implementation to get a sense of how a full FUSE operation handler works, and how it communicates its results and errors back to FUSE. Every FUSE handler should either pass its successful result using one of the fuse_reply_... routines, or else call fuse_reply_err to report an error.

  Sending back directory information for the READDIR operation is a bit tricky, so we've provided you with much of the necessary code in the dirbuf_add, reply_buf_limited, and fuseserver_readdir methods. All that's left for you to do for READDIR in fuse.cc is to get the directory listing from your yfs_client, and add it to the b data structure using dirbuf_add.

  Though you are free to choose any inumber identifier you like for newly created files, FUSE assumes that the inumber for the root directory is 0x00000001. Thus, you'll need to ensure that when YFS mounts, it is ready to export an empty directory stored under that inumber.

• Misc tips:

  The start.sh scripts redirects the STDOUT and STDERR of the different processes to different files in the current working directory. For example, any output you make from fuse.cc will be written to fuse2yfs1.out. Thus, you should look at these files for any debug information you print out in your code.

  You can get FUSE to print out the requests and results of operations it passes to your file system. To turn this on, add the following line of code to the main() function of fuse.cc, just before the assignment of mountpoint into fuse_argv:

  fuse_argv[fuse_argc++] = "-d";
  

  See the lab overview for general hacking and debugging tips.

  If you wish to test your code with only some of the FUSE hooks implemented, be advised that FUSE may implicitly try to call other hooks. For example, FUSE calls LOOKUP when mounting the file system, so you may want to implement that first. You can use the -d flag (see above) to see exactly what hooks are called at every step.

Handin procedure

Prepare a tar file by executing these commands:

% cd ~/ds-class/lab
% make clean
% cd ..
% tar czvf yfs-lab2.tgz lab/

That should produce a file called yfs-lab2.tgz in your ds-class/ directory. Go to submit site to upload yfs-lab2.tgz

You will receive full credit if your software passes the same tests we gave you when we run your software on our machines.

---