CS 475, Spring 2018, Homework 3, Due Mar 19, 2pm
Start by downloading the handout:
Updates
- 3/11: Added note about not accessing the TagServer field in your client.
- 3/8: Note that RvPredict might falsely say a race is possible in code that is protected by stamped locks. RvPredict can’t know which thread “owns” the lock, and hence might give you a warning. Your choices are to (1) ignore the warnings (keep in mind: they are warnings, not “errors”) or (2) use reentrant locks (you are only required to use stamped locks for the lock file and unlock file methods).
Please post questions on Piazza
The purpose of this assignment is to get you familiar with RMI and locks in Java. We will build on the concepts of HW2, focusing specifically on creating a client-server environment. In particular: it has recently become more and more popular to organize files by keywords and tags, rather than by their location in a hierarchical filesystem. Users of Mac OS X might find themselves turning more to using Spotlight to open documents (rather than manually locating them in a hierarchical path), and Windows users might find themselves using Cortana to do the same.
The remainder of the assignments (and project) in this class will involve building parts of an intelligent file management tool, which you could use to browse and search through your files. We will call this tool VCFS (VeryCoolFileSystem).
For this first assignment, your task will be to build a client-server version of VCFS from HW2. Note that we will not grade you for the concurrency/synchronization issues as in HW2 – you will not be penalized again for those issues. However, your implementation is still expected to reliably run – we will not seek out races, but if they interfere with grading, you may be penalized for them.
Your primary interface for debugging your VCFS will be the same shell driver that we provided for HW2. The main difference will be that when you start the shell, you will specify a hostname and port number of a VCFS server. Now, we will implement most of the functionality in the server. We have provided stubs for all of the various commands you will need to implement, with a wrapper so that you can interact with VCFS interactively. When you run the compiled jar file, you’ll get a command prompt, like this:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
cs475sh help
AVAILABLE COMMANDS
Built-In Commands
clear: Clear the shell screen.
exit, quit: Exit the shell.
help: Display help about available commands.
script: Read and execute commands from a file.
stacktrace: Display the full stacktrace of the last error.
Command
add-tag: Add a tag to the list of known tags
cat: Cat a file
cat-all: Cat all files matching a tag
delete-tag: Delete a tag
echo: Echo text into a file
echo-all: Echo text into all files matching a tag
edit-tag: Edit tag name
get-tags: List all tags on a file
list-files: List the files
remove-tag: Remove a tag from a file
tag-file: Add a tag to a file
tags: List the tags
|
As you implement the various functionality in the parts below, the commands above will begin to work.
We have also provided a baseline suite of unit tests, which you can execute by calling mvn -Ptest test
General requirements:
We have provided you a baseline implementation of VCFS that handles all user interaction and connects to the server, but does not do anything (that is, you will not need to implement any UI or command processors; you will not need to add much RMI boilerplate). You may not include any additional libraries (e.g. download and and require additional JARs), although feel free to use any of the various libraries included with Java 8 or already included by the starter project.
Your VCFS will be compiled and tested using apache maven, which automatically downloads the various dependencies for VCFS and executes the provided JUnit tests. Please install Maven on your computer. Unfortunately, Maven is not installed on Zeus, however you can download the most recent version (e.g. apache-maven-3.5.2-bin.zip) and unzip it in your home directory. Then, to run maven, type ~/apache-maven-3.5.2/bin/mvn in the snippets below (instead of just mvn). Note that you can easily import maven projects into eclipse and IntelliJ.
To compile and run your shell, run mvn package in the top level directory and then, in the server directory run java -jar target/filemanager-server-0.0.3-SNAPSHOT.jar <portnumber> to start the server, and to start the client, in the client directory run java -jar target/filemanager-client-0.0.3-SNAPSHOT.jar <portnumber>. Your can specify any free port number on your computer over 1024; we have hardcoded the client to assume that the client runs on the same computer as the server. You’ll notice that the text-mode interface we’ve provided for you has a handy help command. To build the jar file without running the tests, run mvn -DskipTests package.
Your shell will be automatically graded for correctness (note that there will be a manual grading phase to check hard-to-automatically-catch concurrency issues). Included with your handout is the test script that we will use to grade your assignment. Upon submitting your assignment, our server will automatically compile and test your assignment and provide you with test results. You can resubmit an unlimited number of times until the deadline. To run these tests, simply execute mvn test (of course, if you do this first off, you’ll see that they all fail!)
Note: Your code must compile and run on the autograder, under Java 8. It is unlikely that you will have any difficulties developing on a Mac, Linux or Windows, but please keep in mind the possibility of portability problems. When you feel satisfied with implementing one phase of the assignment, submit to AutoLab and verify that AutoLab agrees.
General coding/grading requirements:
- You should feel free to add whatever additional classes you wish, or any additional methods to the existing edu.gmu.cs475.FileTagManagerServer and edu.gmu.cs475.FileTagManagerClient. You must not modify the edu.gmu.cs475.IFileTagManager interface, the edu.gmu.cs475.AbstractFileTagManagerClient, any of the tests, or any of the internal classes.
- Your code should be thread-safe: concurrent calls to any of these methods (or any other method in IFileTagManager) should not have any races. It should now be clearer how this can occur — you will potentially have multiple clients attempting to interact with the server simultaneously.
- You must not store any state in static fields
- All concurrency-related grading will account for a total 0f 10% of your grade (see Part 5). We will only consider concurrency-concerns for the first 4 parts to the extent that they are preventing your assignment from passing the given tests under normal circumstances.
Part 1: Migrating the existing API to RMI (20%)
Your VCFS tool will now have a single server that maintains the tags and files. Multiple clients will be able to interact with a single server and perform the same operations as before.
The key focus for this part of the assignment will be to separate this functionality into a client-side and server-side component.
Your client and server will communicate over the edu.gmu.cs475.IFileTagManager interface, which mostly contains all of the same operations from HW2, plus a few extra (which we’ll get to in Parts 2 and 3:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
|
/**
* List all currently known tags.
*
* @return List of tags (in any order)
*/
public Iterable listTags() throws RemoteException;
/**
* Add a new tag to the list of known tags
*
* @param name
* Name of tag
* @return The newly created Tag name
* @throws TagExistsException
* If a tag already exists with this name
*/
public String addTag(String name) throws RemoteException, TagExistsException;
/**
* Update the name of a tag, also updating any references to that tag to
* point to the new one
*
* @param oldTagName
* Old name of tag
* @param newTagName
* New name of tag
* @return The newly updated Tag name
* @throws TagExistsException
* If a tag already exists with the newly requested name
* @throws NoSuchTagException
* If no tag exists with the old name
*/
public String editTag(String oldTagName, String newTagName) throws RemoteException, TagExistsException, NoSuchTagException;
/**
* Delete a tag by name
*
* @param tagName
* Name of tag to delete
* @return The tag name that was deleted
* @throws NoSuchTagException
* If no tag exists with that name
* @throws DirectoryNotEmptyException
* If tag currently has files still associated with it
*/
public String deleteTag(String tagName) throws RemoteException, NoSuchTagException, DirectoryNotEmptyException;
/**
* List all files, regardless of their tag
*
* @return A list of all files. Each file must appear exactly once in this
* list.
*/
public Iterable listAllFiles() throws RemoteException;
/**
* List all files that have a given tag
*
* @param tag
* Tag to look for
* @return A list of all files that have been labeled with the specified tag
* @throws NoSuchTagException
* If no tag exists with that name
*/
public Iterable listFilesByTag(String tag) throws RemoteException, NoSuchTagException;
/**
* Label a file with a tag
*
* Files can have any number of tags - this tag will simply be appended to
* the collection of tags that the file has. However, files can be tagged
* with a given tag exactly once: repeatedly tagging a file with the same
* tag should return "false" on subsequent calls.
*
* If the file currently has the special tag "untagged" then that tag should
* be removed - otherwise, this tag is appended to the collection of tags on
* this file.
*
* @param file
* Path to file to tag
* @param tag
* The desired tag
* @returns true if succeeding tagging the file, false if the file already
* has that tag
* @throws NoSuchFileException
* If no file exists with the given name/path
* @throws NoSuchTagException
* If no tag exists with the given name
*/
public boolean tagFile(String file, String tag) throws RemoteException, NoSuchFileException, NoSuchTagException;
/**
* Remove a tag from a file
*
* If removing this tag causes the file to no longer have any tags, then the
* special "untagged" tag should be added.
*
* The "untagged" tag can not be removed (return should be false)
*
* @param file
* Path to file to untag
* @param tag
* The desired tag to remove from that file
* @returns True if the tag was successfully removed, false if there was no
* tag by that name on the specified file
* @throws NoSuchFileException
* If no file exists with the given name/path
* @throws NoSuchTagException
* If no tag exists with the given name
*/
public boolean removeTag(String file, String tag) throws RemoteException, NoSuchFileException, NoSuchTagException;
/**
* List all of the tags that are applied to a file
*
* @param file
* The file to inspect
* @return A list of all tags that have been applied to that file in any
* order
* @throws NoSuchFileException
* If the file specified does not exist
*/
public Iterable getTags(String file) throws RemoteException, NoSuchFileException;
/**
* Return a file as a byte array.
*
* @param file
* Path to file requested
* @return String representing the file
* @throws IOException
* if any IOException occurs in the underlying read
*/
public String readFile(String file) throws RemoteException, IOException;
/**
* Write (or overwrite) a file
*
* @param file
* Path to file to write out
* @param content
* String representing the content desired
* @throws IOException
* if any IOException occurs in the underlying write
*/
public void writeFile(String file, String content) throws RemoteException, IOException;
/**
* Acquires a read or write lock for a given file.
*
* @param name
* File to lock
* @param forWrite
* True if a write lock is requested, else false
* @return A stamp representing the lock owner (e.g. from a StampedLock)
* @throws NoSuchFileException
* If the file doesn't exist
*/
public long lockFile(String name, boolean forWrite) throws RemoteException, NoSuchFileException;
/**
* Releases a read or write lock for a given file.
*
* @param name
* File to lock
* @param stamp
* the Stamp representing the lock owner (returned from lockFile)
* @param forWrite
* True if a write lock is requested, else false
* @throws NoSuchFileException
* If the file doesn't exist
* @throws IllegalMonitorStateException
* if the stamp specified is not (or is no longer) valid
*/
public void unLockFile(String name, long stamp, boolean forWrite) throws RemoteException, NoSuchFileException, IllegalMonitorStateException;
|
Implement your server in the server project, by implementing the empty methods in edu.gmu.cs475.FileTagManagerServer. You should feel free to reuse the code you had from HW2, or write something different (you’ll notice that the API changed slightly).
Added 3/11: Your client must not directly access the field tagServer – to contact the server it should just call the super.XXX methods (e.g. instead of tagServer.heartbeat(...), just heartbeat(...)). This is required so that the tests can correctly and selectively mock out the server behavior.
When you are ready to check your work, you should run just the tests in the test class edu.gmu.cs475.P1RMIMigrationIntegrationTests. To do so from maven, you should run mvn -Dtest=P1RMIMigrationIntegrationTests test.
Precise grading breakdown:
- Automated functional tests (
edu.gmu.cs475.P1RMIMigrationIntegrationTests): 15 points
- 15 JUnit tests, 1 points each
- Manual feedback: 5 points
Part 2: Client operations and Heartbeat (30%)
You might have noticed that echoToAllFiles and catToAllFiles are not listed in the server API. We would like you to implement these functions on the client side. Hence, your FileTagManagerClient has empty stubs for you to implement these methods. The same rules apply as HW2 – your client needs to acquire read/write locks on all files to be read/written, then perform the individual read/writes. We’ve provided you with the basic readFile, writeFile RMI implementation.
We have also added a new method to the client/server API beyond HW2:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
/**
* Notifies the server that the client is still alive and well, still using
* the lock specified by the stamp provided
*
* @param file
* The filename (same exact name passed to lockFile) that we are
* reporting in on
* @param stampId
* Stamp returned from lockFile that we are reporting in on
* @param isWrite
* if the heartbeat is for a write lock
* @throws IllegalMonitorStateException
* if the stamp specified is not (or is no longer) valid, or if
* the stamp is not valid for the given read/write state
* @throws NoSuchFileException
* if the file specified doesn't exist
*/
public void heartbeat(String file, long stampId, boolean isWrite) throws RemoteException, IllegalMonitorStateException, NoSuchFileException;
|
As we will discuss in the context of distributed filesystems (lecture 12, 2/28), a heartbeat protocol can be used to detect when clients have disconnected. In our case, we want to use a heartbeat to determine that a lock should be deemed invalid: if a client crashes, we still want to allow other clients to proceed. The heartbeat protocol you will implement is very simple: if a client holds a lock, they must report to the server every 2 seconds that they are still using that lock. The server will keep track of every file that currently is locked. If the server has not heard from the client that holds a lock for over 3 seconds (note that this includes a 1-second grace period), it will assume that that client has crashed, and can no longer use the lock. In this case, the server will release the lock, allowing other clients to continue to perform contentious operations.
Hence, for this part of the assignment, it’s your job to implement functionality in your client so that whenever a lock is acquired, it will begin sending heartbeats for that lock. As multiple locks are acquired, the client will send multiple heartbeats. As locks are unlocked, those heartbeat timers will be canceled. If a heartbeat request results in an exception (e.g. an IllegalMonitorStateException or NoSuchFileException), your client should stop sending those heartbeats (but of course, continue with any others that haven’t thrown exceptions). You should start sending heartbeats when lockFileSuccess is called, and stop sending them when unlockFileSuccess is called.
We will grade your client and server side implementation of the heartbeat protocol separately. We have provided a set of tests, edu.gmu.cs475.P2HeartbeatClientTests, which automatically mock a correctly functioning server — running your client against this fake server. To run this from maven, you should run mvn -Dtest=P2HeartbeatClientTests test.
Hint: You might find it useful to use Java’s ScheduledExecutorService to implement your timers (but you are free to use other implementations too). Keep in mind that any timer service you use will be inherently concurrent: you’ll have timers executing at the same time as your application; multiple timers might be executing concurrently.
Precise grading breakdown:
- Automated functional tests (
edu.gmu.cs475.P2HeartbeatClientTests): 24 points
- 6 JUnit tests, 4 points each
- Manual feedback: 6 points
Part 3: Server Heartbeats and Leases (30%)
With the client-side protocol implemented, your next task will be to implement server functionality to track which locks have been acquired, and expire locks as necessary. Again, we will grade your server and client implementations separately.
For testing purposes, you’ll also implement two functions to report the list of read and write locked files:
|
1
2
3
4
5
6
7
8
9
|
/**
* Get a list of all of the files that are currently write locked
*/
public List getWriteLockedFiles() throws RemoteException;
/**
* Get a list of all of the files that are currently write locked
*/
public List getReadLockedFiles() throws RemoteException;
|
Recall that although the client will be sending its heartbeat messages every 2 seconds, the server’s task is to keep track of when each lock was last heartbeat’ed, and expire locks that haven’t been reported in on in 3 seconds. Each lock might have been last heard from at a different time, and hence, your server will need to track this information per-lock holder.
Again, we will grade your client and server side implementation of the heartbeat protocol separately. We have provided a set of tests, edu.gmu.cs475.P3ServerTests, which automatically mock a correctly functioning client — running your server against this fake client and simulating error conditions. To run this from maven, you should run mvn -Dtest=P3ServerTests test.
Precise grading breakdown:
- Automated functional tests (
edu.gmu.cs475.P3ServerTests): 24 points
- 6 JUnit tests, 4 points each
- Manual feedback: 6 points
Part 4: Integration tests (10%)
We have also provided a very small suite of integration tests, which run your actual client against your actual server (rather than a fake client or fake server). These integration tests also simulate client failures. You’ll find these integration tests in edu.gmu.cs475.P4IntegrationTests.
Your goal here is simply to pass all of these tests – if you did all three parts above correctly, then these should work fine.
Precise grading breakdown:
- Automated functional tests (
edu.gmu.cs475.P4IntegrationTests): 24 points
- 5 JUnit tests, 2 points each
Part 5: Concurrency (10%)
To receive a top score on this assignment, you will also need to be sure that your code has no races. For this assignment, we will be using the tool, RV-Predict to detect races that may occur while running your tests. RV-Predict will give you precise feedback on the races that it detects, for instance:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
Data race on field account.Account.Bank_Total: {{{
Concurrent write in thread T12 (locks held: {})
----> at account.Account.Service(Account.java:98)
at account.BankAccount.Action(BankAccount.java:41)
at account.BankAccount.run(BankAccount.java:56)
T12 is created by T1
at account.Account.go(Account.java:46)
Concurrent read in thread T13 (locks held: {})
----> at account.Account.Service(Account.java:98)
at account.BankAccount.Action(BankAccount.java:41)
at account.BankAccount.run(BankAccount.java:56)
T13 is created by T1
at account.Account.go(Account.java:46)
}}}
|
AutoLab will automatically run RV-Predict on all of your submissions. You can run it on your own computer by downloading and installing it (it’s free for non-commercial use). When you run your tests with maven, use the command mvn -Drvpredict=/path/to/rv-predict.jar test (on Mac this would be mvn -Drvpredict=/Applications/RV-Predict/Java/lib/rv-predict.jar test).
We will not give you a direct equation to correlate from # of reports from RV-Predict -> a grade on this section. We will manually award up to 10 points for concurrency correctness based on no apparent races and no over-synchronization (again, one way to avoid races could be to force every operation to be serial; this would not be ideal or correct). Moreover, note that RV-Predict will find many races, but will not find all races, which we might by hand! Note also that if you make wide use of stamped locks (instead of reentrant locks or synchronized blocks), RV-Predict will report that races are possible because it can not analyze stamped locks.
Grading
Your assignment will be graded on a series of functional tests, making sure that it implements the specification above.
Hand In Instructions
You must turn in your assignment using Autolab (You MUST be on the campus network, or connected to the GMU VPN to connect to Autolab). If you did not receive a confirmation email from Autolab to set a password, enter your @gmu.edu (NOT @masonlive) email, and click “forgot password” to get a new password.
Create a zip file of the root directory in your assignment (please: .zip, not .tgz or .7z etc) — this is the root directory that includes the client, shared, server directories. When you upload your assignment, Autolab will automatically compile and test it. You should verify that the result that Autolab generates is what you expect. Your code is built and tested in a Linux VM. Assignments that do not compile using our build script will receive a maximum of 50%. Note that we have provided ample resources for you to verify that our view of your assignment is the same as your own: you will see the result of the compilation and test execution for your assignment when you submit it.
You can resubmit your assignment an unlimited number of times before the deadline. Note the course late-submission policy: assignments will be accepted up until 24 hours past the deadline at a penalty of 10%; after 24 hours, no late assignments will be accepted, no exceptions.
Note – You MUST be on the campus network, or connected to GMU VPN to connect to Autolab.
Decoding the output:
Note, AutoLab will run your code on the tests twice: once without RV-Predict (these are the scores used for parts 1-4), and once with RV-Predict (this is informational only). The outcomes should be the same with or without RV-Predict, but we wanted to make 100% sure that adding the tool doesn’t break your otherwise seemingly functioning code.
Questions
Please post questions on Piazza