This is a team assignment to be done with your current partner (if you have one).

Overview

A major application for computer science is in analyzing data that is too big to analyze by hand. My own research is in data mining, which is concerned with how to get computers to automatically find complex and unexpected patterns in data. Data can be textual, artistic, or scientific. For this assignment, we'll use data from the MovieLens project. MovieLens is a research project based out of the University of Minnesota which is designed to make recommendations to people as to what movies they should watch (not unlike what Netflix does). Check out the MovieLens website and play around: it's fun. The MovieLens folks have graciously made some of their data publicly available, and so we can use it for our assignment.

Getting set up

1. Create a directory called movies, and into it download this file. The file is compressed and archived using a program called tar. On our Linux machines, you can pull out the individual files with the following command:

   tar xfvz million-ml-data.tar_.gz
   

   The above will probably also work on a Mac. It won't work under Windows. If you have any trouble with this on your home machine, you'll have to unarchive it on the Linux machines, then transfer the files that come out to your home machine.

   I have also placed the data in /Accounts/courses/cs117/dmusican/movies. You can grab it from there if you're really having trouble, you should at least download the tar file anyway from the above link so that the data owners know how many downloads there have been, and so you have an official copy.

2. Read the README file that you find after you unarchive the data. You'll see that I am unallowed to redistribute this data, which is why I above provided a link to the MovieLens website to pull down the data for yourself. You'll then find a description of the three important files: users.dat (data on the users), movies.dat (data on the movies), and ratings.dat (the actual movie ratings). Skim through these files to see what they actually look like.

3. The ratings file is a little large: it contains 1 million lines! This isn't a big problem for your computers: if your computer is reasonably recent, a Java program should be able to read through it in less than 20 seconds or so. Still, 20 seconds is a long time (try it, and you'll see what I mean) if you are running your program over and over again while debugging it. Therefore, I have provided you with the program FewerRatings.java. This program will read the ratings.dat file and cut it down to only the first 500 users. Therefore, you should compile it, and run it as follows:

   javac FewerRatings.java
   java FewerRatings > fewerratings.dat
   

   This will create a new ratings file with fewer ratings than the big one, and so any analysis you do on this file will run MUCH faster. You should make sure that any code you write does work on the complete ratings.dat file, but debugging is much more pleasant if you work with the smaller one first.

   You should also look carefully through the code for FewerRatings.java. The other reason I supplied it is to give you a sample of how to deal with the delimiters contained in the MovieLens data.

Your assignment

Write a program, or multiple programs, that will determine and print out the following information:

1. How many of the users are artists?
2. How many of the users are artists at least 25 years old?
3. Movies are rated on a 1-5 scale. What is the average movie rating?
4. What is the average rating given for movie id #1? For movie id #2? What are those movies?
5. What is the gender and zip code of the user who rated the most movies?

Your program(s) should automatically answer all of the above questions with no intervention or changes by you along the way. For example, for the last part above: once your program has determined the user id of the person who rated the most movies, your program should store that in a variable and then look up that id in the users data. You should NOT print out the user id of that person, then type in that user id into another program, compile, and run.

If you can successfully answer all of the above questions, well done! If all is correct and your program has proper style, you will receive 19 out of 20 points. If you want to shoot for the 20th point (I consider this optional), here's a bonus question:

Find the most interesting or surprising piece of information in this data that you can. Print this out as well.

Wrap-up

If you submit multiple programs, make sure that it is exceedingly clear to the graders which programs do what. They should all be contained in your movies directory.

Before submitting your programs, move the data to another directory first. That way, we don't clog up the submission server with 50 copies of the movie data.

If you have been reading ahead and happen to know about arrays, you might be tempted to use them for this assignment. Don't do it. That would be a dramatic waste of computer memory, and it wouldn't even work on a dataset of Netflix proportions.

Have fun, and good luck!