Homework 5

W4118 Fall 2012

UPDATED: Monday 11/19/2012 at 11:02am EST

DUE: Wednesday 11/28/12 at 11:59pm EST

All non-programming, written problems in this assignment are to be done by yourself. Group collaboration is permitted only on the kernel programming problems. All homework submissions (both individual and group) are to be made via Git. Git repository access will use the same public/private key-pair you used for previous homeworks (see these Git instructions).

Individual Written Problems:

Solutions are available here.

The Git repository you will use for the individual, written portion of this assignment can cloned using: git clone gitosis@os1.cs.columbia.edu:UNI/hmwk5.git (replace UNI with your own UNI). This repository will be accessible only by you.

Exercise numbers refer to the course textbook, Operating System Concepts Essentials. Each problem is worth 5 points.

  1. Exercise 7.16

  2. Exercise 7.19

  3. Exercise 7.27

  4. Exercise 8.23

  5. Exercise 8.26

  6. Exercise 8.28

Group Programming Problems:

Solutions are available here.

Group programming problems are to be done in your assigned groups. The Git repository your entire group will use to submit the group programming problems can be cloned using: git clone gitosis@os1.cs.columbia.edu:TEAM/hmwk5.git (Replace TEAM with the name of your team, e.g. team1). This repository will be accessible to all members of your team, and all team members are expected to commit (local) and push (update the server) changes / contributions to the repository equally. You should become familiar with team-based shared repository Git commands such as git-pull, git-merge, git-fetch. One important thing to remember is to pull any changes into your local repository before trying to push any commits to the class server.

All team members should make at least five commits to the team's Git repository. The point is to make incremental changes and use an iterative development cycle. Please follow the Linux kernel coding style.
Hint(1): use the checkpatch.pl script to ensure your code conforms to this standard. Points will be deducted for non-compliance!

It's a good idea to clone your repository after your final push (but before the deadline) and test the cloned code to be sure it behaves as expected.

All kernel programming assignments in this year's class are done on the Android operating and targeting the ARM architecture. For more information on how to get your development environment configured, please see this page: Development Guide.

The kernel programming for this assignment will be done using a Nexus 7 tablet. The Android platform can run on many different architectures, but the specific platform we will be targeting is the ARM CPU family. The Nexus 7 tablets use an ARMv7 quad-core CPU which is embedded in the Nvidia Tegra 3 SoC.

Because the target CPU is different from the CPU running in your personal computer, you will have to cross-compile any software, including the linux kernel, to run on the different platform. You should use the same virtual machine you used for homeworks 2 and 3.

  1.   (60 pts.) Userspace Mapped Page Table
    Important update: Your git repos are being updated with a modification to the system call wrapper for this assignment. Updates will start at at 11:45pm on 11/16/12, and may take up to an hour to update all groups. Please be sure to git pull.
    A process' page table serves to map virtual memory addresses to physical memory addresses. In addition to this basic mapping, the page table also includes information such as, which pages are readable in userspace, which are dirty, which are read only, and more. Ordinarily, a process' page table is privileged information that is largely only accessible from within the kernel. In this assignment, inspired by Dune, we want to allow processes to view their own page table, including all the extra bits of information stored along with the physical address mapping. The Dune paper goes into detail on some of the advantages exposing this information provides, among the most notable being usefulness for garbage collectors. This would be particularly helpful for Android as user-level programs are Java-based and efficient garbage collection for these Java-based programs on a memory-constrained mobile device could have noticeable performance benefits.
    In the Linux kernel, as you'll soon learn in class, the page table does not exist simply as a flat table with an entry for each virtual address. Instead, it is broken into layers. In userspace, we have much more leeway when it comes to how we can store our page table and so there we shall treat it as if it was just one large basic table with an entry for every page. We can do this by mapping each individual page table entry in the kernel into a memory region in userspace. By doing a remapping, updates to the page table entries will seamlessly appear in the userspace memory region. We can also do the mapping in such a way that sections of the page table without any page table entries present will appear as empty (null) just as they would in the kernel (this part has been done for you). This can even be done without incurring RAM usage overhead.
    Your task is to create a system call, which after calling, will expose the processes' page table in a read-only form and will remain up to date as the process executes. Do not allow the process to modify the page table. You will do this by remapping page table entries in the kernel into a memory region mapped in userspace (not by copying any data!).

    Obtaining the Mapping:
    To set up the page table mapping in user space, you are to implement the following system call:
    /* Map the current processes' page table into userspace.
     * After successfully completing this call, addr will contain
     * the current process' page table indexed by page number with
     * each index containing the corresponding page table entry for
     * that page number (i.e. virtual address).
     * @addr A previously mmap'ed memory region prepped for use
     *       with this system call. A wrapper has been provided
     *       for you to prep this memory region.
     */
    int expose_page_table(void *addr);
    
    The address space passed to expose_page_table() must be of sufficient size and accessibility, otherwise it is an error. Errors should be handled appropriately. On success, 0 is to be returned. On failure, the appropriate negative errno is to be returned.

    Hints:
    Citations:
  2.   (10 pts.) Investigate
    Demonstrate that your system call works with a test program that prints out the page table in the following format:
    [page num] [virt addr] [phys addr] [young bit] [file bit] [dirty bit] [read only bit] [user bit]
    e.g.
    266789 0x41225000 0xafffb 1 1 0 1 1
    
    Example updated above.
    If a page is not present, do not print it.
    In addition to the above test program, framework for another test program has been provided. In this program there exists two functions in an already compiled object file:
    /**
     * Hides a message somewhere in memory.
     */
    void store_super_double_secret_msg(void);
    
    /**
     * Decodes a message hidden in memory.
     * @addr - Address of the page the message is stored in.
     * Returns 0 on success, -1 on failure.
     */
    int super_double_secret_msg_decoder(void *addr);
    
    You should record the state of the page table prior to calling store_super_double_secret_msg(), then check the state again after calling it. Look for newly paged-in pages and pass the virtual address for each new page you find into super_double_secret_msg_decoder() to decode the super secret secret message.
    Implement this test program so that it can decode the message. You do not have to write the message down anywhere, just be sure your test program can find it when run.

Additional Hints/Tips

Kernel Hacking:

Linux users: If you are a Linux user and cannot connect your device to your VM to flash the system (or you would just like a more efficient set up), you may want to develop on your Linux host machine. While we do not have the resources to support your desktop installations, you may find this more convenient. To make it work, you should follow these steps:
  1. Obtain the Android SDK from your VM. It's located at /home/w4118/utils/android-sdk_r20.0.3-linux-jellybean.tgz
  2. Extract it to ~/ (once extracted, there will be a ~/android-sdk-linux directory)
  3. Add ~/android-sdk-linux/tools and ~/android-sdk-linux/platform-tools to your path (e.g. in ~/.bashrc)
  4. Download the code-sourcery cross-compiler from here
  5. Extract this tar somewhere (e.g. ~/tools)
  6. Add the extracted arm-2010.09/bin directory to your path (e.g. in ~/.bashrc)
  7. Set the CROSS_COMPILE variable to arm-none-linux-gnueabi-
Hint: You can use your VM as a guide by looking at its ~/utils directory and ~/.bashrc file.