Programming Assignments Development Guide

W4118 Fall 2011

UPDATED Saturday 09/05/2011 at 01:20am EST


All kernel programming assignments for this course will be done using a Google Android Developer Phone 1, ADP1, (CVN students may complete their assignments using the emulator). The Android platform can run on many different architectures, but the specific platform we will be targetting is the ARM CPU family. The Google ADP1 phones use an ARMv6 CPU (specifically, ARM1136) which is embedded in a Qualcomm System-On-a-Chip (SoC) processor called the MSM 7201a. You don't need to worry too much about these details though.

Because the target CPU is different from the CPU running in your personal computer, which you will use to compile the kernel, you will have to cross-compile any software, including the Linux kernel, to run on the different platform. The relevant Android/ARM SDK files have been pre-installed in a new virtual machine available for download here (note that this new VM has also been equipped with a modular kernel and VMWare Tools pre-installed: it should suffice for all the homework assignments in the course). You are also free to install these tools on your own computer, however there will be no technical support for these installations.

The login for the VM is: w4118
The password for the VM is: os2011 (same for root account)
We recommend that you change the password on your VMs for security purposes.

You will need to decompress and untar the downloaded VM file. Depending on your host OS, the commands are this:

To run the VM you need the VMware software on your host machine. The department has licenses for student use available if you have a CS account. See http://www.cs.columbia.edu/~crf/ for more info.

Emulator Display

In order to work with the emulator, you will need an X11 display. This can be accomplished in one of two ways:


Installing / Running on the Android Emulator:

In order to run the programs you compile, you will have to move them to the Android emulator's filesystem. This is done using the multi-purpose Android Debug Bridge (adb) utility. After starting up your Android virtual device with the custom kernel, you can check your connection by issuing: adb devices If you see the emulator listed there, then you can proceed to install your custom executable, and run it.

To move a file to the emulator: adb push /path/to/local/file /data/misc The /data/misc directory is read/write on the emulator, and you should be able to execute your program from there. To execute, you can either enter a shell on the emulator, or call the program directly from adb: adb shell
# /data/misc/exe_name
or adb shell /data/misc/exe_name
To pull a file out of the emulator: adb pull /path/in/emulator /local/path
To debug on the emulator: arm-none-linux-gnueabi-gdb exe_name
(gdb) target remote localhost:1234>

Installing / Running programs on the ADP1

In order to run the programs you compile on the device, you will have to move them to the ADP1's filesystem. This is done using the multi-purpose Android Debug Bridge (adb). In fact, these instructions should work for both the emulator, and the device. If you have just one devices or one emulator running on your system, then adb will automatically choose the correct mode for you. However if you have a devices plugged in, and an emulator running both will show up in the list of devices that the emulator sees: adb devices You can explicitly target the emulator by using the -e switch, or specifically target the phone by using the -d switch to adb (this switch works for all adb commands).

To move a file to the phone/emulator: adb push /path/to/local/file /data/misc The /data/misc directory is read/write on the emulator, and you should be able to execute your program from there (if the directory doesn't exist, you can easily create it: adb shell mkdir /data/misc). To execute, you can either enter a shell on the emulator, or call the program directly from adb: adb shell
# /data/misc/exe_name
or adb shell /data/misc/exe_name
To pull a file out of the emulator: adb pull /path/in/emulator /local/path

Kernel Development

There are many ways to edit kernel code. However, since the kernel is a rather large piece of contained software it is typically not efficient to load the kernel in a standard integrated IDE such as Eclipse. Instead, files are more typically modified using a simple editor, such as vi or emacs.

Often developers will use cross-referencing databases to aid the development. It can be extremely useful to easily be able to lookup the definition of a structure or function when editing code. We recommend using the cscope tool, which integrates with both emacs and vi. In the kernel root directory, after you have configured your kernel, you can run this simple command:
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- cscopeAfter which you can use the generated database files from your editor. For more information on how to integrate cscope and vi, see here. Emacs users should be clever enough to figure this out by themselves and using Google.

Note: that we do not provide support on how to use the above tools, this information servers purely as additional help for you to get started develop efficiently.

WiFi with custom kernel

You may have noticed that when you boot a custom kernel on the ADP1 phone, the 802.11 wireless LAN connection no longer works. We have provided an additional git repository which includes the TI WLAN driver. Use the following commands to build the WLAN driver and regain wireless internet access on the phone: git clone gitosis@os1.cs.columbia.edu:hmwk/utils.git

cd utils/
./wlan.compile.sh /path/to/kernel/dir
{boot up phone}
./wlan.install.sh
NOTE: this must be done after compiling your kernel so that the correct version information is embedded into the driver!

Emulator Users

Linux Users

On some older (distro-provided) versions of the kernel, the VMware Workstation and Player applications have been known to be buggy when interacting with USB devices. If you are running the course VM on a Linux machine and you cannot get the fastboot commands to complete from within the VM, you may have to copy the android-sdk-os.f2011 directory to your host and setup the host machine to run the fastboot/adb utilities using the instructions below. All the binaries in the android-sdk-os.f2011 directory should work on most Linux distributions. This problem does not seem to occur on Windows or MacOS hosts.

If you wish to do development directly on your Linux host, this is certainly possible, but we cannot support your setup. Follow these steps to configure your environment:

  1. Download the Android SDK from here
  2. Extract it to ~/android-sdk-os.f2011
  3. Inside the platforms subdirectory, extract the file android-os.f2011.tar.bz2
  4. Download the code-sourcery cross-compiler from here
  5. Extract this tar somewhere (e.g. ~/tools)
  6. Add the arm-2011.03/bin directory to your path in for example ~/.bashrc
  7. Set the CROSS_COMPILE variable to arm-none-linux-gnueabi-

To run the fastboot commands on your host, you can copy the SDK directory to your host and work from there. However, you will need to modify your machine's udev configuration to set proper permissions on the ADP1 devices. Copy the following snippet of udev rules into the file: /etc/udev/rules.d/51-android.rules

SUBSYSTEMS=="usb", ATTRS{idVendor}=="0bb4", ATTRS{idProduct}=="0c01", GROUP="plugdev", SYMLINK+="android%n"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="0bb4", ATTRS{idProduct}=="0c02", GROUP="plugdev", SYMLINK+="android%n"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="0bb4", ATTRS{idProduct}=="0fff", GROUP="plugdev", SYMLINK+="android-fastboot%n"
You will most likely have to reboot for the changes to take effect. As long as your local user is a member of the plugdev group does not exist on your machine please change the reference to a group which exits, to which your local user belongs.

Debugging

To debug on the emulator: w4118@osvm:~$ adb shell gdbserver :1234 /path/to/exe &
w4118@osvm:~$ adb forward tcp:1234 tcp:1234
w4118@osvm:~$ arm-none-linux-gnueabi-gdb /path/to/exe
GNU gdb (Sourcery G++ Lite 2010q1-202) 7.0.50.20100218-cvs
Copyright (C) 2010 Free Software Foundation, Inc.
...[snip]
(gdb) target remote :1234
Remote debugging using :1234
(gdb) b main
(gdb) c

To debug your kernel on the emulator: w4118@osvm:~/hmwk/linux-cm-2.6.35$ emulator -avd NameOfAVD -ramdisk ~/android-sdk-os.f2011/platforms/android-os.f2011/images/ramdisk.img -show-kernel -verbose -kernel arch/arm/boot/zImage -qemu -monitor telnet::4444,server -s&

...
w4118@osvm:~/hmwk/linux-cm-2.6.35$ echo "stop" | telnet localhost 4444
w4118@osvm:~/hmwk/linux-cm-2.6.35$ arm-none-linux-gnueabi-gdb ./vmlinux
GNU gdb (Sourcery G++ Lite 2010q1-202) 7.0.50.20100218-cvs
Copyright (C) 2010 Free Software Foundation, Inc.
...
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x00000000 in ?? ()
(gdb)

To pause a running kernel in the emulator which is being debugged (as per instructions above): w4118@osvm:~/hmwk/linux-cm-2.6.35$ echo "stop" | telnet localhost 4444
...
...or use the interactive qemu terminal (be careful not to Ctrl-C):
...
w4118@osvm:~/hmwk/linux-cm-2.6.35$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
QEMU 0.10.50 monitor - type 'help' for more information
(qemu) cont
(qemu) stop
(qemu) help