INSTRUCTIONS FOR WRITING UP YOUR FINAL PROJECT Since this is an experimental project of your choice, we care about the ideas and motivation behind what you did, and the results, not the implementation details. Therefore, read carefully through this web page to get a sense on how to write this up. It should be at least 4 pages, and certainly no more than 10. Probably 5 or 6 is what you should have in mind -- but don't think about that -- just follow these instructions to get the point clear. If you follow the outline suggested below, it "writes itself." If you have a team of multiple people, you should agree on different people writing different sections of the paper -- see below for a highly recommended outline for your paper. Pay heed to the recommended order for writing it. It is also recommended that you have another team co-review writeups with you, to find out what is unclear in your writeup. This proof-reading will make a big difference. Do so if you possibly can schedule it. Please write it up in html so I can have it forever in my archives. However, if for some reason you choose to not do this (e.g., you really really like Word), it will admittedly not influence your grade. In any case, if for some reason you would like me to not make it world accessible, just let me know and I'll be glad to not include it. But in either case, copy your writeup and all figures and anything else needed for the html into my directory. On AcIS, I made ~es66/public_html/finalproj/ writable -- make a subdir with the last name of a team member, and copy your stuff in there. Make the main file called index.html (not .index.html). While you are at it, please do so for your temporal writeup (if it is text or HTML) in /temporal/ (writeup and implementation!) and your NN writeup, if it is in HTML, in /nn/ -- THANKS! The ultimate goal it to allow the reader to judge what the goals of your work are, whether the work is worthwhile, and whether the project is feasible (i.e., has a chance of succeeding). Also, try to convey how your goals and approach relate to other relevant work, although this is not as important for the purposes of this assignment -- you have not been assigned to do a survey of related work. Cite a research paper (e.g., your own) that was presented in class, if there was one that is somehow relevant. Your write-up should contain a written description of the experiments you conducted. Break the paper into several section, as follows. The introduction should start with the motivation, and then give an overview of the solution. If the problem area or domain being approached has many details, you can expand on those details in another section directly after the intro. Either in this section, or in the intro, the question that your work will answer or the technical problem it will solve (or will "approach" -- an euphemism for "attempt to solve"). The statement of the problem should include a clear statement WHY THE PROBLEM IS IMPORTANT (OR INTERESTING). [What has been done before on the problem, and what has not. What am I contributing that is new. -- not so strict for class work.] The end of the intro which describes what you did to approach the problem is therefore describing your CONTRIBUTION (in research terms). A separate "(technical) approach" section gives the details of your modification/experiment -- how did you approach the problem? Focus here on the major design decisions. In most cases, you should give enough details that the reader, in principle, could implement and repeat exactly what you did. The results section gives the numerical results and some analysis. Finally, the conclusions section reminds the reader of your primary CONTRIBUTION, and puts the capital P on "perspective". The conclusions section should not offers yet further details, but rather should step back and offer the reader a "take home" package. You can give a paragraph describing interesting facts about your implementation (e.g., what language), and give a link to your code (and a demo, if possible!), but, other than this, your writeup is about the "AI" ideas and concepts, the conceptual contribution. The content of your writeup is independent of the implementation details, which are usually considered relatively mundane. Therefore, the outline is: introduction problem name this section by the name of the problem or domain this section is optional approach can rename this section with a name of the approach results related work (optional, and can be moved to before the approach) conclusions often repeats the main result I highly recommend you write the paper in the following order: First approach and results sections, which go together. Then problem section, if it is separate from the intro. Then the conclusions, then the intro. Write the intro last since it glosses the conclusions in one of the last paragraphs. (If this were a research paper you were submitting to a conference, you would write the abstract (i.e., summary) at this point, but you need not write an abstract for this assignment.) Last, give your paper a title. For all sections, the first sentence of each paragraph should be the main point of that paragraph. And if the flow of ideas is well organized into paragraphs, it should flow nicely to just read the first sentence of each paragraph. I am a real stickler about the introduction section of your paper. This must (A.) motivate your work by pinpointing the problem you are addressing and then (B.) give an overview of your approach and/or contributions (and perhaps even a general description of your results). In this way, the intro sets up my expectations for the rest of your paper -- it provides the context, and a preview. More than often a paper with an unclear introduction is filled with good ideas. In this case, the lack of organization makes it REALLY hard to sort out what the specific research contributions are. Your paper must have one main contribution that it focuses on. Your paper should not be an autobiography of the work you did over the last 7 months. EXAMPLE BAD INTRODUCTION: Here at the institute for computer research, me and my colleagues have created the SUPERGP system and have applied it to several toy problems. We had previously fumbled with earlier versions of SUPERGPSYSTEM for a while. This system allows the programmer to easily try lots of parameters, and problems, but incorporates a special constraint system for parameter settings and LISP S-expression parenthesis counting. The search space of GP is large and many things we are thinking about putting into the supergpsystem will make this space much more colorful. EXAMPLE GOOD INTRODUCTION: (THE PROBLEM:) Many new domains for genetic programming require evolved programs to be executed for longer amounts of time. For example, it is beneficial to give evolved programs direct access to low-level data arrays, as in some approaches to signal processing \cite{teller5}, and protein segment classification \cite{handley,koza6}. This type of system automatically performs more problem-specific engineering than a system that accesses highly preprocessed data. However, evolved programs may require more time to execute, since they are solving a harder task. (PREVIOUS OR OBVIOUS APPROACH: (Note that you can also have a related work section that gives more details about previous work.)) One way to control the execution time of evolved programs is to impose an absolute time limit. However, this is too constraining if some test cases require more processing time than others. To use computation time efficiently, evolved programs must take extra time when it is necessary to perform well, but also spend less time whenever possible. (APPROACH/SOLUTION/CONTRIBUTION. THE *FIRST* SENTENCE OF A PARAGRAPH LIKE THIS SHOULD SAY WHAT THE CONTRIBUTION IS. ALSO GLOSS THE RESULTS.) In this chapter, we introduce a method that gives evolved programs the incentive to strategically allocate computation time among fitness cases. Specifically, with an {\it aggregate computation time ceiling} imposed over a series of fitness cases, evolved programs dynamically choose when to stop processing each fitness case. We present experiments that show that programs evolved using this form of fitness take less time per test case on average, with minimal damage to domain performance. We also discuss the implications of such a time constraint, as well as its differences from other approaches to {\it multiobjective problems}. The dynamic use of resources other than computation time, e.g. memory or fuel, may also result from placing an aggregate limit over a series of fitness cases. (OVERVIEW:) The following section surveys related work in both optimizing the execution time of evolved programs and evolution over Turing-complete representations. Next we introduce the game Tetris as a test problem. This is followed by a description of the aggregate computation time ceiling, and its application to Tetris in particular. We then present experimental results, discuss other current efforts with Tetris, and end with conclusions and future work. -------------------------------------------- See the following for further stylistic guidelines: http://www.cs.columbia.edu/~hgs/etc/writing-style.html -------------------------------------------- More words of wisdom from Gail Kaiser: Your "body of paper" with problem approach, architecture results would turn into motivation, with at least one example scenario, preferably two, with illustrating figures, followed by a crisp generic problem statement model, i.e., functionality, particularly emphasizing "new" functionality, may or may not include formalisms, O(n) kinds of characterizations go here, not in evaluation architecture, of proposed system(s) to achieve this model, should be more generic than your own peculiar implementation, always include at least one figure realization, actual implementation details when implementing architecture isn't totally straightforward, mention briefly implementation language, platform, loc, dependencies on other packages and minimum resource usage if pertinent evaluation, how does it really work in practice, real or simulated performance metrics, end-user studies, mention external technology adoptors if any, etc. in most of the areas I publish in. Almost all the acronyms you mentioned as not needing to be spelled out would be gibberish to any of my readers. Or misinterpreted, e.g., ATM means the machine outside the bank where you get cash. Also, shouldn't related work section always come before summary and future work? Hint: In the case of a conference, make sure to cite the work of the PC co-chairs and as many other PC members as are remotely plausible, as well as from anything relevant from the previous two proceedings. In the case of a journal or magazine, cite anything relevant from last 2-3 years or so volumes. -------------------------------------------- Oded Goldreich wrote an essay entitled "how not to write a paper", with recommendations on style and organization http://www.wisdom.weizmann.ac.il/~oded/writing.html Don Knuth has online the TeX source of a book on "Mathematical Writing" (also useful for Computer Science). http://www-cs-faculty.Stanford.EDU/~knuth/klr.html -------------------------------------------- Folks, fyi, here are the "Guidelines for Experimental Papers" set forth for researchers submitting articles to the journal, "Machine Learning". I forward it to you to stimulate thinking -- your writeup requirements are certainly less stringent. 1.Papers that introduce a new learning "setting" or type of application should justify the relevance and importance of this setting, for example, based on its utility in applications, its appropriateness as a model of human or animal learning, or its importance in addressing fundamental questions in machine learning. 2.Papers describing a new algorithm should be clear, precise, and written in a way that allows the reader to compare the algorithm to other algorithms. For example, most learning algorithms can be viewed as optimizing (at least approximately) some measure of performance. A good way to describe a new algorithm is to make this performance measure explicit. Another useful way of describing an algorithm is to define the space of hypotheses that it searches when optimizing the performance measure. 3.Papers introducing a new algorithm should conduct experiments comparing it to state-of-the-art algorithms for the same or similar problems. Where possible, performance should also be compared against an absolute standard of ideal performance. Performance should also be compared against a naive standard (e.g., random guessing, guessing the most common class, etc.) as well. Unusual performance criteria should be carefully defined and justified. 4.All experiments must include measures of uncertainty of the conclusions. [IGNORE: These typically take the form of confidence intervals, statistical tests, or estimates of standard error.] Proper experimental methodology should be employed. For example, if "test sets" are used to measure generalization performance, no information from the test set should be available to the learning process. 5.Descriptions of the software and data sufficient to replicate the experiments must be included in the paper. [IGNORE THIS: Once the paper has appeared in Machine Learning, authors are strongly urged to make the data used in experiments available to other scientists wishing to replicate the experiments. An excellent way to achieve this is to deposit the data sets at the Irvine Repository of Machine Learning Databases. Another good option is to add your data sets to the DELVE benchmark collection at the University of Toronto. For proprietary data sets, authors are encouraged to develop synthetic data sets having the same statistical properties. These synthetic data sets can then be made freely available.] 6.Conclusions drawn from a series of experimental runs should be clearly stated. Graphical display of experimental data can be very effective. Supporting tables of exact numerical results from experiments should be provided in an appendix. 7.Limitations of the algorithm should be described in detail. Interesting cases where an algorithm fails are important in clarifying the range of applicability of an algorithm.