(Partial) Transcript of 23 Mar 98 - Panel on Statistical and Symbolic Methods The following is a transcript of the panel on "Statistical and Symbolic Methods" hosted at the AAAI 1998 Spring Symposium, held at Stanford, California. ------------------------------------------------------------------------ Panelists Speak Panelists were first invited to discuss the balance between statistical and symbolic methods and (partially) answer a few of the questions on this list of questions. Panelists include: Branimir Boguraev, Graeme Hirst, Kathleen McKeown, Vibhu Mittal, Karen Sparck-Jones and Stan Szpakowicz. Branimir Boguraev: Definition of summarization: no definition. Avoid use of term, substitute "document characterization" instead. No canonical summary, so it must be task-driven. Statistical or Symbolic methodology doesn't matter. o collection of highly salient topical phrases o embedded in contextualized layers of "snippets" o etc... Then the task is: finding cohesive relations to identify "topic stamps", with partial parsing and anaphora resolution. We have the tools that are approximations to the real needs of summaries. This leads to the evaluation issue: best method is user studeies to validate. Should leverage the natural language knowledge to get better summaries, not to model language. Graeme Hirst: Do we require deep understanding for summaries? Real life summaries (i.e. TV Guide and sports reports) have bits of natural language. Fiction summaries can be done; we cite a categorization of plot types in folk tales into twenty-odd plot types and subtypes and instruments used. Thus summarization is a condensed representation of knowledge. At worse: need deep understanding, but most have elements to generated summaries. The shorter the summary, the less explicit the elements have to be. However, these techniques may not scale up. Kathleen McKeown: Let's talk about the technique of sentence extraction. Sure, they derive key content, but we don't really understand how all the metrics and switches interrelate to give good or better results. At best, sentence extraction processes can fail in coherence and sometimes mislead readers as well. On the other hand, pessimistically, sentence extraction fails in all cases for those reasons. But the good news is that people can tolerate errors. We need: the pairing of goals/tasks to summary types/genres. Especially the categorization along these axes: reports vs. briefings vs. indicative subtypes of summaries, document genres, and domain specific tasks. Also need a better understanding of evaluation methodology. The timeframe in which I believe NL Summarization should be looking at: o Sentence extraction: now o Symbolic: 1-2 years Vibhu Mittal: Statistical methods are your friends if you need a fast implementation of a summarization system, especially if youdon't care or don't understand the natural language phenomenon behind it. The down side is that you need corpora of articles and summaries, but these are becoming more available now. With statistical methods it is possible to get reasonable summaries, but they are short. We should be looking at augmenting these summaries with information extraction techniques. These shallow techniques will allow us to produce longer, more informative summaries: an incremental approach for improving the performance of summarization program on a larger percentage of texts. Karen Sparck-Jones: I'm a fundamentalist. I view summarization is a condensing transformation on the important content of the text. We can do analysis on two levels: o Words to passages and phrases; this is a weak method, but is more general o Frames to discourse; this is a stronger method, but is then more specific Thus the challenge is: to bridge the gap between these methodologies. We need to respond to the text instead imposing a "frame" structure onto the text and "fitting it" in. Combining methodology then doesn't mean tweaking statistical switches, but perhaps doing a "logical form" type of analysis, with the following sequential stages: 1. Start with sentences and process them into... 2. Propositions with argument structure, and then... 3. Combine them into a source network, which... 4. Could be pruned into a core subnets for use in generating a summary. The natural language processing community has the analysis techniques now, so the proposal methods are flexible. However, the traditional view of logical forms are only "quasi-propositional"; they have no pragamatic considerations nor the flexibility for partial analyses. This results in underspecified entities (i.e. tense, refernce, quantifiers); For example, during the linking phase, filling underspecified slots can be performed incorrectly; text structures can be destroyed when merging information from different initial positions. To help, we can call on keyword methodologies and other statistical methods. Stan Szpakowicz: We should do what we now know to do, now. Use symbolic methods in combination on the short text, single document genre. Use noun phrases and keywords and "smoothing" (in the future) to assist. I view summary as a "distortion" of the text. Now I will answer some of the questions that Dragomir posed for the panel: (N.B. I only caught some of the answers) but with respect to symbolic analysis, rather than statistical. o Can symbolic methods augment sentence extraction methods? Yes o Can symbolic methods help statistical domain models? Probably, but not much. o Can statistical techniques be evaluated? yes, using the same techniques as statistical, i.e. Precision/Recall o Statistical models given enough corpora turn into symbolic model o Dumb parsers may help, for specific tasks. ------------------------------------------------------------------------ Question and Answer Session N.B. I have tried to attribute the questions and the answers to specific members of the audience and panel. They are many missing attributions and some of the attributions may be in error. Question (Hovy): Why does summarization work? Why do short and long forms works? Shouldn't there be only one form? Answer (Jones): Even though there's a summary, you are including unspoken information. Answer (Szpakowicz): The reader fills in the gaps that you don't express. Summaries are incomplete, but to the purpose that the average reader must be able to fill in the gaps. Question: How well do humans do summarization? Answer : Well, given a "defined" task, they do pretty well. Answer : We should review the "genre" model, this is different from generic "summaries" Question (Baldwin): Is sentence extraction considered natural language processing? Answer: Yes. Sentence extraction is heavily emphasized because it is easier to perform evaluation on. Also, sentence extraction does not necessarily mean a statistical method; it can be a symbolic methodology as well. Comment: Note that human summarizers also do sentence extraction. See the New York Times online summaries as an example. Answer: But sentence extraction will not work for poorly written articles. Question (Delannoy): Does text has a "good" form (i.e. for sentence extraction)? Answer: Yes, for some text, evidently because sentence extraction does work for some texts. Answer: Maybe we can establish some criteria for the "good" form, perhaps it is genre-dependent. Question: Can core summarization be achieved via shallow technologies? Answer: Yes, should be dobable. ------------------------------------------------------------------------ Details of Panel Discussion: * Held at: Stanford University, History Rm 2 * Held on: Monday 23 March 1998, 4:00 - 4:30 pm ------------------------------------------------------------------------ Min-Yen Kan Created on: Fri Mar 27 11:42:16 1998 | Version: 1.0 | Last modified: Fri Mar 27 14:34:37 1998