Summary Natural Language Generation
A. Introduction: two kinds of NLG
In this first part of the course, you will get acquainted with the idea of an NLG system. We discuss some actually deployed NLG systems
and use these to get acquainted with a computational architecture often used by NLG systems, known as the pipeline architecture. This
architecture will be compared to a somewhat similar architecture proposed by researchers in psycholinguistics. We discuss a famous
theory about what it means to speak and listen cooperatively, namely the Gricean Maxims, and we discuss the relevance of this theory for
NLG. This discussion will chime with issues that will be covered in parts B and D of the course.
Lecture 1: Introduction NLG
Lecture 2: Software Engineering of NLG
Lecture 3: A classic view of NLG systems (Reiter and Dale)
Lecture 4: The "speaking" pipeline in psychology
Lecture 5 & 6: Gricean maxims (Paul Grice)
Lecture 7: Babytalk: BT-45
Lecture 8: Deep learning basics
Lecture 9: Deep learning for NLG
B. Module in focus: Referring Expression Generation (REG)
In this part of the course we focus on one component of an NLG system, responsible for Referring Expressions Generation (REG). Our
perspective will alternate between algorithms and experiments with human participants. We will start by familiarising ourselves with the
REG task and looking at a number of REG algorithms, whose performance will be assessed in various ways.
Lecture 10: The REG task and some classic REG algorithms
Lecture 11: Classic REG algorithms - continued
Lecture 12: Computational Interpretations of the Gricean Maxims in the Generation
of Referring Expressions (Dale and Reiter, 1995)
Lecture 13 : Evaluating REG algorithms: The TUNA project
Lecture 14a: The role of common ground in communication
Lecture 14b: “Don’t talk about the pink elephants!” (Lane et al. 2006)
C. Issues in Machine Learning-based and neural NLG.
In this final part of the course, we look at some computational metrics that are often used when NLG algorithms/systems are evaluated, we
look at some concrete studies, and we discuss limitations of current technologies.
Lecture 15a: BLEU: a Method for Automatic Evaluation of Machine Translation (Papenini et al, 2002).
Lecture 15b: A Structured Review of the Validity of BLEU (Ehud Reiter).
Lecture 16a: Towards Generating Colour Terms for Referents in Photographs: Prefer the Expected
or the Unexpected? (Zarriess & Schlangen 2016).
Lecture 16b: Survey of Hallucination in Natural Language Generation (Ji et al. 2022).
Lecture 17a: Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
(Bender & Koller 2020).
Lecture 17b: On the dangers of stochastic parrots: Can language models be too big? (Bender,
Gebru, McMillan-Major, and Shmitchel 2021).
Sources: lecture slides NLG, student presentations & lecture notes by Laura Hoekstra
Lecture 1 (Introduction NLG) Natural language generation systems are computer
= Practical NLG algorithms/systems which produce text in English or other
human languages.
-> So the gathered data is explained
, (made understandable), in words and/or sentences, to sentences based on decisions made in previous
human users. stages (e.g. capitalize the first word of a sentence,
-> Usually from data to text. subject-verb agreement, etc.).
Two aims: -> Structure: form “legal” HTML, RTF, or whatever
Understanding language production (Theoretical NLG) output format is desired.
Building partially useful systems (Practical NLG)
* Physical realization
NLG applications:
Automatic journalism
Reporting on sports results The “NLG pipeline” (be aware: the data analytics part is not yet
Textual feedback on health (e.g. Fitbit) discussed in here, and should be above document planning):
Agent (e.g. Siri) and dialogue systems
Financial reporting for companies
Image labeling
Weather forecasts
Blogging birds
How do NLG systems work? (traditionally 4 stages)
1. Data analytics and interpretation: making sense of the
data, looking for trends and patterns in the data. Multi-modal NLG: Sometimes the output is speech (i.e. spoken);
Text may be combined with visualizations.
2. Document planning: decide on the content and
structure of the text. Document planning consists of Knowledge sources
two elements: Where does the knowledge building a NLG system come from?
e.g. imitating a corpus (when available) -> most of the time this is
* Content selection/determination: of all things, I not available.
could inform you about, what should I tell you?
-> Depends on what is important or what makes Evaluation
a good narrative? Does the system help people? Do people like the text? etc. -> not
* Document structure: how should I organize this only evaluation on the text, but also look at the other phases
content as a text? (e.g. usefulness).
-> In what order do I say things? (what
information should you know first?) NLG (generation) vs. NLU (understanding)
-> What rhetorical structure? (the organization NLG is about generating/producing rather than understanding
that describes relations that hold between language. NLG and NLU are often combined: chatbots, Machine
parts of the text, ‘linking words’ cue phrases: Translation and Automated Text Summarisation.
also, because, but, etc.)
3. Micro-planning: decide how linguistically express text Natural language processing:
(which words, sentences, etc. to use; how to identify
objects, actions, times, etc.).Micro-planning consists of
three elements:
* Aggregation: how should the information be
distributed across sentences and paragraphs?
(e.g. phrase merging: he ate an apple, he ate an
orange → he ate an apple and an orange).
* Lexical/syntactic choice: which words and linguistic
structures do you use? (e.g. risky vs dangerous, 12 min
vs 720 sec, etc.)
* Reference (generation of referring expressions): How
should the text refer to objects and entities? (e.g.
when eating an apple: bite, first bite, first bite at 3:00
am, etc.)
4. Surface realization: Surface realization consists of two Lecture 2 (Software Engineering of NLG)
elements: = Practical NLG
* Linguistic realization: grammatical details (e.g. Software engineering
children vs. childs, an apple vs. a apple). Creating a real-world AI system requires more than coding and
algorithms:
-> Grammars (linguistic): form “legal” English * Requirements gathering: what should the system do?
, how easy is it to install and modify the system? However, “human” evaluation, as above, is very time intensive.
* Design: what are the modules, how do they interact? Computational metrics can be the alternative.
* Test and Q&A: does the system work?
* Support: adapting system to a changing world (updating), Metric example: BLUE
explaining system to customers. Task: let an algorithm measure the similarity between the “gold
standard” translation (which is usually written by an expert
When there is no corpus available: target text human translator) and a machine-made translation (made by the
Using corpus analysis for requirement analysis .But most of the designed NLG system).
times there is no corpus, or the corpus does not good represent
what the customers wants/needs. The basic idea of BLUE: compute n-gram overlap between
-> Solution: using “target text”: creating an text from data that machine translation output and the reference translation.
looks like the outcome you would like from the NLG system. Compute precision for n-grams of size 1 to 4 (this because shared
-> NLG developer will analyse from this what is possible and 5-grams tend to be rare).
what is not (and what is possible but hard to do). Can also -> Precision is the proportion of items in output that match an
suggest improvements. item in the reference
-> This is the opposite of recall, where the proportion of items
Example of a target text: in the reference match an item in the output).
Blue calculation (with size correction)
Step 1: Compute precision for n-grams of size 1 to 4.
->
Analysis of target text
Easy: rose and fell with ..%. -> But the precision would be high with common words
Hard: how to determine what very well. like ‘the’ -> so the Blue calculation needs a correction:
Impossible: the causality why it rose or fell. size correction. -> This is done in step 2.
-> maybe possible with other sources e.g. the New York times
for causality information. Step 2: Add brevity penalty (for too short translations). This
because, short words have an unfair advantage.
Target: “your portfolio did very well last week” -> Now the calculation will be like this:
It is easy to compute the numerical change in the portfolio.
But how should those numbers map to words like “very well”?
Target: “Your worst stock was Apple, which fell 8%”
Edge case 1: what if your worst stock still went up? (maybe use
other words choices such as: your least impressive stock was Blue example
Apple, which only increased by 2%).
Edge case 2: what if you have only one stock in your portfolio? or
not even one? (maybe drop this sentence in this case?).
Edge case 3: what if there is no rise or fall? or what if the change
is so small it will almost be none (0.00000042%)? Is it then even
worth mentioning?
NLG evaluation
Be aware!: you should not test the NLG system on the learning
cases, since of course then the output the NLG system gives will
be right, since you learned these examples to the system.
-> you would like to know if similar data would produces the
right output.
Discussion (Blue)
But safety and security do we really want to punish the system
for that? Also what to do if the system gives a better translation
compared to the human one? How/what should we calculate
then? And what about the difference between responsible and
responsibly? Also system A gets a 0 score, even though it
matched on certain things, is this fair? What about using two
references?
Babytalk: evaluation of BT45 Lecture 3 (A classic view of NLG systems: Reiter and Dale)
= Practical NLG
Questionnaires used on nurses:
165 trials: 90% of nurses said understandable, 70% said accurate The presentation
60% said helpful. Comments about BT45: many software bugs, NLG provides an information system. It should be easy to
more information wanted, few comments about language. understand (also for non-experts), while meeting the end-users
needs. The system uses text, graphics or a combination of both.
