Teaching Machines to Ask Clarification Questions

Teaching Machines to Ask Clarification Questions Sudha Rao Thesis Proposal Presentation May 12th

Natural Language Understanding

2

Natural Language Understanding Understan ding Tell me the recipe for lasagna

When was Barack Obama born?

3



Please bring me my coffee mug from the kitchen

4



Please bring me my coffee mug from the kitchen

5

Humans Interactions

6

Humans Interactions Hey Marge!

Hey Homer

7

Humans Interactions Hey Marge!

Hey Homer Today’s Today’s math ma th class clas s was sure sure fun

You You bet!

8

Grounding in Communication Hey Marge!


You You bet!

Clark, Herbert H., and Susan E. Brennan. "Grounding "Grounding in communication." Perspectives Perspectives on socially shared cognition 13.1991 (1991): 127-149. 9

Absence of of shared shared knowledge: knowledge: Ask a clarification question! question!

Hey Marge!


You You bet! Let’s Let’s meet m eet tomorrow tom orrow at 10 am to discuss our group assignment Sure. Where do you want to meet meet though?

10

Absence of of shared shared knowledge: knowledge: Ask a clarification question! question!

Hey Marge!


You You bet! Let’s Let’s meet m eet tomorrow tom orrow at 10 am to discuss our group assignment Sure. Where do you want to meet meet though? In 3rd floor grad lounge Sounds good!

11

Teach Machines to Ask Clarification Clarification Questions

12

Teach Machines to Ask Clarification Clarification Questions Please bring me my coffee mug from the kitchen

What color is your coffee mug?

13



Tell me how to bake a cake! Sure! Which cake do you want to bake?

14



Context-aware questions about missing information in text

Tell me how to bake a cake! Sure! Which cake do you want to bake?

15

This is a hard problem!

16

PRIOR WORK WORK IN QUESTION GENERATION

17

RELATED WORK

Reading Comprehension Questions My class is going to the movies on a field trip next week. We have to get permission permi ssion slips signed before we go. We also need to ask our parents if they will drive to the movie theatre. We are going to see a movie that tells the story from a book we read. We love it when movies are made from books. It is fun to compare movie to the book. I usually like the book better. We get to the movie early so we can buy popcorn. Some of us buy candy and slushes too. We all enjoy watching the movie. When we return to the school, we talk about things that were in the movie and the book. The movie and book are similar.

Michael Heilman. 2011. Automatic factual question generation from text Ph.D. thesis, CMU 18

RELATED WORK

Reading Comprehension Questions My class is going to the movies on a field trip next week. We have to get permission permi ssion slips signed before we go. We also need to ask our parents if they will drive to the movie

Q: What do the students need to do before going to the movies?

theatre. We are going to see a movie that tells the story from a book we read. We love it when movies are made from books. It is fun to compare movie to the book. I usually like the book better. We get to the movie early so we can buy popcorn. Some of us buy candy and slushes too. We all enjoy watching the movie. When we return to the school, we talk about things that were in the movie and the book. The movie and book are similar.


RELATED WORK

Reading Comprehension Questions My class is going to the movies on a field trip next week. We have to get permission permi ssion slips signed before we go. We also need to ask our parents if they will drive to the movie

Q: What do the students need to do before going to the movies?

theatre. We are going to see a movie that tells the story from a book we read. We love it when movies are made from books. It is fun to compare movie to the book. I usually like the book better. We get to the movie early so we can buy popcorn. Some of us buy candy and slushes too. We all enjoy watching the movie. When we return to the school, we talk about

GOAL

Assess someone’s someone’s understanding of the text

things that were in the movie and the book. The movie and book are similar.


RELATED WORK

Questions for f or Literature Literature Review Writing Support writ e a better literature review review Goal: Help students write

Liu, Ming, Rafael A. Calvo, and Vasile Rus. "Automatic question generation for literature review writing support." International Conference on Intelligent Tutoring Systems . Springer Berlin Heidelberg, 2010. 21

RELATED WORK


Cannon (1927) challenged this view mentioning that physiological changes were were not sufficient to discriminate emotions


RELATED WORK



Why did Cannon challenge this view?


RELATED WORK



Why did Cannon challenge this view?

What evidence is provided by Cannon to prove the opinion?


RELATED WORK

Bootstrapping semantic parsing parsi ng from conversations conversations SYSTEM : How can I help you? you? USER:

I would ould lik like to fly fly fr from At Atla lant nta a Geor Georgia gia to London England on September 24th in the early evening. I would like to return on October 1 st departing from London in the late morning.

Artzi, Yoav oav,, and Luke Zettlemoyer Zettlemoyer.. "Bootstrapping semantic parsers from from conversations” EMNLP 2011. 25

RELATED WORK



SYSTEM : Leaving what city? USER:

Atlanta Georgia


RELATED WORK



SYSTEM : Leaving what city? USER:

Atlanta Georgia

SYSTEM : Going to which city? USER:

London

[conversation [conversation continues]


RELATED WORK

Natural Questions about Images

Mostafazadeh, Nasrin, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. "Generating natural questions about an image." Association of Computational Linguistics 2016 28

RELATED WORK


Caption : A man standing standin g next to a motorcycle


RELATED WORK


Q: Was anyone injured in the crash? Q: Is the motorcyclist alive? Q: What caused the accident?

Caption : A man standing standin g next to a motorcycle


Talk Outline o

Problem Overview

o

Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l

o

Clarification Questions for Question-Answering forums

o

Clarification Questions for Dialogues

o

Proposed Work

o

Generalizability bey beyond Q&A forums and Dialogues

31

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


32

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


33

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


34

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


35

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


36

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


37

Problem Overview Information Content

38


process

Achieve Achieve a goal

39


process


!

Information retrieval retrieval ! Problem solving !

Taking an action

40


process


!


Taking an action

Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?

Setup a meeting

41


process


!


Taking an action

Missing Information


Setup a meeting

42


process


!


Taking an action

Missing Information


Setup a meeting

43


process


!


Missing Information

Taking an action

Clarification Question


Setup a meeting

44


process


!


Missing Information

Taking an action



Setup a meeting

Hey Homer, Sure. Where do you want to meet though? 45


process


!


Missing Information

Taking an action



Setup a meeting

Hey Homer,

In 3rd floor grad lounge

Sure. Where do you want to meet though? 46

Problem Overview 1

Information Content

process


!


Missing Information

Taking an action



Setup a meeting

Hey Homer,



Problem Overview 1

Information Content

process


!


Missing Information

2

Taking an action



Setup a meeting

Hey Homer,



Problem Overview 1

Information Content

process


!


3

Missing Information

2

Taking an action



Setup a meeting

Hey Homer,



Problem Overview 1

Information Content 3

Missing Information

2


50

Problem Overview 1


Missing Information

1

2


2

Context

3

Question

Answer

51

Problem Overview 1


Missing Information

1

2


2

Context

Let us meet at 10 am tomorrow to discuss our group assignment?

3

Question

Where do you want to meet though?

Answer


52

Problem Formulation

Context


53

Problem Formulation Generate Question Candidates Context


Who are you?


What will we be discussing?

54

Problem Formulation Generate Question Candidates Context


Rank the question candidates

Who are you?

1



2



10

Who are you?

55

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


56

Expected Value Value of Perfect Information (EVPI) inspired model

Key Idea Hey Marge, M arge, Let us meet at 10 am tomorrow to discuss our group assignment

Possible questions (a) Where do you want to meet?

" Just right

57


Key Idea Hey Marge, M arge, Let us meet at 10 am tomorrow to discuss our group assignment

Possible questions (a) Where do you want to meet?

" Just right

(b) Is (b) Is the moon waning waning or waxing? waxing?

"

Not useful

(c) Did (c) Did you you see the new new homework homework? ?

"

Does not add value

58


Definition of EVPI

Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 59


Definition of EVPI

o

What is the value of gathering an additional information inf ormation x?



Definition of EVPI

o




Definition of EVPI

o


o

Since we have not acquired x, we define its value in expectation



Definition of EVPI

o


o

o

Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood.



Definition of EVPI

o


o

o

Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood.

EVPI =



Definition of EVPI

o


o

o

Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood. Likelihood Likelihood of x

EVPI =



Definition of EVPI

o


o

o

Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood. Likelihood Likelihood of x

EVPI = Value Value or Utility of x Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 66


EVPI formulation for our problem

67


EVPI formulation for our problem

c:

given given information content

q: question from a set of question candidates Q

68


EVPI formulation for our problem Ask question ‘q’ that maximizes the expected expected utility of the updated information

c:


q: question from a set of question candidates Q

69


EVPI formulation for our problem Ask question ‘q’ that maximizes the expected expected utility of the updated information

c:


q: question from a set of question candidates Q a:

answer from a set of answer candidates A

70


EVPI formulation for our problem Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’

c:


from a set of question candidates candidates Q q: question from a:

answer from a set of

answer candidates candid ates A

71



Utility of updating content ‘c’ with answer ‘a’

c:





72




c:




1


73


EVPI formulation for our problem 2

Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’


c:




1


74



Answer Modeling


c:




1


75



Answer Modeling

3 c:





1


76



Answer Modeling

3 c:

Utility Calculator




1


77


1

Question & Answer candidates

78


1


79


1


80


1


81


1


82


1


83


84


2

Answer Answer Modeling


85


2



Approach: !

We define the likelihood function as: Pr(ak|c j,q j) = closeness(a k , F(c j, q j) )

86


2



Approach: !


! F(c j, q j): Trained to be close to the true answer a j

87


2



Approach: !

We define the likelihood function as:



Pr(ak|c j,q j) = closeness(a k , F(c j, q j) ) ! F(c j, q j): Trained to be close to the true answer a j


88


2



Approach: !



and to answers whose questions are similar to q j

89


2




Approach: !

We define the likelihood function as:

Do you have a place in mind?

Pr(ak|c j,q j) = closeness(a k , F(c j, q j) ) ! F(c j, q j): Trained to be close to the true answer a j

In a coffee shop?


90


2



Approach: !




91


2

1



92


2

1



3

Utility Calculator

93


3

Utility Calculator


94


3

Utility Calculator

Utility of updating content ‘c’ with answer ‘a’ Approach

Initial contents

low utility

Updated contents

high utility

95


3

Utility Calculator


!

Initial contents

low utility

Updated contents

high utility

Let us meet tomorrow at 10am to discuss the next group assignment?

Let us meet tomorrow at 10am in 3rd floor grad lounge to discuss the next group assignment?

96


3

Utility Calculator


Initial contents

low uti utility

y=0

Updated contents

high utility

y=1



97


3

Utility Calculator


!

Initial contents

low uti utility

y=0

Updated contents

high utility

y=1

!

Train F(c,a) to minimize cross-entropy

!

U(c+a) U(c+a ) = Value between 0 and 1



98


3

Utility Calculator

Utility of updating content ‘c’ with answer ‘a’ Approach !

Initial contents

low uti utility

y=0

!

Updated contents

high utility

y=1

!

Train F(c,a) to minimize cross-entropy

!

U(c+a) U(c+a ) = Value between 0 and 1 99


Our EVPI inspired model


Answer Modeling

Utility Calculator

100


Our EVPI inspired model


Answer Modeling

Trained using a joint loss function =

Utility Calculator

!i loss ans + loss utiil 101

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


102

Clarification Questions for f or Question-Answering Forums Forums

Dataset

103


How to configure path or set environment variables for installation? i'm aiming to install ape, a simple code for pseudopotential generation. i'm having this error message while running ./configure

Initial Post

So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!

104



Initial Post


Question comment

What version of ubuntu do you have?

105



Initial Post

Question comment


What version of ubuntu do you have? i'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. i'm having this error message while running ./configure So I have the library but the program installation isn't finding it.

Updated Post

Any help? help? Thanks Thanks in advance! advance!

106



Initial Post

Question comment


What version of ubuntu do you have? i'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. i'm having this error message while

Edit as an answer to the question

running ./configure So I have the library but the program installation isn't finding it.

Updated Post


107



Initial Post

Question comment


What version of ubuntu do you have? i'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. i'm having this error message while

Edit as an answer to the question

running ./configure So I have the library but the program installation isn't finding it.

Updated Post


108


Dataset Creation (post, question, answer) triples Post:

Original post

Question:

Clarification question posted in comments

Answer:

Edit made to the post answering the question

109



Original post

Context

Question:


Question

Answer:


Answer

110



Original post

Context

Question:


Question

Answer:


Answer

We extract a total of 37,000 such triples from three related domains on StackExchange: askubuntu, unix & superuser 111

Clarification Questions for f or Question-Answering Forums Forums USING OUR EVPI MODEL

112


Question & Answer Candidate Generator

Using Lucene

113



Using Lucene

Answer Modeling

Utility Calculator

Using Long Short Term Memory (LSTM) Model

114

Clarification Questions for f or Question-Answering Forums Forums NEURAL NETWORK MODEL

Answer Modeling

Utility Calculator

115

Clarification Questions for f or Question-Answering Forums Forums LONG SHORT TERM MEMORY (LSTM)

Sepp Hochreiter and J¨urgen Schmidhuber. 1997. Long short-term memory. Neural computation , 9(8):1735–1780. Jeffrey Jeffrey Pennington, Pennington, Richard Socher, Socher, and Christopher D Manning. 2014. “Glove: “Glove: Global vectors for word word representation” representation” In Empirical Methods on Natural Language Processing. Processing. 116



117



118



119



120



121



122


Utility Calculator

123


Utility Calculator i'm aiming to install ape, a simple code

F(p, a)

!

0

for pseudopotential generation. i'm having this error message while running ./configure

Initial Post


124


Utility Calculator i'm aiming to install ape, a simple code

F(p, a)

!

0

for pseudopotential generation. i'm having this error message while running ./configure

Initial Post


i'm aiming to install ape in Ubuntu 14.04 LTS,

F(p, a)

!

1

a simple code for f or pseudopotential generation. i'm having this error message

Updated Post

while running running ./configure ./configure So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance! 125


Experiments

126


Experimental Results Accuracy

MRR

Recall@3

Recall@5

Random Bag-of-ngrams Neural baseline EVPI Neural

Accuracy: Accuracy: How How often is correct correct the top 1 in 10 MRR:

Mean Reciprocal Rank

Recall@3: How often is correct in top 3

Dataset Askubuntu Unix Superuser

~14K ~7K ~17 K


127


Experimental Results Random

Accuracy

MRR

Recall@3

Recall@5

10.0

29.3

30.0

50.0

Bag-of-ngrams Neural baseline EVPI Neural





~14K ~7K ~17 K


128



MRR

Recall@3

Recall@5

Random

10.0

29.3

30.0

50.0

Bag-of-ngrams

11.6

31.3

32.5

54.6

Neural baseline EVPI Neural





~14K ~7K ~17 K


129


Neural Baseline

EVPI Neural

Answer Modeling

Utility Calculator

130



MRR

Recall@3

Recall@5

Random

10.0

29.3

30.0

50.0

Bag-of-ngrams

11.6

31.3

32.5

54.6

Neural baseline

17.4

37.8

43.2

63.9

EVPI Neural





~14K ~7K ~17 K


131



MRR

Recall@3

Recall@5

Random

10.0

29.3

30.0

50.0

Bag-of-ngrams

11.6

31.3

32.5

54.6

Neural baseline

17.4

37.8

43.2

63.9

EVPI Neural

23.3

43.4

51.0

70.3





~14K ~7K ~17 K


132


How does our model compare to non-expert humans?

133


Annotation scheme

o

o

14 computer science graduate students perform annotations on 50 examples Given a post and a set of ten candidate questions #

Mark the one you think is the right one # Mark all the ones that you think may be valid o

Human annotators found this task very hard!

134


Sample human annotation task

135


Our model choses the right question

136


Our model choses a valid question

137


Our model choses a wrong question

138


Findings from human annotations

o

o

o

Annotator’s Annotator’s accuracy was 36%, significantly better than our best system (23%) If we limit to examples on which they were more confident, their accuracy raised to 42%, but never surpassed that Aver Average age number of “valid” answers for a single post was 4.26 (out of ten)

139

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


140


Ubuntu Chat Dataset

is anyone having trouble with ubuntu ubuntu server 8.04 with a slow apt-get update? what country country do you you live live in and where are you getting the sources from ?

us.archive.ubuntu.com live in the US which state? state? michigan

thanks!

you could try using a mirror that is closer to home: https://launchpad.net/ ubuntu/+archivemirrors

Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue.

141


Ubuntu Chat Dataset


CONTEXT

us.archive.ubuntu.com live in the US which state? state? michigan

thanks!



142


Ubuntu Chat Dataset


CONTEXT

us.archive.ubuntu.com live in the US which state? state?

RESPONSE

michigan

thanks!



143


Our Dialogue Dataset


CONTEXT


QUESTION

michigan

thanks!


144


Our Dialogue Dataset


CONTEXT


ANSWER

QUESTION

michigan

thanks!


145

Clarification Questions for Dialogues (CONTEXT, QUESTION, ANSWER) TRIPLES

Context:

Context of the conversation

Question:

Clarification question following the context

Answer:

Response Respons e following the question questi on

146

Clarification Questions for Dialogues (CONTEXT, QUESTION, ANSWER) TRIPLES

Context:

Context of the conversation

Question:

Clarification question following the context

Question

Answer:

Response Respons e following the question questi on

Answer

Context

147

Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o

Identify responses with ‘?’

148



is anyone having trouble with ubuntu ubuntu server 8.04 8.04 with a slow slow apt-get apt-get update? what country country do you you live in and where are you getting the sources from ? us.archive.ubuntu.com live in the US which state? state?

149



o

Not all questions are clarifications

150



o


hi, what is the best filesystem to use on a usb thumb drive? ext2 and ext3 seem to constantly write to the drive

what's wrong wrong with vfat ?

isnt that too microsoftish?

i'd just use whatever works, works, honestly honestly :)

151



o

Not all questions are clarifications Collect human annotations on a subset : Is this a clarification question?

152



o


Train a classifier to identify clarifications

153



o



o

Not all clarifications clar ifications are answer answered ed in the next turn

154



o



o

Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?

155


ONGOING WORK

EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o


o



o


156


ONGOING WORK



o


90%

Collect human annotations on a subset : Is this a clarification question?


o


157


ONGOING WORK



o


90%

Collect human annotations on a subset :

45%

Is this a clarification question?


o


158


ONGOING WORK



o


90%


45%



o


35%

159


ONGOING WORK



o


90%


45%



o


o

35%

35% of 1 million conversations from Ubuntu Dataset

160


ONGOING WORK



o


90%


45%



o


o

35%


Classifier confidence 161



o


90%


45%



o


o

35%


Classifier confidence

~200K (context, question, answer) 162

Clarification Questions for Dialogues USING OUR EVPI MODEL

163



Using Lucene

164



Using Lucene

Answer Modeling

Utility Calculator

Using an Ensemble of Neural Dialogue Model

165

Clarification Questions for Dialogues LONG SHORT TERM MEMORY MEMORY (LSTM) BASELINE

Context LSTM C

Hello there there how how are you you ?

σ(cTMr

+ b)

Response LSTM

I am fine thank you you .

r

Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue. 166

Clarification Questions for Dialogues LONG SHORT TERM MEMORY MEMORY (LSTM) BASELINE

Context LSTM o

C

Hello there there how how are you you ?

o

σ(cTMr

+ b)

Two LSTMs with tied parameters Trained to minimize crossentropy between all context, response pairs

Response LSTM

I am fine thank you you .

r


Clarification Questions for Dialogues ATTENTION OVER OVER THE CONTEXT

C

Σ

Hello there how are

you

?

168

Clarification Questions for Dialogues ATTENTION OVER OVER THE CONTEXT

C

C

Σ

Hello there how are

A

you

?

Hello there how are

you

?

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conf erence on Learning Representations . 169

Clarification Questions for Dialogues HIERARCHICAL MODEL OVER THE UTTERANCES

Token level LSTM

A

hum ,

just

mail

A

me

.

I’ll

follow the mail on the internal list

A

Yes Yes .

Aaron Courville, and Yoshua Bengio.“A hierarchical latent latent variable encoder-decoder model for generating dialogues AAAI AAAI 2017 170

Clarification Questions for Dialogues HIERARCHICAL MODEL OVER THE UTTERANCES A

C

Utterance level LSTM

Token level LSTM

A

hum ,

just

mail

A

me

.

I’ll

follow the mail on the internal list

A

Yes Yes .

Aaron Courville, and Yoshua Bengio.“A hierarchical latent latent variable encoder-decoder model for generating dialogues AAAI AAAI 2017 171

Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM

What? Extract features below token level

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 172


What? Extract features below token level Why?

Misspelling, elided whitespaces, tokenization errors



What? Extract features below token level Why?

Misspelling, elided whitespaces, tokenization errors

How?

Trigram histogram


Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM Example: What? Extract features below token level

“Bananas”

ban - 1 ana - 2

Why?

Misspelling,

nan - 1

elided whitespaces,

nas - 1

tokenization errors How?

Trigram histogram



“Bananas”

ban - 1 ana - 2

Why?

Misspelling,

nan - 1

elided whitespaces,

nas - 1


Trigram histogram

Trigram histogram 0 1 0 0 10 2 0 0 1 0 Size of trigram vocabulary



“Bananas”

ban - 1 ana - 2

Why?

Misspelling,

nan - 1

elided whitespaces,

nas - 1


Trigram histogram 0 1 0 0 10 2 0 0 1 0

Trigram histogram

Size of trigram vocabulary

Input representation

Vocabulary Vocabulary

OOV

Trigram

OOT



ban - 1 “Bannas”

ana - 2

Why?

Misspelling,

nan - 1

elided whitespaces,

nas - 1


“Bananas”

Trigram histogram 0 1 0 0 10 2 0 0 1 0

Trigram histogram

Size of trigram vocabulary

Input representation

Vocabulary Vocabulary

OOV

Trigram

OOT



Experiments

179

Clarification Questions for Dialogues PRELIMINARY PRELIMINARY RESULTS ON NEXT UTTERANCE CLASSIFICATION

TASK: ASK:

Giv Given con conte text xt of of a con convers versat atio ion, n, select the correct response from 10 responses chosen at random


Clarification Questions for Dialogues PRELIMINARY PRELIMINARY RESULTS ON NEXT UTTERANCE CLASSIFICATION

TASK: ASK:

Giv Given con conte text xt of of a con convers versat atio ion, n, select the correct response from 10 responses chosen at random

Model

Accuracy

Recall @2

Recall @5

TF-IDF

0.48

0.58

0.76

LSTM

0.55

0.72

0.92

+ Attention

0.60

0.77

0.95

+ Hierarchical Hierarchical

0.62

0.79

0.95

+ Trigram Histogram

0.64

0.80

0.96


Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


182

1. TEMPLATE BASED QUESTION QUESTION GENERATION

183

PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION KEY IDEA

“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?”

184



Unseen context about ‘gcc’

I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs with GCC. I removed and reinstalled everything but every time I get this error as: unrecognized option ‘--32’

185




Template:


What version of ____ are are you using?

186




Template:


What version of ____ are are you using?

What version of gcc are you using?

187

PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD

1.

Identify groups of similar questions

“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt -get are you running?” running ?”

188


1.

Identify groups of similar questions Clustering algorithms


189


1.


2.


Generate a template for each group

190


1.


2.

“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?” running ?”

Generate a template for each group !

Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds

!

Remove them to form template

191


1.


2.




!


What version of ____ are you using?

192


1.


2.

3.




!



Given a context, select a template templat e from a candidate candid ate set of templates templa tes

193


1.


2.

3.




!



Given a context, select a template templat e from a candidate candid ate set of templates templa tes !

Using our EVPI model

194


1.


2.

3.

“What version of yum are you using?” “What version of apt-get are you running?” running ?”



!




4.

“What version of Ubuntu are you using?”


Fill in the blanks of the template templa te using topic words words from the context

195


1.


2.

3.




!




4.



Fill in the blanks of the template templa te using topic words words from the context !

Identify candidate topic words from the context

196


1.


2.

3.




!




4.



Fill in the blanks of the template templa te using topic words words from the context !

Identify candidate topic words from the context

!

Train a model to select the correct topic word to fill in the template

197

2. SEQUENCE-TO-SEQUENCE BASED QUESTION GENERATION

198

PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o

Sequence-to-sequence neural network models !

Machine Translation

!

Dialog Generation

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 199


o


Machine Translation

!

Dialog Generation

Given an input sequence, generate output sequence one word at a time



o


Machine Translation

!

Dialog Generation

Given an input sequence, generate output sequence one word at a time

A

B

C



o


Machine Translation

!

Dialog Generation

Given an input sequence, generate output sequence one word at a time W

A

B

C



o


Machine Translation

!

Dialog Generation


A

B

C

X

W



o


Machine Translation

!

Dialog Generation


A

B

C

X

Y

W

X



o


Machine Translation

!

Dialog Generation


A

B

C

X

Y

Z

W

X

Y

Z


PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION Neural Generative Question Answering Model

He is 2.29 m and visible from space

How Ho w tall is Yao Ming ?

Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative generative question answering. answering. In North American Association of Computational Linguistics . 206






Knowledge Base




PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION PROPOSED METHOD o

We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context

Which version of Ubuntu ar are e you using?

How to configure environment variable? ….

209


o

We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context At each time step, step, our model will decide whether to



210


o

We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context At each time step, step, our model will decide whether to !

Generate a template word OR

Which version version of ____ are are you you using?


211


o

We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context At each time step, step, our model will decide whether to !

Generate a template word OR

!

Generate a topic specific word from context



212

3. USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION

213

PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION

KEY IDEA

Post related to Ubuntu Operating System

214


KEY IDEA

Operating systems !

!

Knowledge Base


215


KEY IDEA

Operating systems !

!

Knowledge Base


What version of Ubuntu are you using?

216


KEY IDEA

Operating systems !

!

Knowledge Base

Songs !

!

Post related to Ubuntu

Conversation

Operating System

about a song

What version of Ubuntu are you using?

When was the song released?

217


PROPOSED METHOD

1. Identify topics of discussion in a given context

218


PROPOSED METHOD



219


PROPOSED METHOD

1. Identify topics of discussion in a given context GCC

2. Extract attribute information from Dbpedia

!

!

220


PROPOSED METHOD


I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs


with GCC. I removed and reinstalled everything but every time I get this error as:

3. Identify the missing attribute in the context

unrecognized option ‘--32’

221


PROPOSED METHOD


I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs


with GCC. I removed and reinstalled everything but every time I get this error as:

3. Identify the missing attribute in the context

unrecognized option ‘--32’

4. Ask 4. Ask a question about the missing attribute What version of gcc are you using?

222

Talk Outline o

Problem Overview

o


o


o


o

Proposed Work

o


223

GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES INTERACTIVE QUERY SEARCH o

Traditional search approaches rely heavily on relevance feedback

224


o

Traditional search approaches rely heavily on relevance feedback However with the increase of interactive search agents like Siri, Alexa etc, we will no longer longer be clicking on links

225


o

o

Traditional search approaches rely heavily on relevance feedback However with the increase of interactive search agents like Siri, Alexa etc, we will no longer longer be clicking on links Hence there is a need for the system to narrow down on the search criteria by asking questions

226


o

o

o

Traditional search approaches rely heavily on relevance feedback However with the increase of interactive search agents like Siri, Alexa etc, we will no longer longer be clicking on links Hence there is a need for the system to narrow down on the search criteria by asking questions Ask clarification questions is one way of doing that!

227

GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES WRITING ASSISTANCE

Hi Marge, Let us meet tomorrow tomorrow at 10am to discuss the next group assignment? Hey Homer, H omer, Sure. Where do you want to meet though? Oh right. Forgot to mention that. Let us meet in the 3 rd floor grad lounge.

228

GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES WRITING ASSISTANCE

Hi Marge,

Do you want to suggest a place?

Let us meet tomorrow tomorrow at 10am to discuss the next group assignment?

Hi Marge, Let us meet tomorro tomor row w at 10am in 3rd floor grad lounge to discuss the next group assignment? Sounds good!

229

CONCLUSION !

Identify the importance of teaching machines to ask clarification clar ification questions

!

Introduce two novel datasets for this problem o o

Question-Answering Forums Dialogues

!

Introduce a novel model inspired by Expected Value of Perfect Perfect Information to rank candidate questions

!

Propose methods to generate questions (instead of select)

230

TIMELINE November 2017


March 2017

Template based question generation

June 2018

Sequence-to-sequence based question generation generation

September 2018

Identify missing information using u sing knowledge knowledge bases

December 2018

Thesis writing and defense

231

Sudha Rao, Yogarshi Vyas, Hal Daumé III, and Philip Resnik, ”Parser for Abstract Meaning Representation using Learning to Search", NAACL 2016 Workshop Workshop on Meaning Representation Parsing

SEMANTICS Sudha Rao, Daniel Marcu, Kevin Knight and Hal Daumé III, "Biomedical Event Extraction using Abstract Meaning Representation" In Submission

Sudha Rao, Hal Daumé III, " Are you asking the the right questions? Automatically Generating Generating Clarification Questions" In submission

Rao, Sudha, Allyson Ettinger, Hal Daumé III, and Philip Resnik. "Dialogue focus tracking for zero pronoun resolution.” NAACL N AACL 2015

QUESTION GENERATION

DIALOGUES Sudha Rao, Paul Mineiro, " A Play on Words: Redefining Vocabulary for Dialogue Modeling" In submission

232

233

Teaching Machines to Ask Clarification Questions

Recommend Documents