Week 3

Improving the Generation Component + Thinking About The Evaluations

Posted by Stuart Chen on June 16, 2019

The Week 3

This week, I have replaced the exited tools with my own innovative and independent modules pertinent to DBpedia. However, after doing so, the problem emerges while doing the BLEU evaluation.

I have buried myself into coding and checking on this weekends, while spending the whole weekend in evaluating and considering the flexibility of the generation component.

At present, the component only accepts the limited questions matched to certain regexes and generate the template queries that do not contain explicit URLs.

  1. the current component rejects the questions’ formats in annotations_monument.csv

  2. the function is so restricted that, the question which it had not accepted would be accepted well after fine-tuning some uppper/lower class or even arranging the order: 1) the syntactic parsing dependency is not working very well in transforming the similar question in the templates; 2) the regex matching stays fragile, not robust enough to catch the syntactical structure of a new question; 3) while doing the BLEU, I replace the variables in the templates with exact vocabularies, and the module could not handle it and refused to generate the template queries, which tells me that it is far from perfect.

The work in this week’s goals:

  1. Improving the Generation Component

  2. Thinking About The Evaluations

1. Improving the Generation Component

Thinking deeper to find a method to more intelligently represent the syntactic structures of the questions.

For two reasons:

1) It helps to improve the automation of templates generation, especially in query templates generation, for which I have worked on three paths: Syntactic parsing with entity detection to seize the structure, that?s the currently primary option but not highly automatic; Use sequence to sequence LSTM to train a model for natural language to query, which is estimated to be quite > time-consuming; Deploy the Universal Sentence Encoder to embed the syntactic structures into embedding vectors as the representation, which would also help a lot in the next phase of project while this requires some endeavors.

2) If the representation of the syntactic structures is efficient enough, it will accelerate the computation for matching the templates in the Templates Bank. It reinforces the grounding of Templates Comparison, which is to be one of the goals in next week too. And from this, it derives two aspects of further consideration: the storing of the templates and their representation, whether to store in the database or, maybe more efficient for computing the matching, in Hadoop/Spark via the python API; Still, the representation of the templates or their syntactic structures, for easier matching, like calculating the similarity of two embedding vectors to get the most matched.

2. Thinking About The Evaluations

1) After two days’ trial on evaluating the performance, the lacking in robustness is apparent after the changing in this week. So, how to improve the model based on the experiments that we had. First, the dependency parsing might be a good idea. Second, the regexes should be more intelligent.

2) We will try on the GERBIL plateform, for its convienience.

The Commit of This Week

This is the summing-up the works that I have done.

For the next week, I will do more works concerning about the dependency parsing which pertains to the automated query template generation with robustness.