{"id":28133,"date":"2022-12-07T07:14:18","date_gmt":"2022-12-07T12:14:18","guid":{"rendered":"https:\/\/www.bloomberg.com\/company\/stories\/bloombergs-ai-engineering-group-cto-publish-5-nlp-research-papers-at-emnlp-2022\/"},"modified":"2024-04-12T10:21:03","modified_gmt":"2024-04-12T14:21:03","slug":"bloombergs-ai-engineering-group-cto-publish-5-nlp-research-papers-at-emnlp-2022","status":"publish","type":"post","link":"https:\/\/www.bloomberg.com\/company\/stories\/bloombergs-ai-engineering-group-cto-publish-5-nlp-research-papers-at-emnlp-2022\/","title":{"rendered":"Bloomberg\u2019s AI Engineering Group & CTO Office Publish 5 NLP Research Papers at EMNLP 2022"},"content":{"rendered":"
\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t
\n \n

During the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)<\/a> in Abu Dhabi this week, researchers from Bloomberg\u2019s AI Engineering Group<\/a> and CTO Office are showcasing their expertise in natural language processing (NLP) by publishing five papers. Three papers will appear in Findings of EMNLP 2022. One of these, along with another paper, is also being presented in the virtual poster session during the Second Version of Generation, Evaluation & Metrics (GEM) Workshop 2022<\/a>. A fifth paper will be presented at the 2nd Multilingual Representation Learning (MRL) Workshop<\/a>.<\/p>\n

Through these papers (links and PDFs of the papers will be added once the Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em> is available online), the authors and their collaborators highlight a variety of NLP applications, novel approaches and improved models used in key tasks, and other advances to the state-of-the-art in the field of computational linguistics.<\/p>\n

We asked some of the authors to summarize their research and explain why the results were notable:<\/p>\n

\n

<\/h3>\n

Sequentially Controlled Text Generation
\n<\/strong>Alexander Spangher (USC Viterbi School of Engineering\/Bloomberg Data Science Ph.D. Fellow), Yao Ming\/Xinyu Hua (Bloomberg), Nanyun Peng (UCLA)<\/p>\n

Findings of EMNLP 2022
\n<\/em>GEM Workshop<\/a>, Virtual Poster Session (Wednesday, December 7, 2022 @ 9:00 PM GST)<\/em>
\n<\/em><\/p>\n\n<\/div>\n

\n\t
\n<\/div>\n
\n \n

Please summarize your research. Why are your results notable?<\/strong><\/p>\n

Xinyu:<\/strong> We started off with a basic problem: different consumers of news require varied levels of background context, prefer specific types of article structures depending on what platform they\u2019re reading the news and how much time they have to consume it, and often have different reasons for reading the content. So, given the main subject of a news article, could we fill in relevant contextual information? Could we experiment with different structural modes of storytelling? Could we personalize the news article to fit the different users\u2019 needs?<\/p>\n

The unifying principle behind all these questions is: how can we control the macro-structure of the news story we wish to write? We identified this core research question and formulated a novel AI task to address it, something we call \u201csequentially controlled text generation.\u201d In this task, the algorithm is provided with a topic (i.e., in the form of a prompt) and a sequence of structural codes (i.e., \u201cMain Event\u201d -> \u201cBackground\u201d -> \u201cPrevious Event\u201d) that govern \u2013 on a sentence-level \u2013 the macro-structure we wish the generated story to exhibit.<\/p>\n

Alex:<\/strong> In our work, we (1) develop baseline methods to solve this task and (2) further study how much structural awareness during story generation contributes to well-structured and fluent text. In our experiments, we use headlines as textual input to the language model and Van Dijk discourse tags (News as Discourse<\/a>. Routledge. 2013.) as control codes, as illustrated in the Figure below, where \u201cMain Event,\u201d \u201cHistorical Event,\u201d and \u201cExpectation\u201d are the control codes, conditioned on the title \u201cNeo-Nazi murder gang member jailed for life in Germany.\u201d<\/p>\n\n<\/div>\n

\n \"\"\"\"\n
Figure 1<\/i><\/b>: An example illustrating the sequential structure of a news article. Our model aims to faithfully generate the sentences conditioned on the headline and discourse tags (Main Event, Historical Event, and Expectation).<\/span><\/i><\/figcaption>\n<\/figure>\n
\n \n

Our system combines two important approaches in text generation: generation and editing. During generation, we first perturb the output of a language model using a structurally-aware classifier and generate the next word by sampling from the perturbed distribution. Editing is performed at the end of each generated sentence: we identify the most \u201csalient\u201d words, which contribute the most to the discriminator\u2019s prediction on the desired discourse class.<\/span><\/p>\n

We use heuristics based on part-of-speech to encourage the editor to introduce explicit discourse markers, and we fine-tune a label-aware text infilling model to generate candidate edits given the masked input, which is repeated until there is an increase in likelihood of the desired control code.<\/span><\/p>\n\n<\/div>\n

\n \"\"\"\"\n
Figure 2:<\/i><\/b> Our system\u2019s generation process. We first generate word xi by sampling from a language model which is controlled by a discriminator reflecting the desired discourse tag. Then, during our editing stage, the discriminator helps identify class-salient words, which are then masked and re-generated to boost the likelihood of the desired discourse tag.<\/figcaption>\n<\/figure>\n
\n \n

We experiment on the <\/span>NewsDiscourse dataset<\/span><\/a> and conduct human evaluation against four metrics: Accuracy, Grammar, Logical Flow, and Topicality. Our results show that (1) past structural information boosts class accuracy the most; (2) weak discriminators can still impose accurate control; and (3) editing has an overall positive effect on both generation accuracy and quality.<\/span><\/p>\n

In text generation, the discourse structure impacts both human and machine comprehension. Although naive language models have made impressive advancements and generate fluent text, the text is structurally dissimilar to text written by humans. We envision this system will enable journalists to quickly prototype different structures for their work, or fill in missing structural components to aid in human-in-the-loop computational journalism.<\/span><\/p>\n

How does your research advance the state-of-the-art in the field of natural language processing?<\/b><\/p>\n

Xinyu: <\/b>The controllability issue of neural generation models is the key hurdle for their real world adoption. Existing controllable models focus on single control code (i.e., one signal per document). Our work tackles structured control, which allows for more fine-grained, sequential control based on discourse structures. Our system could vary the degree of control, from local-only to past-aware, and even to full-sequence control. Furthermore, we employ editing as part of our pipeline to continuously improve the output quality. We showcase this system on a news dataset and conduct extensive human evaluation to confirm the improved generation quality and resemblance to human-written text over the original, uncontrolled GPT-based model.<\/span><\/p>\n\n<\/div>\n\n<\/div>\n\n\n\t\t\n\t<\/div>\n<\/div>\n

\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t
\n\t
\n\t\t
\n\t\t\t

Make it happen here.<\/p>\n\t\t\t

<\/p>\n\t\t<\/div>\n\t\tSEARCH NOW<\/a>\n\t<\/div>\n<\/div>\n\n<\/div>\n\n\n\t\t\n\t<\/div>\n<\/div>\n

\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t
\n \n

Realistic Data Augmentation Framework for Enhancing Tabular Reasoning<\/strong>
\nDibyakanti Kumar (IIT Guwahati), Vivek Gupta (University of Utah\/Bloomberg Data Science Ph.D. Fellow), Soumya Sharma (IIT Kharagpur), Shuo Zhang (Bloomberg)<\/p>\n

Findings of EMNLP 2022<\/em><\/p>\n\n<\/div>\n

\n\t
\n<\/div>\n
\n \n

Please summarize your research. Why are your results notable?<\/strong><\/p>\n

Shuo:<\/strong> The challenge of classifying a hypothesis as entailment, contradiction, or neutral depending on the provided premise is known as Natural Language Inference (NLI). For NLI tasks like semi-structured table reasoning, there are currently two main ways to create training data: via crowdsourcing or through fully automatic techniques. Notably, the former restricts scalability since it is costly and time-consuming, while the latter frequently yields simplistic instances that could lack complex reasoning.<\/p>\n

Our work develops a realistic semi-automated framework for data augmentation for tabular inference. Instead of manually generating a hypothesis for each table, our methodology generates hypothesis templates that are transferable to similar tables (see Table 1 for examples). In addition, our framework entails the creation of rational counterfactual tables based on human-written logical constraints and premise paraphrasing. We employ InfoTabs<\/a>, an entity-centric tabular inference dataset, for our study. We found that our methodology could provide examples of tabular inference that resembled those made by humans. As such, this approach could help with training data augmentation, especially in the case of limited supervision.<\/p>\n\n<\/div>\n

\n \"\"\"\"\n
Table 1.<\/strong> An example of an original and counterfactual table in the “Person” category. Here, we illustrate how multiple operations can be used to alter different keys. We also have shown how the labels (E – Entail, C – Contradict) for a specific hypothesis can be altered. In the \u201cJanet Leigh\u201d sample table, the first column represents the keys (e.g., Born, Died, etc.), while the second column contains the relevant values (e.g., July 6, 1927; October 3, 2004, etc.).
\n<\/span><\/figcaption>\n<\/figure>\n
\n \n

How does your research advance the state-of-the-art in the field of natural language processing?<\/strong><\/p>\n

Existing approaches for curating training data are limited in scale or suffer from biases, especially for reasoning on semi-structured tabular data. Our proposed semi-automatic framework can exploit the tabular structure for hypothesis generation, which can then be further transferred to similar tables. Data augmentation would then be another option for curating training data for table reasoning tasks.<\/p>\n\n<\/div>\n\n<\/div>\n\n\n\t\t\n\t<\/div>\n<\/div>\n

\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t
\n \n

Weakly Supervised Headline Dependency Parsing<\/strong>
\nAdrian Benton (Bloomberg), Tianze Shi (Cornell University\/Bloomberg Data Science Ph.D. Fellow), Ozan \u0130rsoy\/Igor Malioutov (Bloomberg)<\/p>\n

Findings of EMNLP 2022<\/em><\/p>\n\n<\/div>\n

\n\t
\n<\/div>\n
\n \n

Please summarize your research. Why are your results notable?<\/strong><\/p>\n

Igor:<\/strong> The unique syntactic properties of English news headlines have been noted in linguistics literature since the 1930s. However, headlines have received surprisingly little attention from the Natural Language Processing (NLP) community in the context of automatic syntactic analysis or parsing. This presents an important limitation regarding headline processing, since correctly analyzing the syntactic structure of text is often critical for accurate semantic understanding, as well as the effectiveness of downstream systems for summarization, information extraction, and question answering.<\/p>\n

We bridge this gap by providing the first annotated English news headline corpus of Universal Syntactic dependency parse trees, which enables us to evaluate state-of-the-art dependency parsers on news headlines. To improve the accuracy of English news headline parsing, we develop a method to automatically bootstrap noisy training data from pairs of unlabeled headlines and the lede sentences from the body text.<\/p>\n\n<\/div>\n

\n \"\"\"\"\n
Example parses given by EWT (baseline) (Bottom) and Both (finetuned) (Top) on an example headline from the Google Sentence Compression Corpus (GSC). Differing edges are highlighted in green and red for finetuned and baseline, respectively.<\/figcaption>\n<\/figure>\n
\n \n

Our approach is based on a key observation: headlines convey similar semantic content as the news story body \u2013 and they typically share many local substructures. The first sentence in an article, known as a lede sentence, often serves a similar function as the headline. It is meant to grab the reader\u2019s attention and states essential facts about a news event; lede sentences are sometimes direct expansions of the headlines.<\/p>\n

Our bootstrapping algorithm projects and carries over the syntactic dependency analysis from the lede sentence to the headline by pruning the lede sentence’s dependency trees. Because existing models are accurate in parsing long-form news body texts, the resulting silver headline parsed from our bootstrapping algorithm is typically more accurate than applying existing models to headlines directly.<\/p>\n\n<\/div>\n

\n \"\"\"\"\n
An example promotion during our projection algorithm. “release” is promoted to be the new root of the headline.<\/figcaption>\n<\/figure>\n
\n \n

How does your research advance the state-of-the-art in the field of natural language processing?<\/strong><\/p>\n

Models trained on our bootstrapped silver headline parses demonstrate significant improvements in performance over models trained solely on gold-annotated long-form texts — after training with silver parses, we can accurately identify the main predicate\/verb of a sentence up to 98% of the time (up from 75%), and can correctly identify passive construction with up to 91% accuracy (from 11%). Furthermore, we show that these gains translate to downstream improvements in the quality of output extracted by a state-of-the-art open domain information extraction system from headlines.<\/p>\n

We hope our data, models, and methodology will encourage further research to improve dependency parsers for overlooked registers of English. In addition, we hope the development of accurate headline dependency parsers will improve the performance of existing headline understanding and processing tasks and enable more subtle linguistic analysis, such as the identification of “crash blossoms<\/a>” (i.e., ambiguous news headlines).<\/p>\n\n<\/div>\n\n<\/div>\n\n\n\t\t\n\t<\/div>\n<\/div>\n

\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t
\n \n

What Makes Data-to-Text Generation Hard for Pretrained Language Models?<\/strong>
\nMoniba Keymanesh, Adrian Benton, Mark Dredze (Bloomberg)<\/p>\n

GEM Workshop<\/a>, Virtual Poster Session (Wednesday, December 7, 2022 @ 9:00 PM GST)<\/p>\n\n<\/div>\n

\n \"Image\"Image\n
The poster for the paper “What Makes Data-to-Text Generation Hard for Pretrained Language Models?” that Moniba Keymanesh, Adrian Benton & Mark Dredze are presenting during the virtual poster session in the 2nd Generation, Evaluation & Metrics (GEM) Workshop at EMNLP 2022 on Wednesday, December 7, 2022.<\/figcaption>\n<\/figure>\n
\n\t
\n<\/div>\n
\n \n

Please summarize your research. Why are your results notable?<\/strong><\/p>\n

Moniba:<\/strong> In its role as a leading provider of financial information, Bloomberg maintains a massive database of structured information. Access to this information usually comes in the form of API calls. While users can access this information programmatically, delivering it to users often requires formatting the information as natural language. For example, a Bloomberg journalist may fact-check information about a company or look up an executive\u2019s bio using the Bloomberg Knowledge Graph (BBKG) before incorporating it into a story. This process can even be semi-automated or fully automated, but both processes initially require complex engineering efforts. Moreover, question answering from the Terminal commandline seamlessly provides direct access to structured information, but supporting different question and information types requires significant engineering effort.<\/p>\n

The central challenge across these tasks is how to take structured information and express it as fluent and accurate natural language. The goal of this project is to develop a system that takes as input one or more sets of structured facts or relations (triples) and produces natural language text that expresses this information.<\/strong> This task is closely related to data-to-text generation<\/strong> in natural language processing literature. Automated data-to-text generation systems take as input a set of relations, where each relation is a (subject, predicate, object) triple(see an example below). Applications of this technology include story or dialogue generation, open-domain question-answering, and text summarization. Domains may span journalism, weather, finance, sports, and even the summarization of patient medical histories.<\/p>\n\n<\/div>\n

\n \"\"\"\"<\/a>\n
An input\/output example of data-to-text generation.<\/figcaption>\n<\/figure>\n
\n \n

Previous work shows that pre-trained language models (PLMs) perform remarkably well on this task after supervised learning on a significant amount of data. However, dataset creation for this task can be challenging and not feasible in many domains. On the other hand, while auto-regressive PLMs such as GPT can generalize from a few task examples, their data efficacy at data-to-text is largely unexplored. Furthermore, we have an incomplete understanding of the limits of PLMs on this task. These issues make the path forward for data-to-text generation research unclear.<\/p>\n

In this work, we conduct an evaluation of PLMs for data-to-text generation generation, focusing on two classes of challenging examples: examples with novel (unseen) relations (predicates) and examples where the source and target sequences are lexically very different (i.e., not amenable to purely extractive data-to-text generation systems). We consider how GPT-2, adapted with few-shot learning, prompt tuning, and the addition of predicate descriptions, performs on these example classes as compared to a state-of-the-art fine-tuned T5. While GPT-2 performs poorly on DART in the zero-shot setting, we show that its performance can be drastically improved by employing the above techniques.<\/p>\n

How does your research advance the state-of-the-art in the field of natural language processing?<\/strong><\/p>\n

In this work, we contribute to the data-to-text generation research by benchmarking and analyzing the limitations of two popular PLMs on the multi-domain DART dataset. We also provide recommendations for future model and dataset research in data-to-text generation. Essentially, we make the following contributions:<\/p>\n