{"id":20723,"date":"2019-08-06T15:01:25","date_gmt":"2019-08-06T19:01:25","guid":{"rendered":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/"},"modified":"2022-02-17T11:29:09","modified_gmt":"2022-02-17T16:29:09","slug":"bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance","status":"publish","type":"post","link":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/","title":{"rendered":"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance"},"content":{"rendered":"
\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t
\n \n

Within the finance industry, detecting rare events — or anomalies — in data is key to solving business problems that can have high monetary costs such as financial crimes detection and risk modeling. On Monday, August 5, 2019, at the 2nd KDD Workshop on Anomaly Detection in Finance<\/a>, which is co-located with the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2019)<\/a> in Anchorage, Alaska this week, Bloomberg researchers showcased some of their research on calibrating anomaly detectors and textual outlier detection in financial reporting.<\/p>\n

Through their work, Bloomberg researchers and their collaborators have highlighted techniques that provide more refined data analysis within the finance industry. Adrian Benton, Senior Research Scientist in Bloomberg\u2019s AI Group, published \u201cCalibration for Anomaly Detection,\u201d while Machine Learning Engineer Leslie Barrett was the lead author on a paper titled \u201cTextual Outlier Detection and Anomalies in Financial Reporting,\u201d which summarizes research by a team of Bloomberg Law engineers. This paper was presented by Sidney Fletcher, one of the authors, during the conference.<\/p>\n

\u201cBloomberg has ongoing investments that foster collaborations between academia and institutions, and this workshop provides an opportunity for participants to continue working together to develop solutions that will address existing business problems within finance,\u201d explained Anju Kambadur, head of Bloomberg\u2019s AI Group, who helped organize the Workshop on Anomaly Detection in Finance<\/a>.<\/p>\n\n<\/div>\n

\n \n

Textual Outlier Detection and Anomalies in Financial Reporting<\/strong><\/h3>\n

It can be time-consuming for a financial analyst following a company to review SEC filings associated with that entity every quarter. However, if the analyst received an alert when something unusual has occurred, they could then conduct a more in-depth review on that document.<\/p>\n

Unusual events described in the text are outliers — or observations that deviate from the normal expected descriptions of corporate risk. Such events may be associated with unexpected management changes, litigation or earnings decline. Outlier detection models are typically unsupervised, calculating distances or densities with respect to certain focal points in the data.<\/p>\n

The high-dimensional nature of text makes distance calculations challenging, and outlier detection in text relies on a specific subset of models suited to this task. Furthermore, the concept of an outlier in text can be defined in different ways, including stylistic or factual outliers, requiring an understanding of a word\u2019s meaning in relation to its surrounding text.<\/p>\n\n<\/div>\n

\n \"Machine\"Machine\n
Machine learning engineer Sidney Fletcher presents the Bloomberg Law team’s research into “Textual Outlier Detection and Anomalies in Financial Reporting.”<\/figcaption>\n<\/figure>\n
\n \n

Similar to credit card fraud, which identifies unusual spending behavior, risk factors that represent outliers tend to contain very specific language with unusual phraseology, noted Barrett. The \u201cnormal\u201d risk factor language tends to be very vague, but specific language and reference tends to indicate high-risk events.<\/p>\n

To perform outlier detection, the text is first converted to a mathematical representation from which word-co-occurrence distances can be calculated. In the general approach, a centroid is calculated and words that are very far from that centroid don\u2019t fit in. Those become the outliers that we flag.<\/p>\n

\u201cIn our research, we found the best model was based on non-negative matrix factorization, which is a fancy way of saying you\u2019re trying to reconstruct a piece of data. Where you can\u2019t do it, the \u201cresiduals\u201d — or parts that don\u2019t fit — tend to be associated with anomalous language,\u201d said Barrett. \u201cWe\u2019re looking at these very closely and pulling them out as outliers \u2013 unusual words or phrases that can\u2019t be factorized. This method turned out to be very robust at finding the instances of true outliers.\u201d<\/p>\n\n<\/div>\n

\n \"A\"A\n
A poster for Textual Outlier Detection and Anomalies in Financial Reporting, which details two steps it took, along with its abstract, what outlier detection is, outliers in its risk factors, takeaways, future work, and references.<\/figcaption>\n<\/figure>\n
\n \n

Calibration for Anomaly Detection<\/strong><\/h3>\n

A well-calibrated machine learning model produces scores that reflect the true probability that certain events will occur. Any domain where a human must make a decision based on model scores can benefit from well-calibrated models, especially when the financial or human cost of making an ill-informed decision is great. Examples could include automated medical diagnosis, detecting credit card fraud, or predicting market volatility.<\/p>\n

A model outputting a score of 0.9 for an input might seem more likely than a score of 0.4, but that 0.9 may not reflect a 90% probability — unless the model is well-calibrated. By calibrating a machine learning model, scores can be reliably interpreted as probabilities.<\/p>\n

\u201cWell-calibrated models help decision-makers take action based on these scores and also allow them to combine scores from many models,\u201d said Benton. \u201cWell-calibrated models can be used in an ensemble to produce a better classifier in a long pipeline.\u201d<\/p>\n

Models are typically trained on data with balanced classes, with all labels roughly being equally likely. However, this research focuses on calibrating models trained on datasets with many more negative examples than positive ones.<\/p>\n\n<\/div>\n

\n \"Senior\"Senior\n
Senior Research Scientist Adrian Benton talks with workshop attendees about his \u201cCalibration for Anomaly Detection” research<\/figcaption>\n<\/figure>\n
\n \n

Models can be calibrated in several ways, including temperature scaling. \u201cWhile this is the simplest technique, it has been shown to be extremely effective across a range of datasets and models,\u201d said Benton. \u201cAll you do is divide model scores by a fixed temperature \u2013 if the temperature is high, it\u2019ll pull the model scores sharply to 50%, while if temperature is small, it\u2019ll only temper model scores slightly.\u201d For example, a model score of 0.1 would be moved to 0.15, and a score of 0.9 would be lowered to 0.85.<\/p>\n

While temperature scaling works very well for standard benchmark datasets, classifiers trained as anomaly detectors require a different approach. Classifiers trained on very imbalanced classes tend to be well-calibrated when predicting negative examples, but are poorly calibrated when predicting anomalous classes.<\/p>\n

The proposed model, called Charcoal Grill Scaling, is an extension of temperature scaling that offers more flexibility to calibrate score distributions generated by anomaly detectors. Rather than having a single temperature to reduce model confidence for all predictions, the temperature depends on the region the score is in. This allows the calibration model to avoid shifting scores where the classifier is well calibrated, and reducing model confidence in regions where the classifier is poorly calibrated.<\/p>\n\n<\/div>\n

\n \"A\"A\n
A poster of Calibration for Anomaly Detection, by Adrian Benton at Bloomberg LP. The poster looks at model calibration, anomaly detection in finance, the difficulty with calibrating anomaly detectors, using charcoal grill scaling as a model, along with experiments and takeaways.<\/figcaption>\n<\/figure>\n<\/p>\n\n<\/div>\n\n\n\t\t\n\t<\/div>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"

A deep dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting<\/p>\n","protected":false},"author":184,"featured_media":19072,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1466],"tags":[1508,1630,1418,1633,1472,1635,1636,1634,1477,1570],"class_list":["post-20723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech-at-bloomberg","tag-kdd2019","tag-anomaly-detection","tag-data","tag-data-mining","tag-data-science","tag-financial-reporting","tag-knowledge-discovery","tag-outlier-detection","tag-research","tag-research-and-development"],"yoast_head":"\nBloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance | Bloomberg LP<\/title>\n<meta name=\"description\" content=\"Dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting that Bloomberg engineers published at KDD 2019.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance | Bloomberg LP\" \/>\n<meta property=\"og:description\" content=\"Dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting that Bloomberg engineers published at KDD 2019.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/\" \/>\n<meta property=\"og:site_name\" content=\"Bloomberg L.P.\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bloomberglp\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-08-06T19:01:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-02-17T16:29:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"4032\" \/>\n\t<meta property=\"og:image:height\" content=\"3024\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"chaas30\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg\" \/>\n<meta name=\"twitter:creator\" content=\"@bloomberg\" \/>\n<meta name=\"twitter:site\" content=\"@bloomberg\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"chaas30\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/\",\"url\":\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/\",\"name\":\"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance | Bloomberg LP\",\"isPartOf\":{\"@id\":\"https:\/\/www.bloomberg.com\/company\/#website\"},\"datePublished\":\"2019-08-06T19:01:25+00:00\",\"dateModified\":\"2022-02-17T16:29:09+00:00\",\"author\":{\"@id\":\"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/4d4a18aae79d6fcc1ea98181a906905e\"},\"description\":\"Dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting that Bloomberg engineers published at KDD 2019.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":\"1\",\"name\":\"Home\",\"item\":\"https:\/\/www.bloomberg.com\/company\/\"},{\"@type\":\"ListItem\",\"position\":\"2\",\"name\":\"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/#website\",\"url\":\"https:\/\/www.bloomberg.com\/company\/\",\"name\":\"Bloomberg L.P.\",\"description\":\"Bloomberg L.P. is the leader in global business and financial information, enabling customers to make smarter, faster, more informed business decisions.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bloomberg.com\/company\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/4d4a18aae79d6fcc1ea98181a906905e\",\"name\":\"Bloomberg L.P.\",\"url\":\"https:\/\/www.bloomberg.com\/company\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance | Bloomberg LP","description":"Dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting that Bloomberg engineers published at KDD 2019.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/","og_locale":"en_US","og_type":"article","og_title":"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance | Bloomberg LP","og_description":"Dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting that Bloomberg engineers published at KDD 2019.","og_url":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/","og_site_name":"Bloomberg L.P.","article_publisher":"https:\/\/www.facebook.com\/bloomberglp\/","article_published_time":"2019-08-06T19:01:25+00:00","article_modified_time":"2022-02-17T16:29:09+00:00","og_image":[{"width":4032,"height":3024,"url":"https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg","type":"image\/jpeg"}],"author":"chaas30","twitter_card":"summary_large_image","twitter_image":"https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg","twitter_creator":"@bloomberg","twitter_site":"@bloomberg","twitter_misc":{"Written by":"chaas30","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/","url":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/","name":"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance | Bloomberg LP","isPartOf":{"@id":"https:\/\/www.bloomberg.com\/company\/#website"},"datePublished":"2019-08-06T19:01:25+00:00","dateModified":"2022-02-17T16:29:09+00:00","author":{"@id":"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/4d4a18aae79d6fcc1ea98181a906905e"},"description":"Dive into research on calibrating anomaly detectors and textual outlier detection in financial reporting that Bloomberg engineers published at KDD 2019.","breadcrumb":{"@id":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bloomberg.com\/company\/stories\/bloomberg-researchers-present-2nd-kdd-workshop-anomaly-detection-finance\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":"1","name":"Home","item":"https:\/\/www.bloomberg.com\/company\/"},{"@type":"ListItem","position":"2","name":"Bloomberg Researchers Present at the 2nd KDD Workshop on Anomaly Detection in Finance"}]},{"@type":"WebSite","@id":"https:\/\/www.bloomberg.com\/company\/#website","url":"https:\/\/www.bloomberg.com\/company\/","name":"Bloomberg L.P.","description":"Bloomberg L.P. is the leader in global business and financial information, enabling customers to make smarter, faster, more informed business decisions.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bloomberg.com\/company\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/4d4a18aae79d6fcc1ea98181a906905e","name":"Bloomberg L.P.","url":"https:\/\/www.bloomberg.com\/company"}]}},"featured_image_rendered":"<img srcset='https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg 4032w, https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg 300w, https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg 768w, https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg 1024w, https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg 170w, https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg 140w' src='https:\/\/assets.bbhub.io\/image\/v1\/resize?width=auto&type=webp&url=https:\/\/assets.bbhub.io\/company\/sites\/51\/2019\/08\/20190805_111722.jpg' alt='Machine learning engineer Sidney Fletcher presents the Bloomberg Law team's research into \"Textual Outlier Detection and Anomalies in Financial Reporting.\"' \/>","category_info":{"name":"Tech At Bloomberg","blog_landing_name":"Tech At Bloomberg"},"_links":{"self":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts\/20723","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/users\/184"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/comments?post=20723"}],"version-history":[{"count":1,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts\/20723\/revisions"}],"predecessor-version":[{"id":20983,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts\/20723\/revisions\/20983"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/media\/19072"}],"wp:attachment":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/media?parent=20723"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/categories?post=20723"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/tags?post=20723"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}