{"id":20613,"date":"2017-12-07T13:30:04","date_gmt":"2017-12-07T18:30:04","guid":{"rendered":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/"},"modified":"2022-02-17T11:53:40","modified_gmt":"2022-02-17T16:53:40","slug":"powerfulseal-testing-tool-kubernetes-clusters","status":"publish","type":"post","link":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/","title":{"rendered":"PowerfulSeal: A testing tool for Kubernetes clusters"},"content":{"rendered":"
\n \n\t\n\t\n\t
\n\t\t\n\t\t\t
\n\t

\n \"\"\"\"\n \n<\/figure>\n
\n \n

Bloomberg has adopted Kubernetes<\/a>, the open source system for deploying and managing containerized applications which has gained a great deal of industry momentum<\/a>, in its infrastructure. As a result, systems are becoming more distributed than ever before, running on machines scattered around the globe and across the cloud. This means there are more moving parts, any of which could fail for a long list of reasons.<\/p>\n

Systems engineers want to feel confident that the complex systems they\u2019ve built will withstand problems and keep running. To do that, they run batteries of elaborate tests designed to simulate all sorts of problems. But it\u2019s impossible to imagine every potential problem, let alone plan for all of them.<\/p>\n

\u201cWhen you have something so complex that it\u2019s difficult to predict things, the only reasonable way to deal with that is to simulate the kinds of failure you\u2019re expecting to see before they happen to you in production,\u201d says Mikolaj Pawlikowski, a London-based software engineer with Bloomberg\u2019s Data Technologies team .<\/p>\n

Because problems in the real world don\u2019t occur on a schedule, the best way to test is to cause problems at random. Netflix, the streaming video provider, developed a tool it calls Chaos Monkey<\/a>,<\/u> which randomly terminates virtual machines running on Amazon Web Services (AWS). \u201cIt would take down nodes so that engineers could gain confidence the application would keep running anyway,\u201d Pawlikowski said.<\/p>\n

Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented<\/a>\u00a0at KubeCon + CloudNativeCon North America 2017<\/a> in Austin, TX.<\/p>\n

The tool is aimed specifically at Kubernetes, and includes the ability to describe the objects running in each container so that it knows precisely which things it needs to break for testing purposes.<\/p>\n\n<\/div>\n

\n \"High-level\"High-level\n
High-level schema of required setup for PowerfulSeal<\/figcaption>\n<\/figure>\n
\n \n

It also has an interactive mode that allows systems engineers to experiment and see how it behaves on their clusters and, over time, build their own testing policies.<\/p>\n

\u201cYou can just point it at a cluster and ask it to delete things, take things up and down, and execute arbitrary commands,\u201d Pawlikowski says. \u201cIt lets you get a good idea of how resilient the application is, and then, with that experience, you can write policies in YAML<\/a> and deploy them.\u201d<\/p>\n

Those policies can be fine-tuned lots of ways, including creating rules for the time of day, probability, how much of the application to break, or where to break it. Once deployed, the tool runs in autonomous mode. And like Chaos Monkey before it, PowerfulSeal<\/a> is being released as an open source tool via Bloomberg\u2019s GitHub repository<\/a>.<\/p>\n\n<\/div>\n

\n \"An\"An\n
An example YAML policy file, driving PowerfulSeal’s behavior in Autonomous mode<\/figcaption>\n<\/figure>\n
\n \n

PowerfulSeal currently includes drivers for Kubernetes clusters running on OpenStack<\/a>, but Pawlikowski says he\u2019s hopeful the open source community will create additional drivers to use it with other cloud platforms, including Amazon Web Services EC2, Microsoft Azure, and Google Compute Engine. He\u2019s also hoping the community will create some \u201cpretty crazy filters\u201d that will enable testing for additional kinds of scenarios. \u201cI hope the community will make this PowerfulSeal even more powerful,\u201d he added.<\/p>\n\n<\/div>\n\n<\/div>\n\n\n\t\t\n\t<\/div>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"

Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.<\/p>\n","protected":false},"author":313,"featured_media":19558,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1466],"tags":[1656,1657,1660,1654,1597,1659,1543,1596,1537,1655,1464,1480,1595,1658],"class_list":["post-20613","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech-at-bloomberg","tag-amazon-web-services","tag-aws","tag-azure","tag-chaos-monkey","tag-cloudnative","tag-ec2","tag-github","tag-kubecon","tag-kubernetes","tag-netflix","tag-open-source","tag-openstack","tag-powerfulseal","tag-yaml"],"yoast_head":"\nPowerfulSeal: A testing tool for Kubernetes clusters | Bloomberg LP<\/title>\n<meta name=\"description\" content=\"Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PowerfulSeal: A testing tool for Kubernetes clusters | Bloomberg LP\" \/>\n<meta property=\"og:description\" content=\"Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/\" \/>\n<meta property=\"og:site_name\" content=\"Bloomberg L.P.\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/bloomberglp\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-12-07T18:30:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-02-17T16:53:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/assets.bbhub.io\/company\/sites\/51\/2017\/12\/nodes-video.gif\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"363\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/gif\" \/>\n<meta name=\"author\" content=\"akelber5\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/assets.bbhub.io\/company\/sites\/51\/2017\/12\/nodes-video.gif\" \/>\n<meta name=\"twitter:creator\" content=\"@bloomberg\" \/>\n<meta name=\"twitter:site\" content=\"@bloomberg\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"akelber5\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/\",\"url\":\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/\",\"name\":\"PowerfulSeal: A testing tool for Kubernetes clusters | Bloomberg LP\",\"isPartOf\":{\"@id\":\"https:\/\/www.bloomberg.com\/company\/#website\"},\"datePublished\":\"2017-12-07T18:30:04+00:00\",\"dateModified\":\"2022-02-17T16:53:40+00:00\",\"author\":{\"@id\":\"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/09c9e1a38b7f345ce5c0b4bbde1656a6\"},\"description\":\"Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":\"1\",\"name\":\"Home\",\"item\":\"https:\/\/www.bloomberg.com\/company\/\"},{\"@type\":\"ListItem\",\"position\":\"2\",\"name\":\"PowerfulSeal: A testing tool for Kubernetes clusters\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/#website\",\"url\":\"https:\/\/www.bloomberg.com\/company\/\",\"name\":\"Bloomberg L.P.\",\"description\":\"Bloomberg L.P. is the leader in global business and financial information, enabling customers to make smarter, faster, more informed business decisions.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.bloomberg.com\/company\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/09c9e1a38b7f345ce5c0b4bbde1656a6\",\"name\":\"Bloomberg L.P.\",\"url\":\"https:\/\/www.bloomberg.com\/company\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"PowerfulSeal: A testing tool for Kubernetes clusters | Bloomberg LP","description":"Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/","og_locale":"en_US","og_type":"article","og_title":"PowerfulSeal: A testing tool for Kubernetes clusters | Bloomberg LP","og_description":"Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.","og_url":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/","og_site_name":"Bloomberg L.P.","article_publisher":"https:\/\/www.facebook.com\/bloomberglp\/","article_published_time":"2017-12-07T18:30:04+00:00","article_modified_time":"2022-02-17T16:53:40+00:00","og_image":[{"width":1024,"height":363,"url":"https:\/\/assets.bbhub.io\/company\/sites\/51\/2017\/12\/nodes-video.gif","type":"image\/gif"}],"author":"akelber5","twitter_card":"summary_large_image","twitter_image":"https:\/\/assets.bbhub.io\/company\/sites\/51\/2017\/12\/nodes-video.gif","twitter_creator":"@bloomberg","twitter_site":"@bloomberg","twitter_misc":{"Written by":"akelber5","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/","url":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/","name":"PowerfulSeal: A testing tool for Kubernetes clusters | Bloomberg LP","isPartOf":{"@id":"https:\/\/www.bloomberg.com\/company\/#website"},"datePublished":"2017-12-07T18:30:04+00:00","dateModified":"2022-02-17T16:53:40+00:00","author":{"@id":"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/09c9e1a38b7f345ce5c0b4bbde1656a6"},"description":"Inspired in part by Chaos Monkey, Bloomberg has built its own tool for testing Kubernetes clusters called PowerfulSeal, which Pawlikowski presented\u00a0at KubeCon + CloudNativeCon North America 2017 in Austin, TX.","breadcrumb":{"@id":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.bloomberg.com\/company\/stories\/powerfulseal-testing-tool-kubernetes-clusters\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":"1","name":"Home","item":"https:\/\/www.bloomberg.com\/company\/"},{"@type":"ListItem","position":"2","name":"PowerfulSeal: A testing tool for Kubernetes clusters"}]},{"@type":"WebSite","@id":"https:\/\/www.bloomberg.com\/company\/#website","url":"https:\/\/www.bloomberg.com\/company\/","name":"Bloomberg L.P.","description":"Bloomberg L.P. is the leader in global business and financial information, enabling customers to make smarter, faster, more informed business decisions.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.bloomberg.com\/company\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.bloomberg.com\/company\/#\/schema\/person\/09c9e1a38b7f345ce5c0b4bbde1656a6","name":"Bloomberg L.P.","url":"https:\/\/www.bloomberg.com\/company"}]}},"featured_image_rendered":"<img srcset='' src='https:\/\/assets.bbhub.io\/company\/sites\/51\/2017\/12\/nodes-video.gif' alt='' \/>","category_info":{"name":"Tech At Bloomberg","blog_landing_name":"Tech At Bloomberg"},"_links":{"self":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts\/20613","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/users\/313"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/comments?post=20613"}],"version-history":[{"count":2,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts\/20613\/revisions"}],"predecessor-version":[{"id":20997,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/posts\/20613\/revisions\/20997"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/media\/19558"}],"wp:attachment":[{"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/media?parent=20613"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/categories?post=20613"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bloomberg.com\/company\/wp-json\/wp\/v2\/tags?post=20613"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}