How our engineers keep Terminal data running smoothly around the world

May 21, 2018

The Real-Time Market Data and News Feeds group plays a critical role in keeping data flowing to customers and shaping world markets. Meet two engineers in our APAC offices on the front lines of powering the Terminal with a global team.

System Reliability Engineer (SRE) and Team Lead, Kai To, and Senior Software Engineer and APAC Manager, Carla Nunez.

Real-time information is the lifeblood of financial markets. Engineers in our Real-Time Market Data and News Feeds group play a critical role at Bloomberg. All around our global offices, Software Engineers and System Reliability Engineers on this team work hand-in-hand to make sure data streams are reliable and operational 24 hours a day, seven days a week — to enable the Bloomberg Terminal, Enterprise and Trading Solutions products to continue to power global markets.

We took a moment to get to know two members of the APAC Engineering team and the unique, global challenges they solve every day.

Meet System Reliability Engineer (SRE) and Team Lead, Kai To, and Senior Software Engineer and APAC Manager, Carla Nunez. Each day, their teams work together to tackle a range of challenges as members of the Real-Time Market Data and News Feeds Engineering team in Tokyo and Hong Kong, respectively.

How would you describe what the Real-Time Market Data and News Feeds group works on?

Carla: At a high level, the Feeds group creates and maintains the software that connects to exchanges and news providers around the globe to receive real-time data. The software we develop then takes that information and normalizes it to fit the Bloomberg data model. In essence, we turn it into a stream of information that fuels our back-end databases, which powers the thousands of Bloomberg functions our clients rely on for market data each day.

Kai: It sounds fairly simple, right? But in practice it can be very challenging. We’re responsible for ensuring our clients receive correct, fast and reliable data from hundreds of data sources around the world. This doesn’t mean we sit there eyeballing the data, but instead write software to be able to do this for us automatically. Data traffic is always increasing, so we have lots of interesting challenges to solve. For example, we call each data point that we process and disseminate a “tick” and right now we’re sending over 100 billion ticks a day and this number can only grow.

What are some of those challenges you work together to solve?

Kai: Software standardization is a challenge Carla and I frequently work together on. As an SRE, our main focus is around the production environment and ensuring stability, efficiency, monitoring and many other aspects. Because of this, we work very closely with Carla’s team to make certain the software being created fits into our requirements for production. A specific example would be data recovery. If a circuit to an exchange has problems and the software detects gaps in the data received, then we expect the software would recover this data based on the recovery features available from the exchange.

Carla: Exchange mechanisms for data recovery varies; some exchanges providing multiple tiers of recovery options while others may have no recovery feature at all. Even among exchanges that provide similar recovery features, there can be nuances in how the mechanism is implemented or how the exchange actually behaves.

Kai: We work with Carla’s team on the guidelines for software recovery so that despite the differences in exchange recovery features, the end result of a recovery step is consistent across all of our software. Of course, there are times we disagree and dispute the changes that Carla’s team wants to do.

Carla: When our teams disagree on the changes, it’s actually a crucial part of the collaborative process; each member contributes valuable perspectives and we work together towards the best solution. SREs like Kai play a special role in spotting potential issues before they occur, and quickly rectify any problems before they can impact clients. The input from SREs helps our software engineers take into account operational challenges when designing software and considering the worst case scenarios for production operation.

Our challenge with standardization also extends to how we normalize the data from exchanges to fit the Bloomberg data model. The data we receive can be binary, ASCII or a mix of both. Messages may be compressed, have fixed or variable length, or the content may differ in meaning depending on the context. Even if two data providers follow the same protocol, there is no guarantee that data will behave exactly the same way. Conventions across exchanges and countries can differ. This therefore creates many challenges and unknowns.

Data providers may also limit the technologies we can use. Our software is normally written in C++ and run on Linux machines but if an exchange requires the use of middleware that only runs on a different operating system, then we have no choice but use an alternative approach to getting the data. We work with Kai’s team to ensure that this can still be supported by SREs.

Kai: Now take the challenge of standardization and multiply that by the 530 stock exchanges that we have on Bloomberg, and you can imagine just how difficult it is. At the end of the day — to achieve standardization, Feeds engineers like Carla and SREs like me, work closely together to define the standards for behavior of the feeds. Because SRE teams have to troubleshoot production issues in real-time, it is important that much of the differences in implementation are abstracted away so they can quickly remedy issues without having to know specific details about each individual exchange’s protocols.

Why is collaboration across offices especially important for your roles?

Kai: I was previously in the London office and having worked in different Bloomberg offices, I can really feel we’re all moving in the same direction globally. We have thousands of processes running across hundreds of servers, and our end goal is to make sure a big exchange like the NYSE is treated and appears exactly the same as a smaller exchange like the Laos Stock Exchange. To make this happen, you have to approach software from a global perspective. Our work also impacts the financial markets, which spans the world and partnering to get things done — no matter where you work and what team you’re on — is extremely important. We work closely with the New York and London teams to ensure this global operation runs smoothly 24/7.

Carla: Our work is already complex, and when it also impacts global markets, the need to get things done right is heightened even more. We work with other engineers in our market data pipeline to ensure that our software is standardized, resilient and stable. We also need to make certain that our data models are consistent across markets and we understand the variations in the use of data – a Terminal user has different requirements from an Enterprise data consumer and clients in different regions have different expectations. Navigating the needs of the global markets requires us to closely partner with product and business teams in Bloomberg offices across the world.

What is it like to work in Bloomberg’s Asia offices?

Kai: Tokyo is a smaller office compared to London or New York, but the Bloomberg culture is strong no matter what the office location is. There are many opportunities to work with other departments and to influence how the office is run. For example, I’ve been on the Tokyo Office Committee with the goal to bring Tokyo employees together through events and social activities throughout the year.

Carla: I have worked in the New York, Tokyo and Hong Kong offices. The company culture – innovating, collaborating, knowing the customer and doing the right thing – is the same across all Bloomberg offices. Our Hong Kong office is the largest in the APAC region but we are still growing our Engineering presence here so there are many exciting opportunities to contribute in building our technology community in this part of the region.

What’s coming up/what are you working on now?

Kai: Right now we’re working on some exciting projects that involve a lot of collaboration with our sister teams in London and New York. Software deployment is extremely important for us, and we’re working on a new system which makes this easier for our software engineers to push changes to production quickly and safely.

Carla: We are very much focused on stability efforts, where standardization of our software is just one of the key areas. At the same time, we are working with other Engineering groups on system design changes that will improve our market data pipeline – there are many critical and interesting problems that need to be solved!

What do you like most about working at Bloomberg?

Kai: I love solving complex problems and I feel lucky to be surrounded by talented and driven individuals who are always open to working together to create meaningful solutions for our clients. It’s truly a global effort—it’s an amazing feeling to be part of a team that spans the entire world and working on something with an everyday global impact.

Carla: I have been at Bloomberg and the Feeds team for a decade now and I continue to find our work challenging and enjoyable, as it evolves with the latest technology and market needs. At Bloomberg there’s a lot of trust and freedom, no matter what country you’re in. Employees drive day-to-day decision-making and I always feel empowered by my managers to do the right thing.

To learn more about joining Bloomberg Engineering’s Real-Time Market Data and News Feeds team in Asia, check out current openings here.