The threading model of JVM applications has become a common source of instability and inefficiency in containerized environments. In a company like Nubank, one of the largest fintech, with a microservices architecture comprising over 1500 services, manually tuning the ideal number of threads becomes both daunting and risky.

This presentation introduces a novel approach that addresses these issues by implementing a dynamic control loop and fine-grained load-shedding mechanism. The solution continuously adjusts the number of threads during application execution, utilizing real-time signals such as CPU throttling, CPU usage, memory usage, and runtime pressure. Meanwhile, based on a configurable maximum queuing delay, our load-shedding mechanism ensures that the application remains functional under stress by only rejecting work exceeding its capacity.

This comprehensive solution has significantly improved the stability and performance of our applications while reducing associated costs.

Luiz Hespanha
Senior Staff Software Engineer @ Nubank
@luiz_hespanha

Luiz Hespanha is a Systems Performance engineer at Nubank, the most influential Latin America fintech. Working in the area for more than 20 years, he’s been in microservices since before they were called that and has also built useful large-scale systems using RESTful principles and messaging-centric architectures.

Flavio Brasil
Principal Engineer
@fbrasisil

Flavio is a seasoned engineer with a specialization in high-performance solutions for functional programming on the JVM. Throughout his career, he has navigated a wide range of technologies, spanning from intricate JIT compilers to sophisticated libraries for functional programming. Driven by his passion for open source, Flavio has contributed to numerous projects such as Quill, a prominent library for seamless database access in Scala, and Finagle, the robust library at the heart of Twitter's scalable platform.

----
Recorded Sept 21, 2023 at Strange Loop 2023 in St. Louis, MO.
https://thestrangeloop.com