Recall ai reduced AWS bill by >$1,000,000 / year

 Recall ai reduced AWS bill by >$1,000,000 / year 


Reducing bots CPU usage by up to 50% doing 👇

Problem

They identified majority of CPU time was spent in two "copy memory" functions


1. __memmove_avx_unaligned_erms 

2. __memcpy_avx_unaligned_erms

Biggest callers of these functions are,

1. Python WebSocket that was receiving the data

2. Chromium's WebSocket that was sending the data

Expensive sockets

- Single 1080p 30fps video stream, in uncompressed I420 format = 93.312 MB/s

- Monitoring showed that at scale, the p99 bot receives 150MB/s of video data

Solution

- Shared Memory was implemented simultaneously accessed by multiple processes at a time 

- Chromium writes to a block of memory, which is read directly by video encoder with no copying

- Ring buffer was chosen as high level transport design

Ring Buffer implementation

Three pointers on Ring buffer

- Write pointer: the next address to write to

- Peek pointer: the address of the next frame to read


- Read pointer: the address where data can be overwritten

Step By Step

1. Frames from the peek pointer are fed into media pipeline to support zero-copy reads

2. Read pointer is advanced when the frame has been fully processed

3. Media pipeline is safely holding a reference to the data inside the ring buffer

Recall ai reduced AWS bill by >$1,000,000 / year 

Reducing bots CPU usage by up to 50% doing 👇




Problem


They identified majority of CPU time was spent in two "copy memory" functions


1. __memmove_avx_unaligned_erms 


2. __memcpy_avx_unaligned_erms


Biggest callers of these functions are,


1. Python WebSocket that was receiving the data


2. Chromium's WebSocket that was sending the data




Expensive sockets


- Single 1080p 30fps video stream, in uncompressed I420 format = 93.312 MB/s


- Monitoring showed that at scale, the p99 bot receives 150MB/s of video data




Solution


- Shared Memory was implemented simultaneously accessed by multiple processes at a time 


- Chromium writes to a block of memory, which is read directly by video encoder with no copying


- Ring buffer was chosen as high level transport design




Ring Buffer implementation


Three pointers on Ring buffer


- Write pointer: the next address to write to


- Peek pointer: the address of the next frame to read


- Read pointer: the address where data can be overwritten



Step By Step


1. Frames from the peek pointer are fed into media pipeline to support zero-copy reads


2. Read pointer is advanced when the frame has been fully processed


3. Media pipeline is safely holding a reference to the data inside the ring buffer

4. That reference is guaranteed to be valid until the data is fully processed and the read pointer is advanced

5. Atomic operations are used to update the pointers in a thread-safe manner

🤔 over to you, how to signal that new data is available or buffer space is free? 4. That reference is guaranteed to be valid until the data is fully processed and the read pointer is advanced

5. Atomic operations are used to update the pointers in a thread-safe manner

🤔 over to you, how to signal that new data is available or buffer space is free?

Comments

Post a Comment

Popular posts from this blog

Performance Optimization in Sitecore

  Performance Optimization in Sitecore Caching Strategies Implementing caching properly in Sitecore significantly improves performance. The following caching techniques should be used:   HTML Caching Ø   Store rendered HTML output to reduce processing time. Ø   Enable caching on Sitecore renderings (Caching tab in Sitecore Experience Editor). Ø   Configure vary by parameters to cache different versions of content. Data Caching Ø   Use Sitecore’s Data Cache to store frequently accessed database queries. Ø   Implement custom memory caches using .NET MemoryCache for high-frequency data. Ø   Leverage pre-fetch caching to load important content in advance. Output Caching Ø   Store entire page outputs in the cache for quick retrieval. Ø   Configure Sitecore Output Cache settings at the item level. Ø   Set cache expiration rules to balance freshness and performance.   Load Balancing & Database Optimiz...
Read more

𝗙𝗹𝘂𝗲𝗻𝘁𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗔𝗦𝗣.𝗡𝗘𝗧 𝗖𝗼𝗿𝗲 - 𝗖𝗹𝗲𝗮𝗻, 𝗙𝗹𝗲𝘅𝗶𝗯𝗹𝗲 𝗠𝗼𝗱𝗲𝗹 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗠𝗼𝗱𝗲𝗿𝗻 .𝗡𝗘𝗧 𝗔𝗽𝗽𝘀

Image
FluentValidation is a popular .NET library that makes it easy to check if your objects have valid data. It’s like a set of rules you create to make sure your objects follow the correct format. The best part? It uses a clean and readable style that even beginners can understand. In this guide, we’ll walk you through how to use  FluentValidation  step-by-step, using a practical  User Registration  example. By the end, you’ll see how it simplifies input validation, keeps your code clean, and improves maintainability in your ASP.NET Core applications. 📘 What is FluentValidation? Think of FluentValidation as a rulebook for your objects. Imagine you're building a  library system . You need to ensure: The library has a name Each book has a valid ID, title, and author The library has a valid address, phone number, and at least five books Instead of writing lots of messy  if  conditions,  FluentValidation  allows you to define all these rules in one ...
Read more

Azure Event Grid Sample code

  Event Grid Azure Event Grid is a fully managed event routing service that enables event-driven architectures by: •                Delivering events from sources to subscribers in near real-time. •                Supporting fan-out (sending the same event to multiple subscribers). •                Ensuring high reliability and scalability. How Event Grid Fits in SOA Decoupling Services: SOA often involves tightly coupled services that rely on synchronous communication (e.g., REST or SOAP APIs). Event Grid enables asynchronous communication by routing events between services, reducing coupling and improving scalability. Event-Driven Communication: Instead of services polling for updates or making direct API calls, they can react to events published to Event ...
Read more