# Challenges with Real-time Inference at Scale

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fchallenges-with-real-time-inference-at-scale%2F140367%3E%5C
**Category:** Beginners\
**Created:** [February 12, 2025, 6:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fchallenges-with-real-time-inference-at-scale%2F140367 "2025-02-12T06:34:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JamesLee2295](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@JamesLee2295](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJamesLee2295)%5C
**Post date:** [February 12, 2025, 6:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fchallenges-with-real-time-inference-at-scale%2F140367%2F1 "2025-02-12T06:34:39Z")

</div>

Hello! We’re implementing an AI chatbot that supports real-time customer interactions, but the inference time of our LLM becomes a bottleneck under heavy user traffic. Even with GPU-backed infrastructure, the scaling costs are climbing quickly. Has anyone optimized LLMs for high-throughput applications or found any company provides platforms/services that handle this efficiently? Would love to hear about approaches to reduce latency without sacrificing quality.
