降低延迟的技术
探索并行提示、缓存和流式传输等方法，缩短 LLM 应用的响应时间。
降低延迟的技术 是 CoddyKit 上的免费 Prompt Engineering & LLM Optimization for Developers 课时。 这是第 2 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Prompt Engineering & LLM Optimization for Developers 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Prompt Engineering & LLM Optimization for Developers 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
Understanding LLM Latency
When building applications with Large Language Models (LLMs), one critical factor is latency. Latency refers to the delay between sending a request to the LLM and receiving its response.
High latency can significantly degrade user experience, especially in real-time or interactive applications like chatbots or content generators.
Why Latency Matters
Imagine a user waiting for an AI assistant to reply. A long delay can lead to:
- User frustration and abandonment.
- Application timeouts.
- A perception of a slow, unresponsive system.
Optimizing latency is key to creating smooth, engaging LLM-powered experiences.
Sources of LLM Latency
Latency in LLM applications can stem from several points:
- Network Roundtrip: The time it takes for your request to reach the LLM provider's servers and for the response to return.
- Model Inference: The time the LLM takes to process your input and generate its output.
- Token Generation Speed: LLMs generate responses token by token. The speed at which these tokens are produced impacts the total time.
Parallel Prompting
Parallel prompting is a technique where you send multiple independent LLM requests simultaneously instead of waiting for each one to complete sequentially.
This is highly effective when you have several tasks that don't depend on each other, allowing you to reduce the total wall-clock time for processing a batch of prompts.
Parallel Prompting Demo
This Python example demonstrates how parallel execution can speed up multiple LLM-like calls compared to sequential processing. We use concurrent.futures.ThreadPoolExecutor to simulate this.
import time
from concurrent.futures import ThreadPoolExecutor
# Simulate an LLM API call that takes some time
def call_llm_api(prompt):
time.sleep(1.5) # Simulate API latency
return f"Response for: {prompt[:10]}..."
def main():
prompts = [
"What is the capital of France?",
"Explain quantum physics simply.",
"Write a poem about a cat.",
"Generate a story about AI."
]
print("--- Sequential Calls ---")
start_time_seq = time.time()
for p in prompts:
call_llm_api(p)
end_time_seq = time.time()
print(f"Sequential took: {end_time_seq - start_time_seq:.2f} seconds\n")
print("--- Parallel Calls ---")
start_time_par = time.time()
with ThreadPoolExecutor(max_workers=4) as executor:
_ = list(executor.map(call_llm_api, prompts))
end_time_par = time.time()
print(f"Parallel took: {end_time_par - start_time_par:.2f} seconds")Caching LLM Responses
Caching involves storing the output of an LLM call for a specific input prompt. If the exact same prompt is encountered again, you can return the cached response instantly, completely bypassing the LLM API call.
This technique is excellent for frequently asked, static queries where the response is unlikely to change. It drastically reduces latency and API costs.
Implementing a Cache
You can implement caching using a simple in-memory dictionary or more robust solutions like Redis for distributed caching. The prompt often serves as the cache key, and the LLM's response is the value.
A key consideration is cache invalidation: when should a cached response be considered stale and re-generated?
def main():
cache = {}
def get_llm_response(prompt):
if prompt in cache:
print(f"Cache hit for: '{prompt[:20]}...' - Returning cached.")
return cache[prompt]
else:
print(f"Cache miss for: '{prompt[:20]}...' - Calling LLM...")
# Simulate LLM call (e.g., via API)
response = f"LLM generated: {prompt.upper()}"
cache[prompt] = response
return response
print(get_llm_response("What is AI?"))
print(get_llm_response("What is AI?")) # Cache hit!
print(get_llm_response("Tell me a joke."))
print(get_llm_response("Tell me a joke.")) # Cache hit!Streaming LLM Outputs
Instead of waiting for the LLM to generate its entire response before sending it, streaming delivers the response in small chunks (tokens) as they are generated.
This doesn't reduce the total time taken for the LLM to finish, but it significantly improves perceived latency. Users see text appearing immediately, making the application feel much faster and more interactive, similar to how human conversation flows.
Streaming API Example
Most modern LLM APIs offer a stream=True parameter. This example simulates a streaming client, showing how content can be processed as it arrives.
import time
# Simulate an LLM client that supports streaming
class MockLLMClient:
def chat_completions_create(self, messages, stream=False):
full_response = "The capital of France is Paris. It is known for its Eiffel Tower."
if stream:
print("Streaming response:")
for word in full_response.split():
yield {"choices": [{"delta": {"content": word + " "}}]}
time.sleep(0.1) # Simulate token generation delay
else:
print("Non-streaming response:")
time.sleep(2) # Simulate full response delay
yield {"choices": [{"message": {"content": full_response}}]}
def main():
client = MockLLMClient()
messages = [{"role": "user", "content": "What is the capital of France?"}]
print("--- Non-Streaming Output ---")
for chunk in client.chat_completions_create(messages, stream=False):
print(f"Received full response: {chunk['choices'][0]['message']['content']}")
print("\n--- Streaming Output ---")
stream_content = ""
for chunk in client.chat_completions_create(messages, stream=True):
if "content" in chunk["choices"][0]["delta"]:
token = chunk["choices"][0]["delta"]["content"]
stream_content += token
print(token, end="", flush=True) # Print token as it arrives
print(f"\nFull streamed content: {stream_content}")Check Your Understanding
Which of the following techniques primarily helps reduce perceived latency by delivering LLM outputs incrementally?
Latency Reduction Recap
We've explored crucial strategies to minimize LLM response times for better application performance:
- Parallel Prompting: Execute multiple independent LLM requests concurrently to reduce overall processing time.
- Caching: Store and reuse previous LLM responses for identical queries, eliminating redundant API calls.
- Streaming: Deliver LLM outputs incrementally (token by token) to significantly improve the user's perception of speed and interactivity.
Mastering these techniques is essential for building fast, responsive, and user-friendly LLM-powered applications.
常见问题解答
「降低延迟的技术」课时是免费的吗？
是的 — 「降低延迟的技术」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Prompt Engineering & LLM Optimization for Developers 课程的其余内容，请升级到 CoddyKit PRO。 Prompt Engineering & LLM Optimization for Developers 课程共包含 4 节课。
「降低延迟的技术」这节课中我会学到什么？
探索并行提示、缓存和流式传输等方法，缩短 LLM 应用的响应时间。 你通过在浏览器中直接运行的动手代码来练习 Prompt Engineering & LLM Optimization for Developers，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Prompt Engineering & LLM Optimization for Developers 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Prompt Engineering & LLM Optimization for Developers 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 2 节课，共 4 节。
「降低延迟的技术」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Prompt Engineering & LLM Optimization for Developers 课中编写并运行代码吗？
能。每节 Prompt Engineering & LLM Optimization for Developers 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。