工作负载波动下ASR服务的时间感知调度
Duration Aware Scheduling for ASR Serving Under Workload Drift
摘要
在大规模自动语音识别系统中，调度策略对端到端延迟的确定起着关键作用。然而，目前常用的服务引擎通常采用先到先得原则进行调度，这种方式无法考虑到请求处理时间的差异，导致在负载变化时会出现排队阻塞问题。我们发现，音频处理时间可以作为Whisper等自动语音识别模型中任务处理时间的准确指标，利用这一特性可以实现基于处理时间的调度方式。我们将最短作业优先算法和最高响应比优先算法两种经典算法整合到vLLM中，并在实际且存在波动的负载条件下对它们进行了测试。在LibriSpeech测试集上，与基准情况相比，最短作业优先算法在高负载情况下可将中位数端到端延迟降低73%，但由于长请求被忽略，第90百分位数延迟则增加了97%。而最高响应比优先算法则能减少中位数端到端延迟约28%，同时将尾部延迟增加幅度控制在最多24%以内。这些改进在负载变化的情况下仍然有效，且不会影响吞吐量，每请求的调度开销也小于0.1毫秒。
English Abstract
Scheduling policies in large-scale Automatic Speech Recognition (ASR) serving pipelines play a key role in determining end-to-end (E2E) latency. Yet, widely used serving engines rely on first-come-first-served (FCFS) scheduling, which ignores variability in request duration and leads to head-of-line blocking under workload drift. We show that audio duration is an accurate proxy for job processing time in ASR models such as Whisper, and use this insight to enable duration-aware scheduling. We integrate two classical algorithms, Shortest Job First (SJF) and Highest Response Ratio Next (HRRN), into vLLM and evaluate them under realistic and drifted workloads. On LibriSpeech test-clean, compared to baseline, SJF reduces median E2E latency by up to 73% at high load, but increases 90th-percentile tail latency by up to 97% due to starvation of long requests. HRRN addresses this trade-off: it reduces median E2E latency by up to 28% while bounding tail-latency degradation to at most 24%. These gains persist under workload drift, with no throughput penalty and <0.1\,ms scheduling overhead per request.