技术

Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices

更新于 2026/7/28 17:28:00

简介

A runtime-aware latency prediction framework for LLM deployment on heterogeneous edge devices, using gated fusion of static descriptors and dynamic hardware telemetry, and separating inference into prefill/decode phases.

别名runtime-aware latency prediction frameworkTransferable Latency Prediction framework

实体关系

REFERENCESTransferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices → SmolLM2可信度 95%

The paper states: 'the same SmolLM2 model family reaches 8.42 tokens/s on Orange Pi 5 Pro but 64.38 tokens/s on an RTX 3090-class GPU.'

COOPERATES_WITHTransferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices → NVIDIA可信度 95%

The framework is validated on Jetson Nano (NVIDIA) and an RTX 3090-class GPU (NVIDIA), indicating integration and testing with NVIDIA hardware platforms.

COOPERATES_WITHTransferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices → Google可信度 90%

The framework is evaluated on Pixel 8 and Pixel 8 Pro devices, which are Google products; the evaluation implies use of and alignment with Google's hardware ecosystem.

COOPERATES_WITHTransferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices → Orange Pi可信度 90%

The framework is validated on Orange Pi 5 Pro, a device manufactured by Orange Pi.

关联事件

来源

暂无来源链接。