First Word Latency Calculator
The First Word Latency Calculator computes time until the first output word appears, given model throughput, batching, and network latency.
First Word Latency
The First Word Latency Calculator computes time until the first output word appears, given model throughput, batching, and network latency.