Did you try changing the CPU governor settings? I've found that light workloads don't always push the governor enough to push the CPU into higher frequencies. I have mine set to 100% CPU, always, for lowest latency. See: https://github.com/home-assistant/operating-system/issues/19...