For real-time video analytics, the inference trigger frame selection problem is crucial. While continuous video streams offer rich data, processing every frame is computationally intensive. Thus, modern systems strategically analyze the frames to choose frames for inference, i.e., inference trigger frames (ITF). Between ITFs, they use tracking to leverage temporal coherence. Consequently, the ITF selection problem functions as the cornerstone technology for video analytics in terms of both accuracy and efficiency. Additional to selecting the critical frames for system accuracy, a major requirement for the ITF selection algorithms is ‘efficiency.’ ITF selection algorithms should be able to extract the informative frames while imposing minimal computational overhead to the analytics pipeline. While current solutions, such as ‘frame differencing’ methods, provide promising results, they still fall short of fully addressing this issue.We point out that using frame difference value as the metric can mislead the re-detection necessity, thus falling short in choosing the optimal frame sets for inference. We introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FrameBoost</i>, an intelligent ITF selection method that adopts aggregate tracking error (ATE) as its metric, leveraging object-wise IoU predictions. We study the key factors controlling the IoU prediction performance and propose a lightweight solution that effectively addresses the aforementioned limitations. Through extensive evaluations measuring both accuracy and computational efficiency, we demonstrate that FrameBoost delivers superior performance compared to existing approaches.