Increasing the penetration of variable energy resources (VER) can reduce the inertia and frequency response performance of power systems supported by replacement synchronous power generation. Therefore, it is necessary to manage the VER penetration limit in power systems for stable operation and to increase the operability to the desired level. This study proposes a method to evaluate and quantify the effect of increasing the penetration limit of VER by controlling a battery energy storage system (BESS). The BESS can provide a fast response, but frequency response performance varies depending on the operating conditions. In the proposed quantification method, various control methods of a BESS, operating conditions of the power system, and penetration conditions of additional VER were analyzed, and the effect of the BESS on increasing the penetration limit of VER was evaluated. This evaluation and analysis enabled the selection of the BESS operating conditions to achieve the target VER capacity in the power system. The proposed quantification method was analyzed through simulations based on the Korean power system model. Therefore, it can contribute to estimating the required performance of the BESS for each system operating condition required to achieve the VER target.