Operating the integrated power and gas systems necessitates solving optimal power and gas flow (OPGF) problem, which is notably challenging due to the inherent nonlinearity and complexity of both systems. This study introduces a novel approach to address the OPGF problem by integrating problem decomposition with reinforcement learning (RL)-based cutting plane techniques. The proposed strategy divides the original OPGF problem into a simplified OPGF (SOPGF) sub-problem and a power and gas flow (PGF) calculation sub-problem. In the constraint set of the SOPGF sub-problem, linear inequality constraints (i.e., cutting planes or cuts) are introduced, bounding the feasible region of the SOPGF sub-problem and thereby obtaining the optimal solution to the original OPGF problem. The cuts are determined using an RL algorithm, and the RL agent is trained to maximize the rewards estimated by the PGF calculation. To enhance the efficiency of the agent training, action selection method is utilized in the RL-based cutting plane approach. Case studies are performed to assess the proposed strategy in comparison to conventional methods across diverse test conditions, verifying that the new approach surpasses conventional approaches in both computational efficiency and feasibility.