With the advent of the Internet of Things (IoT), large volumes of sensitive data are produced from IoT devices, driving the adoption of Machine Learning as a Service (MLaaS) to overcome their limited computational resources. However, as privacy concerns in MLaaS grow, the demand for Privacy-Preserving Machine Learning (PPML) has increased. Fully Homomorphic Encryption (FHE) offers a promising solution by enabling computations on encrypted data without exposing the raw data. However, FHE-based neural network inference suffers from substantial overhead due to expensive primitive operations, such as ciphertext rotation and bootstrapping. While previous research has primarily focused on optimizing the efficiency of these computations, our work takes a different approach by concentrating on the rotation keyset design, a pre-generated data structure prepared before execution. We systematically explore three key design spaces (KDS) that influence rotation keyset design and propose an optimized keyset that reduces both computational overhead and memory consumption. To demonstrate the effectiveness of our new KDS design, we present two case studies that achieve up to 11.29× memory reduction and 1.67-2.55× speedup, highlighting the benefits of our optimized keyset.