This paper presents an approach to protecting deep neural network privacy on edge devices using ARM TrustZone. We propose a selective layer protection technique that balances performance and privacy. Rather than executing entire layers within the TrustZone secure environment, which leads to significant performance and memory overhead, we selectively protect only the most sensitive subset of data from each layer. Our method strategically partitions layer computations between normal and secure worlds, optimizing TrustZone usage while providing robust defenses against privacy attacks. Through extensive experiments on standard datasets (CIFAR-100 and ImageNet-Tiny), we demonstrate that our approach reduces membership inference attack (MIA) success rates from over 90\% to near random guess (50\%) while achieving up to 7.3× speedup and 71\% memory reduction compared to state-of-the-art approaches. On resource-constrained edge devices with limited secure memory, our selective approach enables protection of significantly more layers than full layer protection methods, while maintaining strong privacy guarantees through efficient data partitioning and parallel processing across security boundaries.