In cooperative multi-agent reinforcement learning (MARL), the permutation problem—where the state space grows exponentially with the number of agents—reduces sample efficiency. Additionally, many existing architectures struggle with scalability, as they rely on a fixed structure tied to a specific number of agents, limiting their applicability to environments with variable entity counts. While approaches such as graph neural networks (GNNs) and self-attention mechanisms have made progress in addressing these challenges, they have significant limitations: dense GNNs and self-attention mechanisms incur high computational costs. To overcome these limitations, we propose an architecture composed of a novel agent network and a scalable non-linear mixing network, both of which ensure permutation-freeness and scalability, enabling generalization to environments with variable numbers of agents. Our agent network significantly reduces the computational complexity of attention from $O(n^{2}d)$ to $O(nd)$ , and our scalable hypernetwork enables efficient weight generation for non-linear mixing. Additionally, we introduce curriculum learning to improve training efficiency. Experiments on StarCraft Multi-Agent Challenge v2 (SMACv2) and Google Research Football (GRF) show that our proposed method generally outperforms representative MARL baselines, including HPN, UPDeT, VDN, and QMIX-based variants, particularly on SMACv2 scenarios with varying numbers and types of agents.