With the widespread use of Large Language Models (LLMs), there is an increasing demand for personalized models that meet diverse needs of users. To enable private, network-independent personalization of LLMs, on-device fine-tuning is receiving much attention. However, on-device fine-tuning faces efficiency and scalability challenges as sequential execution of compute- and memory-intensive operations often under-utilizes resources. In this letter, we propose ReAx, a framework that accelerates on-device fine-tuning through resource efficient asynchronous parallel execution of memory- and compute-intensive operations. Without increasing memory usage, ReAx improves the average fine-tuning performance and energy consumption by 10.42% and 5.55%, respectively, compared to the baseline. As a positive side effect, asynchronous parameter updates induce gradient noise due to slight delays between streams, which act as a regularizer against adverse updates across mini-batches, without sacrificing accuracy.