The rapid advancement of Large Language Models (LLMs) and Large Vision Models (LVMs) has achieved breakthrough performance but has also created significant deployment challenges due to their massive scale, often exceeding 100 billion parameters. Parameter-Efficient Fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), address this issue by approximating weight updates as products of low-rank matrices, dramatically reducing the number of trainable parameters while maintaining competitive performance. However, LoRA’s effectiveness is fundamentally constrained by the limited effective rank of its update matrices, limiting its ability to capture the rich, high-dimensional changes required for complex adaptation tasks. We propose GroupLoRA, which overcomes this limitation through a divide-and-conquer strategy that partitions weight matrices into g groups, with independent LoRA modules for each group. GroupLoRA introduces two key innovations: (1) an Inter-Group Bridge module that enables information exchange between groups through learnable interaction matrices, and (2) Learnable Scaling Factors that dynamically adjust each group’s contribution to the final output. This design enables group specialization while maintaining cross-group coordination, maximizing model expressiveness within limited parameter budgets. Extensive experiments on LLMs and VLMs demonstrate that GroupLoRA consistently achieves higher performance with fewer parameters than conventional LoRA. Our approach successfully applies group-wise processing principles to parameter-efficient fine-tuning, offering a practical solution for efficient large-model adaptation across diverse tasks.

Multilabel remote sensing image classification is a fundamental task that classifies multiple objects and land covers within an image. However, training deep learning models for this task requires a considerable cost of labeling. While several efforts to reduce labeling costs have been made, they often result in decreased label quality and the inclusion of incorrect (i.e., noisy) labels. To tackle this issue, algorithms for training deep learning models robust to multilabel noise have been proposed in the literature. Nonetheless, the efficacy of these algorithms has been evaluated only under instance-independent multilabel noise, where noise is generated regardless of the individual characteristics and features of each remote sensing image. In this article, we introduce generating instance-dependent multilabel noise into multilabel remote sensing image datasets for the first time. We leverage a vision-language model with zero-shot prediction capabilities to compute categorywise prediction scores for each image, based on which we generate multilabel noise in an instance-dependent manner. We demonstrate that the proposed instance-dependent multilabel noise is more feasibly generated with respect to individual images compared to traditional instance-independent multilabel noise. We also demonstrate that a more challenging noise scenario is generated, which leads to a more complex decision boundary and stronger overfitting during deep learning model training. Finally, we re-evaluate existing noise-robust training algorithms under the generated instance-dependent multilabel noise and observe that several algorithms exhibit limited robustness against instance-dependent multilabel noise.

The rapid advancement of Large Language Models (LLMs) and Large Vision Models (LVMs) has achieved breakthrough performance but has also created significant deployment challenges due to their massive scale, often exceeding 100 billion parameters. Parameter-Efficient Fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), address this issue by approximating weight updates as products of low-rank matrices, dramatically reducing the number of trainable parameters while maintaining competitive performance. However, LoRA’s effectiveness is fundamentally constrained by the limited effective rank of its update matrices, limiting its ability to capture the rich, high-dimensional changes required for complex adaptation tasks. We propose GroupLoRA, which overcomes this limitation through a divide-and-conquer strategy that partitions weight matrices into g groups, with independent LoRA modules for each group. GroupLoRA introduces two key innovations: (1) an Inter-Group Bridge module that enables information exchange between groups through learnable interaction matrices, and (2) Learnable Scaling Factors that dynamically adjust each group’s contribution to the final output. This design enables group specialization while maintaining cross-group coordination, maximizing model expressiveness within limited parameter budgets. Extensive experiments on LLMs and VLMs demonstrate that GroupLoRA consistently achieves higher performance with fewer parameters than conventional LoRA. Our approach successfully applies group-wise processing principles to parameter-efficient fine-tuning, offering a practical solution for efficient large-model adaptation across diverse tasks.

Multilabel remote sensing image classification is a fundamental task that classifies multiple objects and land covers within an image. However, training deep learning models for this task requires a considerable cost of labeling. While several efforts to reduce labeling costs have been made, they often result in decreased label quality and the inclusion of incorrect (i.e., noisy) labels. To tackle this issue, algorithms for training deep learning models robust to multilabel noise have been proposed in the literature. Nonetheless, the efficacy of these algorithms has been evaluated only under instance-independent multilabel noise, where noise is generated regardless of the individual characteristics and features of each remote sensing image. In this article, we introduce generating instance-dependent multilabel noise into multilabel remote sensing image datasets for the first time. We leverage a vision-language model with zero-shot prediction capabilities to compute categorywise prediction scores for each image, based on which we generate multilabel noise in an instance-dependent manner. We demonstrate that the proposed instance-dependent multilabel noise is more feasibly generated with respect to individual images compared to traditional instance-independent multilabel noise. We also demonstrate that a more challenging noise scenario is generated, which leads to a more complex decision boundary and stronger overfitting during deep learning model training. Finally, we re-evaluate existing noise-robust training algorithms under the generated instance-dependent multilabel noise and observe that several algorithms exhibit limited robustness against instance-dependent multilabel noise.

Recent Super-Resolution~(SR) methods mainly adopt Transformers for their strong long-range modeling capability and exceptional representational capacity. However, most SR Transformers rely heavily on relative positional bias~(RPB), which prevents them from leveraging hardware-efficient attention kernels such as FlashAttention. This limitation imposes a prohibitive computational burden during both training and inference, severely restricting attempts to scale SR Transformers by enlarging the training patch size or the self-attention window. Consequently, unlike other domains that actively exploit the inherent scalability of Transformers, SR Transformers remain heavily focused on effectively utilizing limited receptive fields. In this paper, we propose Rank-factorized Implicit Neural Bias~(RIB), an alternative to RPB that enables FlashAttention in SR Transformers. Specifically, RIB approximates positional bias using low-rank implicit neural representations and concatenates them with pixel content tokens in a channel-wise manner, turning the element-wise bias addition in attention score computation into a dot-product operation. Further, we introduce a convolutional local attention and a cyclic window strategy to fully leverage the advantages of long-range interactions enabled by RIB and FlashAttention. We enlarge the window size up to \textbf{96$\times$96} while jointly scaling the training patch size and the dataset size, maximizing the benefits of Transformers in the SR task. As a result, our network achieves \textbf{35.63\,dB PSNR} on Urban100$\times$2, while reducing training and inference time by \textbf{2.1$\times$} and \textbf{2.9$\times$}, respectively, compared to the RPB-based SR Transformer~(PFT).

Recent Super-Resolution~(SR) methods mainly adopt Transformers for their strong long-range modeling capability and exceptional representational capacity. However, most SR Transformers rely heavily on relative positional bias~(RPB), which prevents them from leveraging hardware-efficient attention kernels such as FlashAttention. This limitation imposes a prohibitive computational burden during both training and inference, severely restricting attempts to scale SR Transformers by enlarging the training patch size or the self-attention window. Consequently, unlike other domains that actively exploit the inherent scalability of Transformers, SR Transformers remain heavily focused on effectively utilizing limited receptive fields. In this paper, we propose Rank-factorized Implicit Neural Bias~(RIB), an alternative to RPB that enables FlashAttention in SR Transformers. Specifically, RIB approximates positional bias using low-rank implicit neural representations and concatenates them with pixel content tokens in a channel-wise manner, turning the element-wise bias addition in attention score computation into a dot-product operation. Further, we introduce a convolutional local attention and a cyclic window strategy to fully leverage the advantages of long-range interactions enabled by RIB and FlashAttention. We enlarge the window size up to \textbf{96$\times$96} while jointly scaling the training patch size and the dataset size, maximizing the benefits of Transformers in the SR task. As a result, our network achieves \textbf{35.63\,dB PSNR} on Urban100$\times$2, while reducing training and inference time by \textbf{2.1$\times$} and \textbf{2.9$\times$}, respectively, compared to the RPB-based SR Transformer~(PFT).

Domain generalization (DG) aims to adapt a model using one or multiple source domains to ensure robust performance in unseen target domains. Recently, Parameter-Efficient Fine-Tuning (PEFT) of foundation models has shown promising results in the context of DG problem. Nevertheless, existing PEFT methods still struggle to strike a balance between preserving generalizable components of the pre-trained model and learning task-specific features. To gain insights into the distribution of generalizable components, we begin by analyzing the pre-trained weights through the lens of singular value decomposition. Building on these insights, we introduce Singular Value Decomposed Minor Components Adaptation (SoMA), an approach that selectively tunes minor singular components while keeping the residual parts frozen. SoMA effectively retains the generalization ability of the pre-trained model while efficiently acquiring task-specific skills. Moreover, we freeze domain- generalizable blocks and employ an annealing weight decay strategy, thereby achieving an optimal balance in the delicate trade-off between generalizability and discriminability. SoMA attains state-of-the-art results on multiple benchmarks that span both domain generalized semantic segmentation to domain generalized object detection. In addition, our methods introduce no additional inference overhead or regularization loss, maintain compatibility with any backbone or head, and are designed to be versatile, allowing easy integration into a wide range of tasks.

Zero-Shot Anomaly Detection (ZSAD) aims to identify anomalies in unseen categories or scenarios. Recently, Vision-Language Models (VLMs), most notably CLIP, have been utilized to enhance anomaly detection performance. However, CLIP struggles to capture local anomalies, which has led to the development of additional modules that significantly increase model complexity and computational overhead. To address this challenge, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AULoRA</i>, a novel approach that enhances anomaly understanding by integrating Low Rank Adaptation (LoRA) into CLIP’s visual encoder and efficiently injecting visual context into the textual representation. While preserving CLIP’s general visual knowledge, we utilize Singular Value Decomposition (SVD) to selectively fine-tune only the most relevant singular components, enabling precise identification of semantic anomalies. Nevertheless, anomaly detection often requires capturing highly diverse and category-specific characteristics, which simple text prompts alone struggle to represent adequately. To overcome this, we adapt textual representations based on the visual context extracted from input images, allowing the model to achieve category-aware and anomaly sensitive alignment. AULoRA maintains the original architecture and inference efficiency of CLIP, while achieving state-of-the-art performance on both image-level and pixel-level anomaly detection benchmarks across diverse industrial datasets.

Numerical models have long been used to understand geoscientific phenomena, including tidal currents, crucial for renewable energy production and coastal engineering. However, their computational cost hinders generating data of varying resolutions. As an alternative, deep learning-based downscaling methods have gained traction due to their faster inference speeds. However, most of them are limited to only inference fixed scale and overlook important characteristics of target geoscientific data. In this paper, we propose a novel downscaling framework for tidal current data, addressing its unique characteristics, which are dissimilar to images: heterogeneity and local dependency. Moreover, our framework can generate any arbitrary-scale output utilizing a continuous representation model. Our proposed framework demonstrates significantly improved flow velocity predictions by 93.21% (MSE) and 63.85% (MAE) compared to the LIIF while achieving a remarkable 33.2% reduction in FLOPs. The code will be available on GitHub: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/dslisleedh/LIIFNM</uri>.

Recently, efficient Vision Transformers have shown great performance with low latency on resource-constrained devices. Conventionally, they use <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$4\times 4$</tex> patch embeddings and a 4-stage structure at the macro level, while utilizing sophisticated attention with multi-head configuration at the micro level. This paper aims to address computational redundancy at all design levels in a memory-efficient manner. We discover that using larger-stride patchify stem not only reduces memory access costs but also achieves competitive performance by leveraging token representations with reduced spatial redundancy from the early stages. Furthermore, our preliminary analyses suggest that attention layers in the early stages can be substituted with convolutions, and several attention heads in the latter stages are computationally redundant. To handle this, we introduce a single-head attention module that inherently prevents head redundancy and simultaneously boosts accuracy by parallelly combining global and local information. Building upon our solutions, we introduce SHViT, a Single-Head Vision Transformer that obtains the state-of-the-art speed-accuracy tradeoff. For example, on ImageNet-1k, our SHViT-S4 is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.3\times ,8.l\times$</tex>, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.4\times faster$</tex> than MobileViTv2 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\times l.0$</tex> on GPU, CPU, and iPhone12 mobile device, respectively, while being 1.3% more accurate. For object detection and instance segmentation on MS COCO using Mask-RCNN head, our model achieves performance comparable to FastViT-SA12 while exhibiting <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.8\times and2.0\times lower$</tex> backbone latency on GPU and mobile device, respectively.