Enhancing Trajectory Estimation with Convolutional IMU Transformer
Sunghyun Kim, Sooyoung Noh, Seungik Jeong, Changhyun Kim, Haeyun Lee, Jihun Kim
IEEE Sensors Journal
Accurate trajectory estimation using low-cost inertial measurement unit (IMU) data remains a challenging task due to sensor noise, bias, and drift. In this paper, we propose the Convolutional IMU Transformer (CoIT), a deep neural network that combines Convolutional Token Embedding with a hierarchical Transformer architecture to effectively capture both local motion features and global temporal dependencies from raw IMU measurements. The proposed network is designed to overcome the limitations of traditional dead reckoning methods and prior learning-based approaches that rely heavily on inaccurate orientation estimates. CoIT further incorporates a multi-stage Convolutional Projection and downsampling strategy to improve computational efficiency while preserving representational capacity. We evaluate CoIT on three large-scale public datasets (RIDI, RoNIN, and NeurIT) and demonstrate consistent performance gains over existing state-of-the-art models. On the RoNIN dataset, CoIT achieves an absolute trajectory error (ATE) of 3.39 ± 0.59 m, a time-normalized relative trajectory error (T-RTE) of 2.63 ± 0.57 m, and a distance-normalized relative trajectory error (D-RTE) of 0.17 ± 0.02 m, while also demonstrating robust performance on unseen subjects and competitive cross-domain transfer from pedestrian motion (RoNIN) to indoor robot motion (NeurIT) without fine-tuning. Furthermore, an ablation study on model complexity reveals a clear trade-off between accuracy and latency, demonstrating the architecture’s scalability for practical deployment. These results highlight the robustness and statistical reliability of the proposed model within the evaluated public benchmarks in GPS-denied settings.
Deep Learning-based Synthetic High-Resolution In-Depth Imaging Using an Attachable Dual-element Endoscopic Ultrasound Probe
Hah Min Lew, Jae Seong Kim, Moon Hwan Lee, Jaegeun Park, Sangyeon Youn, Hee Man Kim, Jihun Kim, Jae Youn Hwang
arXiv (Cornell University)
Endoscopic ultrasound (EUS) imaging has a trade-off between resolution and penetration depth. By considering the in-vivo characteristics of human organs, it is necessary to provide clinicians with appropriate hardware specifications for precise diagnosis. Recently, super-resolution (SR) ultrasound imaging studies, including the SR task in deep learning fields, have been reported for enhancing ultrasound images. However, most of those studies did not consider ultrasound imaging natures, but rather they were conventional SR techniques based on downsampling of ultrasound images. In this study, we propose a novel deep learning-based high-resolution in-depth imaging probe capable of offering low- and high-frequency ultrasound image pairs. We developed an attachable dual-element EUS probe with customized low- and high-frequency ultrasound transducers under small hardware constraints. We also designed a special geared structure to enable the same image plane. The proposed system was evaluated with a wire phantom and a tissue-mimicking phantom. After the evaluation, 442 ultrasound image pairs from the tissue-mimicking phantom were acquired. We then applied several deep learning models to obtain synthetic high-resolution in-depth images, thus demonstrating the feasibility of our approach for clinical unmet needs. Furthermore, we quantitatively and qualitatively analyzed the results to find a suitable deep-learning model for our task. The obtained results demonstrate that our proposed dual-element EUS probe with an image-to-image translation network has the potential to provide synthetic high-frequency ultrasound images deep inside tissues.
High-Level Synthesis Design of Scalable Ultrafast Ultrasound Beamformer With Single FPGA
Zhengchang Kou, Qi You, Jihun Kim, Zhijie Dong, Matthew R. Lowerison, Nathiya Vaithiyalingam Chandra Sekaran, Daniel A. Llano, Pengfei Song, Michael L. Oelze
IF 3.8 (2023)
IEEE Transactions on Biomedical Circuits and Systems
Ultrafast ultrasound imaging is essential for advanced ultrasound imaging techniques such as ultrasound localization microscopy (ULM) and functional ultrasound (fUS). Current ultrafast ultrasound imaging is challenged by the ultrahigh data bandwidth associated with the radio frequency (RF) signal, and by the latency of the computationally expensive beamforming process. As such, continuous ultrafast data acquisition and beamforming remain elusive with existing software beamformers based on CPUs or GPUs. To address these challenges, the proposed work introduces a novel method of implementing an ultrafast ultrasound beamformer specifically for ultrafast plane wave imaging (PWI) on a field programmable gate array (FPGA) by using high-level synthesis. A parallelized implementation of the beamformer on a single FPGA was proposed by 1) utilizing a delay compression technique to reduce the delay profile size, which enables both run-time pre-calculated delay profile loading from external memory and delay reuse, 2) vectorizing channel data fetching which is enabled by delay reuse, and 3) using fixed summing networks to reduce consumption of logic resources. Our proposed method presents two unique advantages over current FPGA beamformers: 1) high scalability that allows fast adaptation to different FPGA resources and beamforming speed demands by using Xilinx High-Level Synthesis as the development tool, and 2) allow a compact form factor design by using a single FPGA to complete the beamforming instead of multiple FPGAs. Current Xilinx FPGAs provide the capabilities of connecting up to 1024 ultrasound channels with a single FPGA and the newest JESD204B interface analog front end (AFE). This channel count is much more than the channel count needed by current linear arrays, which normally have 128 or 256 channels. With the proposed method, a sustainable average beamforming rate of 4.83 G samples/second in terms of input raw RF sample was achieved. The resulting image quality of the proposed beamformer was compared with the software beamformer on the Verasonics Vantage system for both phantom imaging and in vivo imaging of a mouse brain. Multiple imaging schemes including B-mode, power Doppler and ULM were assessed to verify that the image quality was not compromised for speed.
High-level synthesis design of scalable ultrafast ultrasound beamformer with single FPGA
Zhengchang Kou, Qi You, Jihun Kim, Zhijie Dong, Matthew R. Lowerison, Nathiya Vaithiyalingam Chandra Sekaran, Daniel A. Llano, Pengfei Song, Michael L. Oelze
arXiv (Cornell University)
Ultrafast ultrasound imaging is essential for advanced ultrasound imaging techniques such as ultrasound localization microscopy (ULM) and functional ultrasound (fUS). Current ultrafast ultrasound imaging is challenged by the ultrahigh data bandwidth associated with the radio frequency (RF) signal, and by the latency of the computationally expensive beamforming process. As such, continuous ultrafast data acquisition and beamforming remain elusive with existing software beamformers based on CPUs or GPUs. To address these challenges, the proposed work introduces a novel method of implementing an ultrafast ultrasound beamformer specifically for ultrafast plane wave imaging (PWI) on a field programmable gate array (FPGA) by using high-level synthesis. A parallelized implementation of the beamformer on a single FPGA was proposed by 1) utilizing a delay compression technique to reduce the delay profile size, which enables both run-time pre-calculated delay profile loading from external memory and delay reuse 2) vectorizing channel data fetching which is enabled by delay reuse, and 3) using fixed summing networks to reduce consumption of logic resources. Our proposed method presents two unique advantages over current FPGA beamformers: 1) high scalability that allows fast adaptation to different FPGA resources and beamforming speed demands by using Xilinx High-Level Synthesis as the development tool, and 2) allow a compact form factor design by using a single FPGA to complete the beamforming instead of multiple FPGAs. With the proposed method, a sustainable average beamforming rate of 4.83 G samples/second in terms of input raw RF sample was achieved. The resulting image quality of the proposed beamformer was compared with the software beamformer on the Verasonics Vantage system for both phantom imaging and in vivo imaging of a mouse brain.
Improved Ultrasound Localization Microscopy Based on Microbubble Uncoupling via Transmit Excitation
Jihun Kim, Mathew R. Lowerison, Nathiya Vaithiyalingam Chandra Sekaran, Zhengchang Kou, Zhijie Dong, Michael L. Oelze, Daniel A. Llano, Pengfei Song
IF 3.6 (2022)
IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control
Ultrasound localization microscopy (ULM) demonstrates great potential for visualization of tissue microvasculature at depth with high spatial resolution. The success of ULM heavily depends on robust localization of isolated microbubbles (MBs), which can be challenging in vivo especially within larger vessels where MBs can overlap and cluster close together. While MB dilution alleviates the issue of MB overlap to a certain extent, it drastically increases the data acquisition time needed for MBs to populate the microvasculature, which is already on the order of several minutes using recommended MB concentrations. Inspired by optical super-resolution imaging based on stimulated emission depletion (STED), here we propose a novel ULM imaging sequence based on MB uncoupling via transmit excitation (MUTE). MUTE "silences" MB signals by creating acoustic nulls to facilitate MB separation, which leads to robust localization of MBs especially under high concentrations. The efficiency of localization accomplished via the proposed technique was first evaluated in simulation studies with conventional ULM as a benchmark. Then, an in-vivo study based on the chorioallantoic membrane (CAM) of chicken embryos showed that MUTE could reduce the data acquisition time by half, thanks to the enhanced MB separation and localization. Finally, the performance of MUTE was validated in an in vivo mouse brain study. These results demonstrate the high MB localization efficacy of MUTE-ULM, which contributes to a reduced data acquisition time and improved temporal resolution for ULM.
Compressed Sensing-Based Super-Resolution Ultrasound Imaging for Faster Acquisition and High Quality Images
Jihun Kim, Qingfei Wang, Siyuan Zhang, Sangpil Yoon
IF 4.756 (2021)
IEEE Transactions on Biomedical Engineering
<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Goal:</i> Typical SRUS images are reconstructed by localizing ultrasound microbubbles (MBs) injected in a vessel using normalized 2-dimensional cross-correlation (2DCC) between MBs signals and the point spread function of the system. However, current techniques require isolated MBs in a confined area due to inaccurate localization of densely populated MBs. To overcome this limitation, we developed the ℓ1-homotopy based compressed sensing (L1H-CS) based SRUS imaging technique which localizes densely populated MBs to visualize microvasculature in vivo. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Methods:</i> To evaluate the performance of L1H-CS, we compared the performance of 2DCC, interior-point method based compressed sensing (CVX-CS), and L1H-CS algorithms. Localization efficiency was compared using axially and laterally aligned point targets (PTs) with known distances and randomly distributed PTs generated by simulation. We developed post-processing techniques including clutter reduction, noise equalization, motion compensation, and spatiotemporal noise filtering for in vivo imaging. We then validated the capabilities of L1H-CS based SRUS imaging technique with high-density MBs in a mouse tumor model, kidney, and zebrafish dorsal trunk, and brain. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Results:</i> Compared to 2DCC and CVX-CS algorithms, L1H-CS achieved faster data acquisition time and considerable improvement in SRUS image quality. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conclusions and Significance:</i> These results demonstrate that the L1H-CS based SRUS imaging technique has the potential to examine microvasculature with reduced acquisition and reconstruction time to acquire enhanced SRUS image quality, which may be necessary to translate it into clinics.
Three-dimensional Shear Wave Elastography Using a 2D Row Column Addressing (RCA) Array
Zhijie Dong, Jihun Kim, Chengwu Huang, Matthew R. Lowerison, Shigao Chen, Pengfei Song
bioRxiv (Cold Spring Harbor Laboratory)
Abstract Objective To develop a 3D shear wave elastography (SWE) technique using a 2D row column addressing (RCA) array, with either external vibration or acoustic radiation force (ARF) as the shear wave source. Impact Statement The proposed method paves the way for clinical translation of 3D-SWE based on the 2D RCA, providing a low-cost and high volume-rate solution that is compatible with existing clinical systems. Introduction SWE is an established ultrasound imaging modality that provides a direct and quantitative assessment of tissue stiffness, which is significant for a wide range of clinical applications including cancer and liver fibrosis. SWE requires high frame-rate imaging for robust shear wave tracking. Due to the technical challenges associated with high volume-rate imaging in 3D, current SWE techniques are typically confined to 2D. Advancing SWE from 2D to 3D is significant because of the heterogeneous nature of tissue, which demands 3D imaging for accurate and comprehensive evaluation. Methods A 3D SWE method using a 2D RCA array was developed with a volume-rate up to 2000 Hz. The performance of the proposed method was systematically evaluated on tissue-mimicking elasticity phantoms. Results 3D shear wave motion induced by either external vibration or ARF was successfully detected with the proposed method. Robust 3D shear wave speed maps were reconstructed for both homogeneous and heterogeneous phantoms with inclusions. Conclusion The high volume-rate 3D imaging provided by the 2D RCA array provides a robust and practical solution for 3D SWE with a clear pathway for future clinical translation.
Improved Ultrasound Localization Microscopy based on Microbubble Uncoupling via Transmit Excitation (MUTE)
Jihun Kim, Mathew R. Lowerison, Nathiya Chandra Sekaran, Zhengchang Kou, Zhijie Dong, Michael L. Oelze, Daniel A. Llano, Pengfei Song
bioRxiv (Cold Spring Harbor Laboratory)
Abstract Ultrasound localization microscopy (ULM) demonstrates great potential for visualization of tissue microvasculature at depth with high spatial resolution. The success of ULM heavily depends on the robust localization of isolated microbubbles (MBs), which can be challenging in vivo especially within larger vessels where MBs can overlap and cluster close together. While MB dilution alleviates the issue of MB overlap to a certain extent, it drastically increases the data acquisition time needed for MBs to populate the microvasculature, which is already on the order of several minutes using recommended MB concentrations. Inspired by optical super-resolution imaging based on stimulated emission depletion (STED), here we propose a novel ULM imaging sequence based on microbubble uncoupling via transmit excitation (MUTE). MUTE “silences” MB signals by creating acoustic nulls to facilitate MB separation, which leads to robust localization of MBs especially under high concentrations. The efficiency of localization accomplished via the proposed technique was first evaluated in simulation studies with conventional ULM as a benchmark. Then an in vivo study based on the chorioallantoic membrane (CAM) of chicken embryos showed that MUTE could reduce the data acquisition time by half thanks to the enhanced MB separation and localization. Finally, the performance of MUTE was validated in an in vivo mouse brain study. These results demonstrate the high MB localization efficacy of MUTE-ULM, which contributes to a reduced data acquisition time and improved temporal resolution for ULM.
Singular value decomposition and 2D crosscorrelation based localization of gas vesicles for super-resolution ultrasound imaging
Jihun Kim, Gyoyeon Hwang, Sunghoon Rho, Sangpil Yoon
We report the potentials of nanometer-sized contrast agents which are called gas vesicles (GVs) for super-resolution ultrasound (SRUS) imaging to diagnose of vasculature deep inside tissue. Thus, we developed the GVs and ultrasound localization microscopy (ULM) based on singular value decomposition and 2D cross-correlation techniques. Furthermore, the SRUS imaging of the vessel-mimicking phantom with the GVs was performed. These results demonstrate that GVs could have potentials as novel contrast agents at nanoscale for implementing the SRUS imaging, thus indicating that ULM with GVs would be used for better visualization of micro-vasculature in vivo.
Compressed sensing-based super-resolution ultrasound imaging for faster acquisition and high quality images
Jihun Kim, Qingfei Wang, Siyuan Zhang, Sangpil Yoon
bioRxiv (Cold Spring Harbor Laboratory)
Abstract Super-resolution ultrasound (SRUS) imaging technique has overcome the diffraction limit of conventional ultrasound imaging, resulting in an improved spatial resolution while preserving imaging depth. Typical SRUS images are reconstructed by localizing ultrasound microbubbles (MBs) injected in a vessel using normalized 2-dimensional cross-correlation (2DCC) between MBs signals and the point spread function of the system. However, current techniques require isolated MBs in a confined area due to inaccurate localization of densely populated MBs. To overcome this limitation, we developed the ℓ 1 -homotopy based compressed sensing (L1H-CS) based SRUS imaging technique which localizes densely populated MBs to visualize microvasculature in vivo. To evaluate the performance of L1H-CS, we compared the performance of 2DCC, interior-point method based compressed sensing (CVX-CS), and L1H-CS algorithms. Localization efficiency was compared using axially and laterally aligned point targets (PTs) with known distances and randomly distributed PTs generated by simulation. We developed post-processing techniques including clutter reduction, noise equalization, motion compensation, and spatiotemporal noise filtering for in vivo imaging. We then validated the capabilities of L1H-CS based SRUS imaging technique with high-density MBs in a mouse tumor model, kidney, and zebrafish dorsal trunk, and brain. Compared to 2DCC, and CVX-CS algorithm, L1H-CS algorithm, considerable improvement in SRUS image quality and data acquisition time was achieved. These results demonstrate that the L1H-CS based SRUS imaging technique has the potential to examine the microvasculature with reduced acquisition and reconstruction time of SRUS image with enhanced image quality, which may be necessary to translate it into the clinics.