RESUMEN

In this paper, a practical discrete-time control method with adaptive image feature prediction for the image-based visual servoing (IBVS) scheme is presented. In the discrete-time IBVS inner-loop/outer-loop control architecture, the time delay caused by image capture and computation is noticed. Considering the dynamic characteristics of a 6-DOF manipulator velocity input system, we propose a linear dynamic model to describe the motion of a robot end effector. Furthermore, for better estimation of image features and smoothing of the robot's velocity input, we propose an adaptive image feature prediction method that employs past image feature data and real robot velocity data to adopt the prediction parameters. The experimental results on a 6-DOF robotic arm demonstrate that the proposed method can ensure system stability and accelerate system convergence.

RESUMEN

PURPOSE: Robot-assisted systems offer an opportunity to support the diagnostic and therapeutic treatment of vascular diseases to reduce radiation exposure and support the limited medical staff in vascular medicine. In the diagnosis and follow-up care of vascular pathologies, Doppler ultrasound has become the preferred diagnostic tool. The study presents a robotic system for automatic Doppler ultrasound examinations of patients' leg vessels. METHODS: The robotic system consists of a redundant 7 DoF serial manipulator, to which a 3D ultrasound probe is attached. A compliant control was employed, whereby the transducer was guided along the vessel with a defined contact force. Visual servoing was used to correct the position of the probe during the scan so that the vessel can always be properly visualized. To track the vessel's position, methods based on template matching and Doppler sonography were used. RESULTS: Our system was able to successfully scan the femoral artery of seven volunteers automatically for a distance of 20 cm. In particular, our approach using Doppler ultrasound data showed high robustness and an accuracy of 10.7 (±3.1) px in determining the vessel's position and thus outperformed our template matching approach, whereby an accuracy of 13.9 (±6.4) px was achieved. CONCLUSIONS: The developed system enables automated robotic ultrasound examinations of vessels and thus represents an opportunity to reduce radiation exposure and staff workload. The integration of Doppler ultrasound improves the accuracy and robustness of vessel tracking, and could thus contribute to the realization of routine robotic vascular examinations and potential endovascular interventions.

RESUMEN

BACKGROUND: The manipulation of compliant objects by robotic systems remains a challenging task, largely due to their variable shapes and the complex, high-dimensional nature of their interaction dynamics. Traditional robotic manipulation strategies struggle with the accurate modeling and control necessary to handle such materials, especially in the presence of visual occlusions that frequently occur in dynamic environments. Meanwhile, for most unstructured environments, robots are required to have autonomous interactions with their surroundings. METHODS: To solve the shape manipulation of compliant objects in an unstructured environment, we begin by exploring the regression-based algorithm of representing the high-dimensional configuration space of deformable objects in a compressed form that enables efficient and effective manipulation. Simultaneously, we address the issue of visual occlusions by proposing the integration of an adversarial network, enabling guiding the shaping task even with partial observations of the object. Afterwards, we propose a receding-time estimator to coordinate the robot action with the computed shape features while satisfying various performance criteria. Finally, model predictive controller is utilized to compute the robot's shaping motions subject to safety constraints. Detailed experiments are presented to evaluate the proposed manipulation framework. SIGNIFICANT FINDINGS: Our MPC framework utilizes the compressed representation and occlusion-compensated information to predict the object's behavior, while the multi-objective optimizer ensures that the resulting control actions meet multiple performance criteria. Through rigorous experimental validation, our approach demonstrates superior manipulation capabilities in scenarios with visual obstructions, outperforming existing methods in terms of precision and operational reliability. The findings highlight the potential of our integrated approach to significantly enhance the manipulation of compliant objects in real-world robotic applications.

RESUMEN

With neutron diffraction, the local stress and texture of metallic components can be analyzed non-destructively. For both, highly accurate positioning of the sample is essential, requiring the measurement at the same sample location from different directions. Current sample-positioning systems in neutron diffraction instruments combine XYZ tables and Eulerian cradles to enable the accurate six-degree-of-freedom (6DoF) handling of samples. However, these systems are not flexible enough. The choice of the rotation center and their range of motion are limited. Industrial six-axis robots have the necessary flexibility, but they lack the required absolute accuracy. This paper proposes a visual servoing system consisting of an industrial six-axis robot enhanced with a high-precision multi-camera tracking system. Its goal is to achieve an absolute positioning accuracy of better than 50µm. A digital twin integrates various data sources from the instrument and the sample in order to enable a fully automatic measurement procedure. This system is also highly relevant for other kinds of processes that require the accurate and flexible handling of objects and tools, e.g., robotic surgery or industrial printing on 3D surfaces.

RESUMEN

The research on acceleration-level visual servoing of manipulators is crucial yet insufficient, which restricts the potential application range of visual servoing. To address this issue, this paper proposes a quadratic programming-based acceleration-level image-based visual servoing (AIVS) scheme, which considers joint constraints. Besides, aiming to address the unknown problems in visual servoing systems, a data-driven learning algorithm is proposed to facilitate estimating structural information. Building upon this foundation, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is proposed, integrating learning and control capabilities. Subsequently, a recurrent neural network (RNN) is developed to tackle the DAIVS scheme, followed by theoretical analyses substantiating its stability. Afterwards, simulations and experiments on a Franka Emika Panda manipulator with eye-in-hand structure and comparisons among the existing methods are provided. The obtained results demonstrate the feasibility and practicality of the proposed schemes and highlight the superior learning and control ability of the proposed RNN. This method is particularly well-suited for visual servoing applications of manipulators with unknown structure.

RESUMEN

Quickly and accurately completing endoscopic submucosal dissection (ESD) operations within narrow lumens is currently challenging because of the environment's high flexibility, invisible collision, and natural tissue motion. This paper proposes a novel stereo visual servoing control for a dual-segment robotic endoscope (DSRE) for ESD surgery. Departing from conventional monocular-based methods, our DSRE leverages stereoscopic imaging to rapidly extract precise depth data, enabling quicker controller convergence and enhanced surgical accuracy. The system's dual-segment configuration enables agile maneuverability around lesions, while its compliant structure ensures adaptability within the surgical environment. The implemented stereo visual servo controller uses image features for real-time feedback and dynamically updates gain coefficients, facilitating rapid convergence to the target. In visual servoing experiments, the controller demonstrated strong performance across various tasks. Even when subjected to unknown external forces, the controller maintained robust performance in target tracking. The feasibility and effectiveness of the DSRE were further verified through ex vivo experiments. We posit that this novel system holds significant potential for clinical application in ESD surgeries.

RESUMEN

Robotic puncture system has been widely used in modern minimally invasive surgery, which usually uses hand-eye calibration to calculate the spatial relationship between the robot and the optical tracking system. However, the hand-eye calibration process is time-consuming and sensitive to environmental changes, which makes it difficult to guarantee the puncture accuracy of the robot. This study proposes an uncalibrated positioning method for puncture robot based on optical navigation. The method divides the target path positioning into two stages, angle positioning and position positioning, and designs angle image features and position image features respectively. The corresponding image Jacobian matrix is constructed based on the image features and updated by online estimation with a cubature Kalman filter to drive the robot to perform target path localization. The target path positioning results show that the method is more accurate than the traditional hand-eye calibration method and saves significant preoperative preparation time by eliminating the need for calibration.

Asunto(s)

RESUMEN

In this article, we present an innovative approach to 2D visual servoing (IBVS), aiming to guide an object to its destination while avoiding collisions with obstacles and keeping the target within the camera's field of view. A single monocular sensor's sole visual data serves as the basis for our method. The fundamental idea is to manage and control the dynamics associated with any trajectory generated in the image plane. We show that the differential flatness of the system's dynamics can be used to limit arbitrary paths based on the number of points on the object that need to be reached in the image plane. This creates a link between the current configuration and the desired configuration. The number of required points depends on the number of control inputs of the robot used and determines the dimension of the flat output of the system. For a two-wheeled mobile robot, for instance, the coordinates of a single point on the object in the image plane are sufficient, whereas, for a quadcopter with four rotating motors, the trajectory needs to be defined by the coordinates of two points in the image plane. By guaranteeing precise tracking of the chosen trajectory in the image plane, we ensure that problems of collision with obstacles and leaving the camera's field of view are avoided. Our approach is based on the principle of the inverse problem, meaning that when any point on the object is selected in the image plane, it will not be occluded by obstacles or leave the camera's field of view during movement. It is true that proposing any trajectory in the image plane can lead to non-intuitive movements (back and forth) in the Cartesian plane. In the case of backward motion, the robot may collide with obstacles as it navigates without direct vision. Therefore, it is essential to perform optimal trajectory planning that avoids backward movements. To assess the effectiveness of our method, our study focuses exclusively on the challenge of implementing the generated trajectory in the image plane within the specific context of a two-wheeled mobile robot. We use numerical simulations to illustrate the performance of the control strategy we have developed.

RESUMEN

In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system's robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system's potential for real-world application.

Asunto(s)

RESUMEN

We propose an improved extreme learning machine (ELM) to solve the decoupling problem between the camera coordinates and the image moment features for robot manipulator image-based visual servoing system, that is, determine the nonlinear relationship between them. First, an improved firefly optimization algorithm (IFOA) based on an adaptive inertial weight and individual variations is proposed. Then, the IFOA is optimized the weight and hidden bias in ELM algorithm; this improves the training accuracy of the ELM. Finally, the improved firefly optimization algorithm is integrated into ELM (IFOA-ELM) to solve the decoupling problem and ensure stable performance. The results of experiment show that the estimated error of the rotation angle around the camera frame in the visual servoing system determined by the IFOA-ELM algorithm is less than 0.25°, confirming that the proposed algorithm exhibits good robustness and stability.

RESUMEN

Some advantages of using cameras as sensor devices on feedback systems are the flexibility of the data it represents, the possibility to extract real-time information, and the fact that it does not require contact to operate. However, in unstructured scenarios, Image-Based Visual Servoing (IBVS) robot tasks are challenging. Camera calibration and robot kinematics can approximate a jacobian that maps the image features space to the robot actuation space, but they can become error-prone or require online changes. Uncalibrated visual servoing (UVS) aims at executing visual servoing tasks without previous camera calibration or through camera model uncertainties. One way to accomplish that is through jacobian identification using environment information in an estimator, such as the Kalman filter. The Kalman filter is optimal with Gaussian noise, but unstructured environments may present target occlusion, reflection, and other characteristics that confuse feature extraction algorithms, generating outliers. This work proposes RMCKF, a correntropy-induced estimator based on the Kalman Filter and the Maximum Correntropy Criterion that can handle non-Gaussian feature extraction noise. Unlike other approaches, we designed RMCKF for particularities in UVS, to deal with independent features, the IBVS control action, and simulated annealing. We designed Monte Carlo experiments to test RMCKF with non-Gaussian Kalman Filter-based techniques. The results showed that the proposed technique could outperform its relatives, especially in impulsive noise scenarios and various starting configurations.

RESUMEN

Robotic devices are gaining popularity for the physical rehabilitation of stroke survivors. Transition of these robotic systems from research labs to the clinical setting has been successful, however, providing robot-assisted rehabilitation in home settings remains to be achieved. In addition to ensure safety to the users, other important issues that need to be addressed are the real time monitoring of the installed instruments, remote supervision by a therapist, optimal data transmission and processing. The goal of this paper is to advance the current state of robot-assisted in-home rehabilitation. A state-of-the-art approach to implement a novel paradigm for home-based training of stroke survivors in the context of an upper limb rehabilitation robot system is presented in this paper. First, a cost effective and easy-to-wear upper limb robotic orthosis for home settings is introduced. Then, a framework of the internet of robotics things (IoRT) is discussed together with its implementation. Experimental results are included from a proof-of-concept study demonstrating that the means of absolute errors in predicting wrist, elbow and shoulder angles are 0.89180,2.67530 and 8.02580, respectively. These experimental results demonstrate the feasibility of a safe home-based training paradigm for stroke survivors. The proposed framework will help overcome the technological barriers, being relevant for IT experts in health-related domains and pave the way to setting up a telerehabilitation system increasing implementation of home-based robotic rehabilitation. The proposed novel framework includes:•A low-cost and easy to wear upper limb robotic orthosis which is suitable for use at home.•A paradigm of IoRT which is used in conjunction with the robotic orthosis for home-based rehabilitation.•A machine learning-based protocol which combines and analyse the data from robot sensors for efficient and quick decision making.

RESUMEN

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Asunto(s)

RESUMEN

For constrained image-based visual servoing (IBVS) of robot manipulators, a model predictive control (MPC) strategy tuned by reinforcement learning (RL) is proposed in this study. First, model predictive control is used to transform the image-based visual servo task into a nonlinear optimization problem while taking system constraints into consideration. In the design of the model predictive controller, a depth-independent visual servo model is presented as the predictive model. Next, a suitable model predictive control objective function weight matrix is trained and obtained by a deep-deterministic-policy-gradient-based (DDPG) RL algorithm. Then, the proposed controller gives the sequential joint signals, so that the robot manipulator can respond to the desired state quickly. Finally, appropriate comparative simulation experiments are developed to illustrate the efficacy and stability of the suggested strategy.

Asunto(s)

RESUMEN

Formation building for multi-small-AUV systems with on-board cameras is crucial under the limited communication underwater environment. A hybrid coordination strategy is proposed for the rapid convergence to a leader-follower pattern. The strategy consists of two parts: a time-optimal local-position-based controller (TOLC) and a distributed asynchronous discrete weighted consensus controller (ADWCC). The TOLC controller is designed to optimize the assignation of AUVs' destinations in the given pattern and guide each AUV to its destination by the shortest feasible distance. The ADWCC controller is developed to direct the AUVs blocked by obstacles to reach their destinations with the information from the perceived neighbors by on-board cameras. The rapidity of the proposed strategy is theoretically discussed. The effectiveness of the proposed algorithm has been verified in the simulation environments in both MATLAB and Blender.

RESUMEN

We propose a novel approach for robotic industrial insertion tasks using the Programming by Demonstration technique. Our method allows robots to learn a high-precision task by observing human demonstration once, without requiring any prior knowledge of the object. We introduce an Imitated-to-Finetuned approach that generates imitated approach trajectories by cloning the human hand's movements and then fine-tunes the goal position with a visual servoing approach. To identify features on the object used in visual servoing, we model object tracking as the moving object detection problem, separating each demonstration video frame into the moving foreground that includes the object and demonstrator's hand and the static background. Then a hand keypoints estimation function is used to remove the redundant features on the hand. The experiment shows that the proposed method can make robots learn precision industrial insertion tasks from a single human demonstration.

RESUMEN

The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block's pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row.

Asunto(s)

RESUMEN

PURPOSE: An inevitable feature of ultrasound-based diagnoses is that the quality of the ultrasound images produced depends directly on the skill of the physician operating the probe. This is because physicians have to constantly adjust the probe position to obtain a cross section of the target organ, which is constantly shifting due to patient respiratory motions. Therefore, we developed an ultrasound diagnostic robot that works in cooperation with a visual servo system based on deep learning that will help alleviate the burdens imposed on physicians. METHODS: Our newly developed robotic ultrasound diagnostic system consists of three robots: an organ tracking robot (OTR), a robotic bed, and a robotic supporting arm. Additionally, we used different image processing methods (YOLOv5s and BiSeNet V2) to detect the target kidney location, as well as to evaluate the appropriateness of the obtained ultrasound images (ResNet 50). Ultimately, the image processing results are transmitted to the OTR for use as motion commands. RESULTS: In our experiments, the highest effective tracking rate (0.749) was obtained by YOLOv5s with Kalman filtering, while the effective tracking rate was improved by about 37% in comparison with cases without such filtering. Additionally, the appropriateness probability of the ultrasound images obtained during the tracking process was also the highest and most stable. The second highest tracking efficiency value (0.694) was obtained by BiSeNet V2 with Kalman filtering and was a 75% improvement over the case without such filtering. CONCLUSION: While the most efficient tracking achieved is based on the combination of YOLOv5s and Kalman filtering, the combination of BiSeNet V2 and Kalman filtering was capable of detecting the kidney center of gravity closer to the kidney's actual motion state. Furthermore, this model could also measure the cross-sectional area, maximum diameter, and other detailed information of the target kidney, which meant it is more practical for use in actual diagnoses.

Asunto(s)

RESUMEN

Robotic puncture system has been widely used in modern minimally invasive surgery, which usually uses hand-eye calibration to calculate the spatial relationship between the robot and the optical tracking system. However, the hand-eye calibration process is time-consuming and sensitive to environmental changes, which makes it difficult to guarantee the puncture accuracy of the robot. This study proposes an uncalibrated positioning method for puncture robot based on optical navigation. The method divides the target path positioning into two stages, angle positioning and position positioning, and designs angle image features and position image features respectively. The corresponding image Jacobian matrix is constructed based on the image features and updated by online estimation with a cubature Kalman filter to drive the robot to perform target path localization. The target path positioning results show that the method is more accurate than the traditional hand-eye calibration method and saves significant preoperative preparation time by eliminating the need for calibration.

Asunto(s)

RESUMEN

Road infrastructure is one of the most vital assets of any country. Keeping the road infrastructure clean and unpolluted is important for ensuring road safety and reducing environmental risk. However, roadside litter picking is an extremely laborious, expensive, monotonous and hazardous task. Automating the process would save taxpayers money and reduce the risk for road users and the maintenance crew. This work presents LitterBot, an autonomous robotic system capable of detecting, localizing and classifying common roadside litter. We use a learning-based object detection and segmentation algorithm trained on the TACO dataset for identifying and classifying garbage. We develop a robust modular manipulation framework by using soft robotic grippers and a real-time visual-servoing strategy. This enables the manipulator to pick up objects of variable sizes and shapes even in dynamic environments. The robot achieves greater than 80% classified picking and binning success rates for all experiments; which was validated on a wide variety of test litter objects in static single and cluttered configurations and with dynamically moving test objects. Our results showcase how a deep model trained on an online dataset can be deployed in real-world applications with high accuracy by the appropriate design of a control framework around it.