•Markerless tumor tracking allows accurate target tracking without implanted markers.•4D planning CTs are subject to inaccuracies, and involve a high dose.•A DNN simulating 4DCT from 3DCT avoided ...many inaccuracies, lowering errors and dose.
Our markerless tumor tracking algorithm requires 4DCT data to train models. 4DCT cannot be used for markerless tracking for respiratory-gated treatment due to inaccuracies and a high radiation dose. We developed a deep neural network (DNN) to generate 4DCT from 3DCT data.
We used 2420 thoracic 4DCT datasets from 436 patients to train a DNN, designed to export 9 deformation vector fields (each field representing one-ninth of the respiratory cycle) from each CT dataset based on a 3D convolutional autoencoder with shortcut connections using deformable image registration. Then 3DCT data at exhale were transformed using the predicted deformation vector fields to obtain simulated 4DCT data. We compared markerless tracking accuracy between original and simulated 4DCT datasets for 20 patients. Our tracking algorithm used a machine learning approach with patient-specific model parameters. For the training stage, a pair of digitally reconstructed radiography images was generated using 4DCT for each patient. For the prediction stage, the tracking algorithm calculated tumor position using incoming fluoroscopic image data.
Diaphragmatic displacement averaged over 40 cases for the original 4DCT were slightly higher (<1.3 mm) than those for the simulated 4DCT. Tracking positional errors (95th percentile of the absolute value of displacement, “simulated 4DCT” minus “original 4DCT”) averaged over the 20 cases were 0.56 mm, 0.65 mm, and 0.96 mm in the X, Y and Z directions, respectively.
We developed a DNN to generate simulated 4DCT data that are useful for markerless tumor tracking when original 4DCT is not available. Using this DNN would accelerate markerless tumor tracking and increase treatment accuracy in thoracoabdominal treatment.
•We developed a deep neural network (DNN)-based artifact reduction method.•Our developed DNN successfully reduced the artifacts in the respective CT image sections.•Additional information such as ...multiple phase images or artifact map increased the artifact reduction accuracy.
Breathing artifact may affect the quality of four-dimensional computed tomography (4DCT) images. We developed a deep neural network (DNN)-based artifact reduction method.
We used 857 thoracoabdominal 4DCT data sets scanned with 320-section CT with no 4DCT artifact within any volume (ground-truth image). The limitations of graphics processing unit (GPU) memory prevent importation of CT volume data into the DNN. To simulate 4DCT artifact, we interposed 4DCT images from other breathing phases at selected couch positions.
Two DNNs, DNN1 and DNN2, were trained to maintain the quality of the output image to that of the ground truth by importing a single and 10 CT images, respectively. A third DNN consisting of an artifact classifier and image generator networks was added. The classifier network was based on residual networks and trained to detect CT section interposition-caused artifacts (artifact map). The generator network reduced artifacts by importing the coronal image data and the artifact map.
By repeating the 4DCT artifact reduction with coronal images, the geometrical accuracy in the sagittal sections could be improved, especially with DNN3. Diaphragm position was most accurate when DNN3 was applied. DNN2 corrected artifacts by using CT images from other phases, but DNN2 also modified artifact-free regions.
Additional information related to the 4DCT artifact, including information from other respiratory phases (DNN2) and/or artifact regions (DNN3), provided substantial improvement over DNN1. Interposition-related artifacts were reduced by use of an artifact positional map (DNN3).
•We have developed a new method to track tumor position using fluoroscopic images.•Tracking accuracy averaged over seven liver cases was 1.30 ± 0.54 mm.•Computation time was < 33 ms for a pair of ...fluoroscopic images.
We have developed a new method to track tumor position using fluoroscopic images, and evaluated it using hepatocellular carcinoma case data.
Our method consists of a training stage and a tracking stage. In the training stage, the model data for the positional relationship between the diaphragm and the tumor are calculated using four-dimensional computed tomography (4DCT) data. The diaphragm is detected along a straight line, which was chosen to avoid 4DCT artifact. In the tracking stage, the tumor position on the fluoroscopic images is calculated by applying the model to the diaphragm. Using data from seven liver cases, we evaluated four metrics: diaphragm edge detection error, modeling error, patient setup error, and tumor tracking error. We measured tumor tracking error for the 15 fluoroscopic sequences from the cases and recorded the computation time.
The mean positional error in diaphragm tracking was 0.57 ± 0.62 mm. The mean positional error in tumor tracking in three-dimensional (3D) space was 0.63 ± 0.30 mm by modeling error, and 0.81–2.37 mm with 1–2 mm setup error. The mean positional error in tumor tracking in the fluoroscopy sequences was 1.30 ± 0.54 mm and the mean computation time was 69.0 ± 4.6 ms and 23.2 ± 1.3 ms per frame for the training and tracking stages, respectively.
Our markerless tracking method successfully estimated tumor positions. We believe our results will be useful in increasing treatment accuracy for liver cases.
Purpose
To perform the final quality assurance of our fluoroscopic‐based markerless tumor tracking for gated carbon‐ion pencil beam scanning (C‐PBS) radiotherapy using a rotating gantry system, we ...evaluated the geometrical accuracy and tumor tracking accuracy using a moving chest phantom with simulated respiration.
Methods
The positions of the dynamic flat panel detector (DFPD) and x‐ray tube are subject to changes due to gantry sag. To compensate for this, we generated a geometrical calibration table (gantry flex map) in 15° gantry angle steps by the bundle adjustment method. We evaluated five metrics: (a) Geometrical calibration was evaluated by calculating chest phantom positional error using 2D/3D registration software for each 5° step of the gantry angle. (b) Moving phantom displacement accuracy was measured (±10 mm in 1‐mm steps) with a laser sensor. (c) Tracking accuracy was evaluated with machine learning (ML) and multi‐template matching (MTM) algorithms, which used fluoroscopic images and digitally reconstructed radiographic (DRR) images as training data. The chest phantom was continuously moved ±10 mm in a sinusoidal path with a moving cycle of 4 s and respiration was simulated with ±5 mm expansion/contraction with a cycle of 2 s. This was performed with the gantry angle set at 0°, 45°, 120°, and 240°. (d) Four types of interlock function were evaluated: tumor velocity, DFPD image brightness variation, tracking anomaly detection, and tracking positional inconsistency in between the two corresponding rays. (e) Gate on/off latency, gating control system latency, and beam irradiation latency were measured using a laser sensor and an oscilloscope.
Results
By applying the gantry flex map, phantom positional accuracy was improved from 1.03 mm/0.33° to <0.45 mm/0.27° for all gantry angles. The moving phantom displacement error was 0.1 mm. Due to long computation time, the tracking accuracy achieved with ML was <0.49 mm (=95% confidence interval CI) for imaging rates of 15 and 7.5 fps; those at 30 fps were decreased to 1.84 mm (95% CI: 1.79 mm–1.92 mm). The tracking positional accuracy with MTM was <0.52 mm (=95% CI) for all gantry angles and imaging frame rates. The tumor velocity interlock signal delay time was 44.7 ms (=1.3 frame). DFPD image brightness interlock latency was 34 ms (=1.0 frame). The tracking positional error was improved from 2.27 ± 2.67 mm to 0.25 ± 0.24 mm by the tracking anomaly detection interlock function. Tracking positional inconsistency interlock signal was output within 5.0 ms. The gate on/off latency was <82.7 ± 7.6 ms. The gating control system latency was <3.1 ± 1.0 ms. The beam irradiation latency was <8.7 ± 1.2 ms.
Conclusions
Our markerless tracking system is now ready for clinical use. We hope to shorten the computation time needed by the ML algorithm at 30 fps in the future.
The prostate moves in accordance with the movement of surrounding organs. Tumor position can change by ≥3 mm during radiotherapy. Given the difficulties of visualizing the prostate fluoroscopically, ...fiducial markers are generally implanted into the prostate to monitor its motion during treatment. Recently, internally motion guidance methods of the prostate using a 99.5% gold/0.5% iron flexible notched wire fiducial marker (Gold Anchor® , Naslund Medical AB, Huddinge, Sweden), which requires a 22 gauge needle, has been used. However, because the notched wire can retain its linear shape, acquire a spiral shape, or roll into an irregular ball, detecting it on fluoroscopic images in real-time incurs higher computation costs.
We developed a fiducial tracking algorithm to achieve real-time computation. The marker is detected on the first image frame using a shape filter that employs inter-class variance for the marker likelihood calculated by the filter, focusing on the large difference in densities between the marker and its surroundings. After the second frame, the marker is tracked by adding to the shape filter the similarity to the template cropped from the area around the marker position detected in the first frame. We retrospectively evaluated the algorithm's marker tracking accuracy for ten prostate cases, analyzing two fractions in each case.
Tracking positional accuracy averaged over all patients was 0.13 ± 0.04 mm (mean ± standard deviation, Euclidean distance) and 0.25 ± 0.09 mm (95th percentile). Computation time was 2.82 ± 0.20 ms/frame averaged over all frames.
Our algorithm successfully and stably tracked irregularly-shaped markers in real time.
•We have developed a fluoroscopic markerless tumor tracking algorithm based on a deep neural network (DNN).•Tracking accuracy averaged over ten patients was 1.64 ± 0.73 mm.•Computation time was ...<40 ms for a pair of fluoroscopic images.
To improve respiratory gating accuracy and treatment throughput, we developed a fluoroscopic markerless tumor tracking algorithm based on a deep neural network (DNN).
In the learning stage, target positions were projected onto digitally reconstructed radiography (DRR) images from four-dimensional computed tomography (4DCT). DRR images were cropped into subimages of the target or surrounding regions to build a network that takes input of the image pattern of subimages and produces a target probability map (TPM) for estimating the target position. Using multiple subimages, a DNN was trained to generate a TPM based on the target position projected onto the DRRs. In the tracking stage, the network takes in the subimages cropped from fluoroscopic images at the same position of the subimages on the DRRs and produces TPMs, which are used to estimate target positions. We integrated the lateral correction to modify an estimated target position by using a linear regression model. We tracked five lung and five liver cases, and calculated tracking accuracy (Euclidian distance in 3D space) by subtracting the estimated position from the reference.
Tracking accuracy averaged over all patients was 1.64 ± 0.73 mm. Accuracy for liver cases (1.37 ± 0.81 mm) was better than that for lung cases (1.90 ± 0.65 mm). Computation time was <40 ms for a pair of fluoroscopic images.
Our markerless tracking algorithm successfully estimated tumor positions. We believe our results will provide useful information to advance tumor tracking technology.
We sought to accelerate 2D/3D image registration computation time using image synthesis with a deep neural network (DNN) to generate digitally reconstructed radiographic (DRR) images from X-ray flat ...panel detector (FPD) images. And we explored the feasibility of using our DNN in the patient setup verification application. Images of the prostate and of the head and neck (H&N) regions were acquired by two oblique X-ray fluoroscopic units and the treatment planning CT. DNN was designed to generate DRR images from the FPD image data. We evaluated the quality of the synthesized DRR images to compare the ground-truth DRR images using the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM). Image registration accuracy and computation time were evaluated by comparing the 2D-3D image registration algorithm using DRR and FPD image data with DRR and synthesized DRR images. Mean PSNR values were 23.4 ± 3.7 dB and 24.1 ± 3.9 dB for the pelvic and H&N regions, respectively. Mean SSIM values for both cases were also similar (= 0.90). Image registration accuracy was degraded by a mean of 0.43 mm and 0.30°, it was clinically acceptable. Computation time was accelerated by a factor of 0.69. Our DNN successfully generated DRR images from FPD image data, and improved 2D-3D image registration computation time up to 37% in average.
We developed a deep neural network (DNN) to generate X-ray flat panel detector (FPD) images from digitally reconstructed radiographic (DRR) images. FPD and treatment planning CT images were acquired ...from patients with prostate and head and neck (H&N) malignancies. The DNN parameters were optimized for FPD image synthesis. The synthetic FPD images’ features were evaluated to compare to the corresponding ground-truth FPD images using mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). The image quality of the synthetic FPD image was also compared with that of the DRR image to understand the performance of our DNN. For the prostate cases, the MAE of the synthetic FPD image was improved (= 0.12 ± 0.02) from that of the input DRR image (= 0.35 ± 0.08). The synthetic FPD image showed higher PSNRs (= 16.81 ± 1.54 dB) than those of the DRR image (= 8.74 ± 1.56 dB), while SSIMs for both images (= 0.69) were almost the same. All metrics for the synthetic FPD images of the H&N cases were improved (MAE 0.08 ± 0.03, PSNR 19.40 ± 2.83 dB, and SSIM 0.80 ± 0.04) compared to those for the DRR image (MAE 0.48 ± 0.11, PSNR 5.74 ± 1.63 dB, and SSIM 0.52 ± 0.09). Our DNN successfully generated FPD images from DRR images. This technique would be useful to increase throughput when images from two different modalities are compared by visual inspection.
A novel successive learning algorithm based on a Test Feature Classifier is proposed for efficient handling of sequentially provided training data. The fundamental characteristics of the successive ...learning are considered. In the learning, after recognition of a set of unknown data by a classifier, they are fed into the classifier in order to obtain a modified performance. An efficient algorithm is proposed for the incremental definition of prime tests which are irreducible combinations of features and capable of classifying training patterns into correct classes. Four strategies for addition of training patterns are investigated with respect to their precision and performance using real pattern data. A real-world problem of classification of defects on wafer images has been dealt with by the proposed classifier, obtaining excellent performance even through efficient addition strategies.
In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation. However, large DNN models ...are needed to achieve state-of-the-art performance, exceeding the capabilities of edge devices. Model reduction is thus needed for practical use. In this paper, we point out that deep learning automatically induces group sparsity of weights, in which all weights connected to an output channel (node) are zero, when training DNNs under the following three conditions: (1) rectified-linear-unit (ReLU) activations, (2) an L2-regularized objective function, and (3) the Adam optimizer. Next, we analyze this behavior both theoretically and experimentally, and propose a simple model reduction method: eliminate the zero weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets, we demonstrate the sparsity with various training setups. Finally, we show that our method can efficiently reduce the model size and performs well relative to methods that use a sparsity-inducing regularizer.
PDF
