https://doi.org/10.1038/s41598-023-29606-9
“To accurately identify atoms on noisy transmission electron microscope images, a deep learning (DL) approach is employed to estimate the map of probabilities at each pixel for being an atom with element discernment. Thanks to a delicately-designed loss function and the ability to extract features, the proposed DL networks can be trained by a small dataset created from approximately 30 experimental images, each with a size of 256 × 256 pixels2. The accuracy and robustness of the network were verified by resolving the structural defects of graphene and polar structures in PbTiO3/SrTiO3 multilayers from both the general TEM images and their imitated images on which intensities of some pixels lost randomly. Such a network has the potential to identify atoms from very few images of beam-sensitive material and explosive images recorded in a dynamical atomic process. The idea of using a small-dataset-trained DL framework to resolve a specific problem may prove instructive for practical DL applications in various fields.
”
“Identifying the atomic positions in the plane of transmission electron microscope (TEM) images at atomic resolution with the precision of a picometer or sub-picometer is a key issue to the solution of characterizing the properties of a nanomaterial. Such a task is beneficial to material research including crystal structure characterization1,2,3, atomic polarization of polar structures4,5,6, stress and strain7, defect or surface mitigation3,8, and sequential catalyst reactions9,10, etc. However, the conventional methods for identifying the positions of atoms11,12,13,14,15 heavily depend on the quality of the acquired images and the factors affecting image quality include the inevitable noise, the intensities relating to the ability of atoms to scatter electrons in the scanning transmission electron microscope (STEM) images, and the lens distortion in the high-resolution transmission electron microscope (HRTEM) images. What’s more, the electron dose is one of the other factors since it is limited when acquiring HRTEM images for a beam-sensitive material16,17.
Application on HRTEM images of graphene
A free-standing graphene is mounted onto a SiN (silicon nitride) TEM grid and heated to 800 °C to clean the amorphous contamination in a DENSsolutions heating holder in in-situ TEM, which is a FEI™ Titan G2 80–300 microscope equipped with a Cs corrector and a monochromator operating at 80 kV. Each frame of the HRTEM images is recorded by a Gatan CCD (Ultrascan 1000) with an exposure time of 1 s and a sampling rate of 0.23 Å/pixel. A total number of 30 frames of images with line defect evolution of graphene at atomic resolution are recorded. According to the measured aberrations33, the simulated images could be similar to the experimental images. Additionally, due to noise with a signal-to-noise-ratio (SNR) of 2.35, the carbon atoms cannot be successfully identified on raw HRTEM images, and the simulated images were contaminated by noise at the same level.
The training set is a combination of the simulated and experimental images, since the astigmatism was not corrected if all the input–output pairs are only collected from the experimental images. Figure 2a and c shows one sample of the experimental and simulated input–output image pairs, in which the identified atoms are plotted in green spots within a dark background, as shown in Fig. 2b and d. In the simulation, carbon atoms are shifted randomly in 3-dimensional space with an average magnitude of 10.0 pm on the projection plane along the incident-electron direction, and each TEM image was simulated from one random configuration of atoms. Therefore, the atoms are not located on the ideal hexagonal lattice, which helps to analyze the precision of the atomic position on the predicted probability map. Additionally, graphene containing line defects and irregular boundaries enriches features on images (See Supplementary Fig. S2 for more input–output pairs). It is worth mentioning that the size of the input–output images is relatively small (256*256 pixels2) with a sampling rate of 0.23 Å/pixel.
Predictions and their accuracy of modes trained via a mixture of simulated images and experimental images. (a–d) Two sets of the input–output image pairs. (a) An input image intercepted from one experimental image, and (b) the output image prepared from the atomic positions identified by using the traditional method. (c) A simulated input image and (d) the output image with atoms plotted according to its known structure. (e) An experimental image in the test set and (f) its prediction. (g) The region extracted from the red box in (e), in which the yellow circles are the positions of the atoms measured from the probability map. (h–m) Evolution of the line defects and the holes in graphene extracted from the (h) 1st, (i) 10th, (j) 15th, (k) 20th, (l) 25th and (m) 30th frames in this image series.
In this paper, an iterative training procedure was adopted. First, only 5 experimental images and 5 simulated images were randomly chosen as the input–output pairs in the training set. The AP-GANs were trained for 400 epochs and the best model was chosen to give the best prediction results for all the selected experimental images with representative features in the test set. Due to the reason that some regions containing specific features on the experimental images were unable to be predicted if applying the AP-GANs trained by the simulated input–output image pairs, more images with specific features were added for network training. The term “specific features” refers to some structural or image features, such as line defects, graphene edges, etc. In the next iteration, 5 additional experimental pairs and 5 random simulated pairs were added to the training set to retrain the AP-GANs. In this experiment, the model converged and achieved satisfactory prediction results for all the experimental images once there were 60 image pairs in total in the training set.
The generator model was able to successfully predict the probability map for the experimental and simulated images in the test set. The probability map of an experimental image of Fig. 2e is shown in Fig. 2f, whilst the highlighted region is shown in Fig. 2g. Additionally, continuous structural evolution was revealed clearly, as shown in Fig. 2h–m (the corresponding raw images are in Supplementary Fig. S3). In this structural evolution, atoms were lost and rearranged ceaselessly around the defects due to beam radiation.
Then, the precision of the atomic positions identified from the predicted probability map was measured by applying the trained AP-GANs to the simulation images in the test set. Two trained AP-GANs were tested: (i) one was trained by the training set consisting of a combination of the simulated and experimental input–output image pairs, which is the same as the AP-GANs that are used for predicting the probability map of Fig. 2e; (ii) the other AP-GANs were trained by the training set containing only 60 simulated input–output image pairs. A simulated image, its probability map predicted from the first AP-GANs and its highlighted region are shown in Fig. 3a–c, respectively. Similarly, the probability map predicted from the second AP-GANs and its highlighted region is shown in Fig. 3d and e. Atoms plotted in yellow circles were measured from the probability maps, and comparably, the ground truth positions of the atoms were known from the atomic structure and marked by the green crosses in Fig. 3c and e. For the first and second AP-GANs, the root mean square errors (RMSEs) between the measured positions of atoms and their ground truths were 6.36 ± 3.62 pm and 6.20 ± 3.43 pm, respectively. Figure 3f and g shows the histogram of position errors, in which the errors and histograms are counted from about 800 positions.
The precision of networks trained by different training sets. (a) A simulated image in the test set, and (b) its probability map predicted from the AP-GANs trained by the training set mixed with the simulated and experimental input–output image pairs. (c) A region extracted from (a), on which yellow circles are atom positions measured from (b). (d) The probability map predicted from the AP-GANs trained by the simulated training set. (e) A region extracted from (a), on which yellow circles are atom positions measured from (d). (f, g) Histogram of position errors between the true and those measured from the (c) and (e) image, respectively. And the errors are counted from approximately 800 positions. (h) Atoms are measured directly from the phase of the simulated wave of (a) with the assistance of CalAtom software. (i) Histogram of errors between the ground truth and those measured from (h).
“