Background US is clinically established for breast imaging, but its diagnostic performance depends on operator experience. Computer-assisted (real-time) image analysis may help in overcoming this limitation. Purpose To develop precise real-time-capable US-based breast tumor categorization by combining classic radiomics and autoencoder-based features from automatically localized lesions. Materials and Methods A total of 1619 B-mode US images of breast tumors were retrospectively analyzed between April 2018 and January 2024. nnU-Net was trained for lesion segmentation. Features were extracted from tumor segments, bounding boxes, and whole images using either classic radiomics, autoencoder, or both. Feature selection was performed to generate radiomics signatures, which were used to train machine learning algorithms for tumor categorization. Models were evaluated using the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity and were statistically compared with histopathologically or follow-up-confirmed diagnosis. Results The model was developed on 1191 (mean age, 61 years ± 14 [SD]) female patients and externally validated on 50 (mean age, 55 years ± 15]). The development data set was divided into two parts testing and training lesion segmentation (419 and 179 examinations) and lesion categorization (503 and 90 examinations). nnU-Net demonstrated precision and reproducibility in lesion segmentation in test set of data set 1 (median Dice score [DS] 0.90 [IQR, 0.84-0.93]; P = .01) and data set 2 (median DS 0.89 [IQR, 0.80-0.92]; P = .001). The best model, trained with 23 mixed features from tumor bounding boxes, achieved an AUC of 0.90 (95% CI 0.83, 0.97), sensitivity of 81% (46 of 57; 95% CI 70, 91), and specificity of 87% (39 of 45; 95% CI 77, 87). No evidence of difference was found between model and human readers (AUC = 0.90 [95% CI 0.83, 0.97] vs 0.83 [95% CI 0.76, 0.90]; P = .55 and 0.90 vs 0.82 [95% CI 0.75, 0.90]; P = .45) in tumor classification or between model and histopathologically or follow-up-confirmed diagnosis (AUC = 0.90 [95% CI 0.83, 0.97] vs 1.00 [95% CI 1.00,1.00]; P = .10). Conclusion Precise real-time US-based breast tumor categorization was developed by mixing classic radiomics and autoencoder-based features from tumor bounding boxes. ClinicalTrials.gov identifier NCT04976257 Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Bahl in this issue.