ML-based Attack on Digitally Authenticated RSA Algorithm via Model Estimation: A Comparative Evaluation of Neural Network Architectures

Abstract

In this paper, we evaluate whether deep learning can learn the number theoretic structure needed to factor RSA semiprimes, with the objective of comparing architectures and feature designs for accuracy, efficiency, and generalization. We study four models: a Dual Output LSTM with binary inputs, an Enhanced Transformer with 107 to 117 mathematical features, a Hybrid CNN RNN, and a Factorization GAN. Datasets are built by exhaustive enumeration: for each upper bound between ten to the third and ten to the sixth, every integer is tested and kept if it has exactly two prime factors, yielding complete corpora of ten to twenty bit semiprimes. Performance is measured with beta k metrics defined by Hamming distance, where beta k counts predictions within k bit errors of the true factor and beta zero is exact match. On the large scale dataset, the Factorization GAN attains the highest exact match accuracy, 53.7 percent, with only 700 thousand parameters. Enhanced feature models exceed 91 percent within four bit tolerance. Training takes 89 seconds to 2.6 hours per model per scale. These findings suggest that deep learning, adversarial training, and mathematical feature engineering may enhance factor prediction performance and offer promising directions for integrating machine learning with classical cryptanalysis.