In optical image registration, the reference control points (RCPs) used as explanatory variables in the polynomial regression model are generally assumed to be error free. However, this most frequently used assumption is often invalid in practice because RCPs always contain errors. In this situation, the extensively applied estimator, the ordinary least squares (LS) estimator, is biased and incapable of handling the errors in RCPs. Therefore, it is necessary to develop new feasible methods to address such a problem. This paper discusses the scaled total least squares (STLS) estimator, which is a generalization of the LS estimator in optical remote sensing image registration. The basic principle and the computational method of the STLS estimator and the relationship among the LS, total least squares (TLS) and STLS estimators are presented. Simulation experiments and real remotely sensed image experiments are carried out to compare LS and STLS approaches and systematically analyze the effect of the number and accuracy of RCPs on the performances in registration. The results show that the STLS estimator is more effective in estimating the model parameters than the LS estimator. Using this estimator based on the error-in-variables model, more accurate registration results can be obtained. Furthermore, the STLS estimator has superior overall performance in the estimation and correction of measurement errors in RCPs, which is beneficial to the study of error propagation in remote sensing data. The larger the RCP number and error, the more obvious are these advantages of the presented estimator.