Comparing the performance of different Classification and Regression Algorithms

For my Machine Learning Course at UCSD, I wanted to push myself by publishing a research paper that closely followed the NeurIPS 2023 format and which was similar to the paper by Richard Caruana and Alexandru Niculescu-Mizil in their paper An Empirical Comparison of Supervised Learning Algorithms. To do this, I determined the algorithms of interest were Gradient XGBoosting for built-in regularization, Random Forest, and Neural Networks which could all be applied to both classification and regression tasks. I then decided on using F1 score as the primary metric for classification to handle class imbalances with ROC-AUC as the secondary measure. For regression, RMSE was the primary measure because it was interpretable and able to be used across models and R² was the secondary measure. I built a comprehensive machine learning pipeline for tabular data and for both regression and classification tasks that compared the performance of Gradient XGBoosting, Random Forest, Neural Networks, and SVMs (though SVMs are excluded from the paper due to time constraints in hyperparameter tuning especially for large datasets for it to be competitive, but you can try it yourself). We compared three different splits 20/80, 50/50, and 80/20 across three independent trials with different seeds using 10-Fold stratified CV for classification and 10-Fold CV for regression for each model and each dataset. The results were very close, but the best model was Gradient Boosting as it had the best F1 score for two out of the three classification datasets and the best RMSE score for one out of the two regression datasets. This research paper followed the NeurIPS 2023 format and was inspired by Richard Caruana and Alexandru Niculescu-Mizil in their paper An Empirical Comparison of Supervised Learning Algorithms. Although there was some overfitting, especially for the neural networks on the smaller datasets, after tweaking hyperparameters further and fully integrating SVM results given more time, I believe I could publish this paper. The paper is not yet submitted for publication, but with some revision like adding more datasets and fully integrating the SVM into the paper, it could be submitted for publication.