Integration of Bayesian Methods in Machine Learning: A Theoretical and Empirical Review

Syaharuddin Syaharuddin

doi:10.23887/insert.v5i2.82710

Authors

Syaharuddin Syaharuddin Universitas Muhammadiyah Mataram

DOI:

https://doi.org/10.23887/insert.v5i2.82710

Keywords:

Metode Bayesian, Pembelajaran Mesin, Tinjauan Teoretis, Tinjauan Empiris

Abstract

Abstrak
Studi ini merupakan sebuah tinjauan literatur sistematis yang mendalami integrasi metode Bayesian dalam pembelajaran mesin. Metode Bayesian telah terbukti memberikan keuntungan signifikan dalam menangani ketidakpastian dan variabilitas data, yang merupakan tantangan utama dalam pengembangan model pembelajaran mesin yang handal. Tinjauan ini menggali literatur terbitan dalam 10 tahun terakhir dari sumber seperti Scopus, DOAJ, dan Google Scholar untuk mengidentifikasi pendekatan teoritis dan empiris dalam penggunaan metode Bayesian. Hasil penelitian menyoroti bahwa metode Bayesian memungkinkan integrasi pengetahuan awal yang informatif dengan data yang ada, memperbaiki estimasi parameter, dan meningkatkan ketahanan model terhadap variasi data yang kompleks. Namun, tantangan utama dalam implementasi metode ini adalah kompleksitas komputasional yang tinggi dalam melakukan inferensi, terutama dalam konteks data multi-modal. Tinjauan ini memberikan wawasan mendalam tentang perkembangan terbaru dalam aplikasi metode Bayesian dalam pembelajaran mesin dan merumuskan arah penelitian mendatang untuk mengatasi tantangan tersebut secara efektif.

Abstrak
This study presents a systematic literature review that delves into the integration of Bayesian methods in machine learning. Bayesian methods have been shown to provide significant advantages in handling data uncertainty and variability, which are major challenges in developing reliable machine learning models. This review explores literature published in the last 10 years from sources such as Scopus, DOAJ, and Google Scholar to identify theoretical and empirical approaches in the use of Bayesian methods. The findings highlight that Bayesian methods enable the integration of informative prior knowledge with existing data, improve parameter estimation, and enhance model robustness against complex data variations. However, the main challenge in implementing these methods is the high computational complexity involved in performing inference, especially in the context of multi-modal data. This review provides in-depth insights into the latest developments in the application of Bayesian methods in machine learning and formulates future research directions to effectively address these challenges.

References

Abdullah, A. A., Hassan, M. M., & Mustafa, Y. T. (2023). Uncertainty Quantification for MLP-Mixer Using Bayesian Deep Learning. Applied Sciences (Switzerland). https://doi.org/10.3390/app13074547

Aflakparast, M., & de Gunst, M. (2019). Data integrative Bayesian inference for mixtures of regression models. Journal of the Royal Statistical Society. Series C: Applied Statistics. https://doi.org/10.1111/rssc.12346

Albert, C. G., Callies, U., & Toussaint, U. von. (2022). A Bayesian Approach to the Estimation of Parameters and Their Interdependencies in Environmental Modeling. Entropy. https://doi.org/10.3390/e24020231

Bertolini, R., Finch, S. J., & Nehm, R. H. (2023). An application of Bayesian inference to examine student retention and attrition in the STEM classroom. Frontiers in Education. https://doi.org/10.3389/feduc.2023.1073829

Cui, T., Marzouk, Y. M., & Willcox, K. E. (2015). Data-driven model reduction for the Bayesian solution of inverse problems. International Journal for Numerical Methods in Engineering. https://doi.org/10.1002/nme.4748

De Franco, C., Nicolle, J., & Pham, H. (2018). Dealing with Drift Uncertainty: A Bayesian Learning Approach. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3285248

Dera, D., Ahmed, S., Bouaynaya, N. C., & Rasool, G. (2024). TRustworthy Uncertainty Propagation for Sequential Time-Series Analysis in RNNs. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2023.3288628

Di Cortona, G. G., Messina, A., & Piacentini, S. (2023). Treatment of systematic uncertainties with Bayesian networks. Journal of Physics: Conference Series. https://doi.org/10.1088/1742-6596/2438/1/012131

Dong, J., Liao, J., Huan, X., & Cooper, D. (2023). Expert elicitation and data noise learning for material flow analysis using Bayesian inference. Journal of Industrial Ecology. https://doi.org/10.1111/jiec.13399

Edinburgh, T., Ercole, A., & Eglen, S. (2023). Bayesian model selection for multilevel models using integrated likelihoods. PLoS ONE. https://doi.org/10.1371/journal.pone.0280046

Feroz, F., Hobson, M. P., Cameron, E., & Pettitt, A. N. (2019). Importance Nested Sampling and the MultiNest Algorithm. The Open Journal of Astrophysics. https://doi.org/10.21105/astro.1306.2144

Held, L., Bové, D. S., & Gravestock, I. (2015). Approximate Bayesian model selection with the deviance statistic. Statistical Science. https://doi.org/10.1214/14-STS510

Hobbs, B. F. (1997). Bayesian methods for analysing climate change and water resource uncertainties. Journal of Environmental Management. https://doi.org/10.1006/jema.1996.0116

Ida Bagus Indra Dewangkara, Nyoman Dewi Ratna Iswara, Made NikenNovianti, Alif Bangun Prakoso, Laisa Santika, Adi Rizky Setiawan, AanTriambudi, Asafik Daroini, Eriz Rifqi Irawan, Ni Wayan Destiena Marella Apsari, M. A. L. (2022). PENGEMBANGAN WEBSITE PROJECT MANAGEMENT SYSTEM SPBE CAMP MENGGUNAKAN METODE DESIGN THINKING DAN FRAMEWORK LARAVEL. INSERT: Information System and Emerging Technology Journal, 3(2), 105–116. https://ejournal.undiksha.ac.id/index.php/insert/article/view/55961

Johnson, M. S., & Green, W. H. (2024). A machine learning based approach to reaction rate estimation. Reaction Chemistry and Engineering. https://doi.org/10.1039/d3re00684k

Jørgensen, A. C. S., Ghosh, A., Sturrock, M., & Shahrezaei, V. (2022). Efficient Bayesian inference for stochastic agent-based models. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1009508

Kaplan, D. (2021). On the Quantification of Model Uncertainty: A Bayesian Perspective. Psychometrika. https://doi.org/10.1007/s11336-021-09754-5

Kottke, D., Herde, M., Sandrock, C., Huseljic, D., Krempl, G., & Sick, B. (2021). Toward optimal probabilistic active learning using a Bayesian approach. Machine learning. https://doi.org/10.1007/s10994-021-05986-9

Kuipers, J., Suter, P., & Moffa, G. (2022). Efficient Sampling and Structure Learning of Bayesian Networks. Journal of Computational and Graphical Statistics. https://doi.org/10.1080/10618600.2021.2020127

Lavecchia, A. (2015). Machine-learning approaches in drug discovery: Methods and applications. In Drug Discovery Today. https://doi.org/10.1016/j.drudis.2014.10.012

Lee, M. D. (2018). Bayesian Methods in Cognitive Modeling. In Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience. https://doi.org/10.1002/9781119170174.epcn502

Luo, X., & Kareem, A. (2020). Bayesian deep learning with hierarchical prior: Predictions from limited and noisy data. Structural Safety. https://doi.org/10.1016/j.strusafe.2019.101918

Manderson, A. A., & Goudie, R. J. B. (2022). A numerically stable algorithm for integrating Bayesian models using Markov melding. Statistics and Computing. https://doi.org/10.1007/s11222-022-10086-2

Marschall, M., Wübbeler, G., Schmähling, F., & Elster, C. (2023). Machine learning based priors for Bayesian inversion in MR imaging. Metrologia. https://doi.org/10.1088/1681-7575/ace3c2

Mezzetti, M., Ryan, C. P., Balestrucci, P., Lacquaniti, F., & Moscatelli, A. (2023). Bayesian hierarchical models and prior elicitation for fitting psychometric functions. Frontiers in Computational Neuroscience. https://doi.org/10.3389/fncom.2023.1108311

Mohammadian mosammam, A., Abbasi, E., & Mateu, J. (2023). Bayesian Approach for Modelling Spatial–Temporal Crime Data. Journal of Statistical Sciences. https://doi.org/10.52547/jss.16.2.435

Musil, F., Willatt, M. J., Langovoy, M. A., & Ceriotti, M. (2019). Fast and Accurate Uncertainty Estimation in Chemical Machine learning. Journal of Chemical Theory and Computation. https://doi.org/10.1021/acs.jctc.8b00959

Nabi, S., Nassif, H., Hong, J., Mamani, H., & Imbens, G. (2022). Bayesian Meta-Prior Learning Using Empirical Bayes. Management Science. https://doi.org/10.1287/mnsc.2021.4136

Panigrahi, S., Mohammed, S., Rao, A., & Baladandayuthapani, V. (2023). Integrative Bayesian models using Post-selective inference: A case study in radiogenomics. Biometrics. https://doi.org/10.1111/biom.13740

Papamarkou, T., Hinkle, J., Young, M. T., & Womble, D. (2022). Challenges in Markov Chain Monte Carlo for Bayesian Neural Networks. Statistical Science. https://doi.org/10.1214/21-STS840

Parker, R. A., Sande, T. A., Laird, B., Hoskin, P., Fallon, M., & Colvin, L. (2022). Bayesian methods in palliative care research: cancer-induced bone pain. BMJ Supportive and Palliative Care. https://doi.org/10.1136/bmjspcare-2019-002160

Pavone, A., Merlo, A., Kwak, S., & Svensson, J. (2023). Machine learning and Bayesian inference in nuclear fusion research: an overview. In Plasma Physics and Controlled Fusion. https://doi.org/10.1088/1361-6587/acc60f

Pirš, G., & Strumbelj, E. (2019). Bayesian combination of probabilistic classifiers using multivariate normal mixtures. Journal of Machine learning Research.

Polson, N. G., & Sokolov, V. (2017). Deep learning: A Bayesian perspective. Bayesian Analysis. https://doi.org/10.1214/17-BA1082

Pratama, R. R. (2020). Analisis Model Machine learning Terhadap Pengenalan Aktifitas Manusia. MATRIK : Jurnal Manajemen, Teknik Informatika Dan Rekayasa Komputer. https://doi.org/10.30812/matrik.v19i2.688

QOSIM, A. L. (2021). PERBANDINGAN METODE KLASIFIKASI SUPPORT VECTOR MACHINE (SVM) DAN NAIVE BAYES CLASSIFIER (NBC) UNTUK MENENTUKAN KUALITAS UDARA. In Industry and Higher Education.

Sambasivan, R., Das, S., & Sahu, S. K. (2020). A Bayesian perspective of statistical machine learning for big data. Computational Statistics. https://doi.org/10.1007/s00180-020-00970-8

Schillings, C., Sprungk, B., & Wacker, P. (2020). On the convergence of the Laplace approximation and noise-level-robustness of Laplace-based Monte Carlo methods for Bayesian inverse problems. Numerische Mathematik. https://doi.org/10.1007/s00211-020-01131-1

Sukarya, I. G. A., Pradnyana, I. M. A., & Sugihartini, N. (2021). Analisis Faktor-faktor yang Mempengaruhi Perilaku Penggunaan Sistem E-Learning Undiksha dengan Model Unified Theory of Acceptance and Use of Technology (UTAUT). INSERT : Information System and Emerging Technology Journal. https://doi.org/10.23887/insert.v1i2.25940

van de Schoot, R., Depaoli, S., King, R., Kramer, B., Märtens, K., Tadesse, M. G., Vannucci, M., Gelman, A., Veen, D., Willemsen, J., & Yau, C. (2021). Bayesian statistics and modelling. In Nature Reviews Methods Primers. https://doi.org/10.1038/s43586-020-00001-2

Wang, Z. A., Pei, J. C., Chen, Y. J., Qiao, C. Y., Xu, F. R., Ge, Z. G., & Shu, N. C. (2022). Bayesian approach to heterogeneous data fusion of imperfect fission yields for augmented evaluations. Physical Review C. https://doi.org/10.1103/PhysRevC.106.L021304

Zeng, X., & Luo, G. (2017). Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection. Health Information Science and Systems. https://doi.org/10.1007/s13755-017-0023-z

Zhang, H., Chen, W. (Wayne), Iyer, A., Apley, D. W., & Chen, W. (2022). Uncertainty-aware mixed-variable machine learning for materials design. Scientific Reports. https://doi.org/10.1038/s41598-022-23431-2

Zhang, Y., & Hu, Z. (2022). Parameter Learning of Bayesian Network with Multiplicative Synergistic Constraints. Symmetry. https://doi.org/10.3390/sym14071469

Zhang, Y., & Lee, A. A. (2019). Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning. Chemical Science. https://doi.org/10.1039/c9sc00616h