Deep Learning for Karolinska Sleepiness Scale Classification Based On Eye Aspect Ratio with SMOTE-Enhanced Data Balancing

Ahmad Zaini; Eko Mulyanto  Yuniarno; Yoyon K  Suprapto; Annida Miftakhul  Farodisa

doi:10.23887/janapati.v13i3.84962

Authors

Ahmad Zaini Electrical Engineering, Institut Teknologi Sepuluh Nopember
Eko Mulyanto Yuniarno Computer Engineering, Institut Teknologi Sepuluh Nopember
Yoyon K Suprapto Computer Engineering, Institut Teknologi Sepuluh Nopember
Annida Miftakhul Farodisa Computer Engineering, Institut Teknologi Sepuluh Nopember

DOI:

https://doi.org/10.23887/janapati.v13i3.84962

Keywords:

Classification, Deep Learning, Eye Aspect Ratio, Karolinska Sleepiness Scale, Multi Layer Perceptron, Sleepiness Scale

Abstract

This paper addresses the challenge of accurately classifying sleepiness levels based on the Karolinska Sleepiness Scale (KSS) using Eye Aspect Ratio (EAR) data, especially when class imbalance leads to biased predictions. The research proposes a deep learning framework that integrates a Multi-Layer Perceptron (MLP) with the Synthetic Minority Over-sampling Technique (SMOTE) to balance the training data. EAR features, representing eye closure patterns, are extracted from video frames, and SMOTE is applied to generate synthetic data for underrepresented sleepiness classes. By training the MLP model on this balanced dataset, the system achieves a 97.6% classification accuracy in distinguishing four distinct sleepiness levels based on the KSS, demonstrating its effectiveness in reducing prediction bias and managing class imbalance, both crucial for real-time drowsiness detection systems

References

AAA Foundation for Traffic Safety, “2022 Traffic Safety Culture Index (Technical Report),” Washington DC.

J. Vicente, P. Laguna, A. Bartra, and R. Bailón, “Drowsiness detection using heart rate variability,” Med. Biol. Eng. Comput., vol. 54, no. 6, pp. 927–937, Jun. 2016, doi: 10.1007/s11517-015-1448-7.

Yongqiang Li, Shangfei Wang, Yongping Zhao, and Qiang Ji, “Simultaneous Facial Feature Tracking and Facial Expression Recognition,” IEEE Trans. Image Process., vol. 22, no. 7, pp. 2559–2573, Jul. 2013, doi: 10.1109/TIP.2013.2253477.

I.-H. Choi, C.-H. Jeong, and Y.-G. Kim, “Tracking a Driver’s Face against Extreme Head Poses and Inference of Drowsiness Using a Hidden Markov Model,” Appl. Sci., vol. 6, no. 5, p. 137, May 2016, doi: 10.3390/app6050137.

R. Tamanani, R. Muresan, and A. Al-Dweik, “Estimation of Driver Vigilance Status Using Real-Time Facial Expression and Deep Learning,” IEEE Sens. Lett., vol. 5, no. 5, pp. 1–4, May 2021, doi: 10.1109/LSENS.2021.3070419.

L. Yang, H. Yang, H. Wei, Z. Hu, and C. Lv, “Video-Based Driver Drowsiness Detection With Optimised Utilization of Key Facial Features,” IEEE Trans. Intell. Transp. Syst., vol. 25, no. 7, pp. 6938–6950, Jul. 2024, doi: 10.1109/TITS.2023.3346054.

G. Du, L. Zhang, K. Su, X. Wang, S. Teng, and P. X. Liu, “A Multimodal Fusion Fatigue Driving Detection Method Based on Heart Rate and PERCLOS,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 11, pp. 21810–21820, Nov. 2022, doi: 10.1109/TITS.2022.3176973.

A. Murata, T. Doi, and W. Karwowski, “Sensitivity of PERCLOS70 to Drowsiness Level: Effectiveness of PERCLOS70 to Prevent Crashes Caused by Drowsiness,” IEEE Access, vol. 10, pp. 70806–70814, 2022, doi: 10.1109/ACCESS.2022.3187995.

P. Lancaric, N. Grilusova, and J. Kostolny, “System for Monitoring Computer User Fatigue,” in 2023 13th International Conference on Dependable Systems, Services and Technologies (DESSERT), Athens, Greece: IEEE, Oct. 2023, pp. 1–5. doi: 10.1109/DESSERT61349.2023.10416530.

Y. Lu, C. Liu, F. Chang, H. Liu, and H. Huan, “JHPFA-Net: Joint Head Pose and Facial Action Network for Driver Yawning Detection Across Arbitrary Poses in Videos,” IEEE Trans. Intell. Transp. Syst., vol. 24, no. 11, pp. 11850–11863, Nov. 2023, doi: 10.1109/TITS.2023.3285923.

Fouzia, R. Roopalakshmi, J. A. Rathod, A. S. Shetty, and K. Supriya, “Driver Drowsiness Detection System Based on Visual Features,” in 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), Coimbatore: IEEE, Apr. 2018, pp. 1344–1347. doi: 10.1109/ICICCT.2018.8473203.

C. B. S. Maior, M. J. D. C. Moura, J. M. M. Santana, and I. D. Lins, “Real-time classification for autonomous drowsiness detection using eye aspect ratio,” Expert Syst. Appl., vol. 158, p. 113505, Nov. 2020, doi: 10.1016/j.eswa.2020.113505.

K. Kaida et al., “Validation of the Karolinska sleepiness scale against performance and EEG variables,” Clin. Neurophysiol., vol. 117, no. 7, pp. 1574–1581, Jul. 2006, doi: 10.1016/j.clinph.2006.03.011.

J. Bai et al., “Two-Stream Spatial–Temporal Graph Convolutional Networks for Driver Drowsiness Detection,” IEEE Trans. Cybern., vol. 52, no. 12, pp. 13821–13833, Dec. 2022, doi: 10.1109/TCYB.2021.3110813.

M. Dreisig, M. H. Baccour, T. Schack, and E. Kasneci, “Driver Drowsiness Classification Based on Eye Blink and Head Movement Features Using the k-NN Algorithm,” in 2020 IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, ACT, Australia: IEEE, Dec. 2020, pp. 889–896. doi: 10.1109/SSCI47803.2020.9308133.

Y. Wu and Q. Ji, “Facial Landmark Detection: A Literature Survey,” Int. J. Comput. Vis., vol. 127, no. 2, pp. 115–142, Feb. 2019, doi: 10.1007/s11263-018-1097-z.

V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH: IEEE, Jun. 2014, pp. 1867–1874. doi: 10.1109/CVPR.2014.241.

C. Álvarez Casado and M. Bordallo López, “Real-time face alignment: evaluation methods, training strategies and implementation optimization,” J. Real-Time Image Process., vol. 18, no. 6, pp. 2239–2267, Dec. 2021, doi: 10.1007/s11554-021-01107-w.

T. Soukupova and J. ˇCech, “Real-Time Eye Blink Detection using Facial Landmarks,” 21st Comput. Vis. Winter Workshop, 2016.

A. Kuwahara, K. Nishikawa, R. Hirakawa, H. Kawano, and Y. Nakatoh, “Eye fatigue estimation using blink detection based on Eye Aspect Ratio Mapping(EARM),” Cogn. Robot., vol. 2, pp. 50–59, 2022, doi: 10.1016/j.cogr.2022.01.003.

Q. Massoz, T. Langohr, C. Francois, and J. G. Verly, “The ULg multimodality drowsiness database (called DROZY) and examples of use,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA: IEEE, Mar. 2016, pp. 1–7. doi: 10.1109/WACV.2016.7477715.

R. Ghoddoosian, M. Galib, and V. Athitsos, “A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA: IEEE, Jun. 2019, pp. 178–187. doi: 10.1109/CVPRW.2019.00027.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953.

Y. Bao and S. Yang, “Two Novel SMOTE Methods for Solving Imbalanced Classification Problems,” IEEE Access, vol. 11, pp. 5816–5823, 2023, doi: 10.1109/ACCESS.2023.3236794.

Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015, doi: 10.1038/nature14539.

H.-T. Nguyen, N.-D. Mai, B. G. Lee, and W.-Y. Chung, “Behind-the-Ear EEG-Based Wearable Driver Drowsiness Detection System Using Embedded Tiny Neural Networks,” IEEE Sens. J., vol. 23, no. 19, pp. 23875–23892, Oct. 2023, doi: 10.1109/JSEN.2023.3307766.

D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” Jan. 29, 2017, arXiv: arXiv:1412.6980. Accessed: Aug. 26, 2024. [Online]. Available: http://arxiv.org/abs/1412.6980

X. Han and L. Dai, “Multimodal Fatigue Recognition State Based on Eyelid Features,” in 2023 IEEE International Conference on Control, Electronics and Computer Technology (ICCECT), Jilin, China: IEEE, Apr. 2023, pp. 856–865. doi: 10.1109/ICCECT57938.2023.10141258.