Automated discomfort detection for infants and toddlers based on video analysis


Infants and toddlers below five years of age cannot clearly orally articulate their emotion, especially discomfort and pain. Based on the research, neglecting pain of infants may cause damage to their nervous system. However, when an infant stays at hospital, it costs great labor efforts for the nurses and the doctors to assess his pain frequently enough. Therefore, an automated discomfort detection system for infant monitoring is strongly demanded by hospitals. The basic idea of building such a system is to use a camera to keep filming the patient lying on bed, while a computer can analyze the video in real time to determine whether the discomfort of the patient occurs.


Based on the computer vision techniques, the algorithm for discomfort detection can be mainly divided into 4 steps: face detection, facial fiducial points (eyes, nose and mouth etc.) localization, appearance normalization, expression classification (comfort or discomfort). Misclassification can stem from each of the mentioned steps. Our current work uses a combination of the Haar cascade face detector, constrained local model, local binary features and support vector machine, which can achieve an accuracy of 83% for discomfort detection. However, there are still two main challenges remaining: (1) it only works well for frontal upright faces, and if yaw and pitch angles change, the algorithm fails; (2) it can only analyse single frames from the video, a temporal filter is required for the continuous detection. Several research activities can be help to solve this:

  • Implementation of a robust facial fiducial points localization algorithm for any face orientation
  • Identification of discriminative features and machine learning methods for robust and accurate discomfort detection
  • Development of a temporal filter for continuous discomfort detection
  • Evaluation of the developed discomfort detection methods on clinical infant videos and a comparative validation to alternative approaches proposed in literature

The algorithm can be developed in Matlab or a combination of OpenCV and C/C++.

This project will be mainly executed internally at the TU/e, but also partly at Veldhoven MMC hospital.
Video content analysis, machine learning, image analysis, healthcare/perinatology
Contact person: