Tutorial: Introduction to computational analysis for behavioral and clinical sciences
Tutorial date/time: 10am – 2pm, 18th of October 2022 (Local time in Japan)
Tutorial presenter(s):
Prof. Albert Ali Salah, Utrecht University, The Netherlands & Boğaziçi University, Turkey
Tutorial description:
This tutorial on computational analysis for behavioral and clinical sciences aims to introduce basic tools of human behavior analysis to students of both computer science and psychology, to enable collaborations between these disciplines. We will cover different application areas, discuss examples to illustrate the possibilities, as well as challenges and pitfalls of methodology.
Structure and Contents:
Part 1: Introduction to computational analysis for behavioral and clinical sciences. Issues in experiment design, including data collection, feature extraction and selection, machine learning pipeline, in-the-wild data, bias, overlearning.
Part 2: Face analysis: FACS coding, automatic analysis of face image and videos, example applications. Annotation and data quality. Affect and emotion estimation. Off the shelf tools (OpenFace).
Part 3: Body Dynamics: Body and pose estimation. Action and activity recognition. Wearable sensors. Temporal models. Off the shelf tools (OpenPose).
Part 4: Speech and paralinguistic analysis. Multimodality. Off the shelf tools (OpenSmile). Interaction analysis, including synchrony, rapport, mimicry, speed, intensity, regularity, extent.
Part 5: Presentations by students about their own related projects and discussion about how their analysis can be extended with these approaches. If you are registered to the tutorial, please send an email to a.a.salah@uu.nl to express interest in a short presentation of your project (5-8 minutes) followed by a joint discussion and feedback from all participants and the tutor.
Tutorial materials:
The following papers are provided as selected reading in this area. We recommend the interested students to peruse this material. The presentation slides and any other material, such as presentation notes, will be made available to the participants after the tutorial.
- Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2), 423-443. (available at: https://par.nsf.gov/servlets/purl/10099426)
- Cohn, Jeffrey F., and Fernando De la Torre. “Automated Face Analysis for Affective Computing.” The Oxford Handbook of Affective Computing (2014): 131.(available at: https://ict.usc.edu/~gratch/CSCI534/Readings/OHAC-10-Face.pdf)
- Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., & Cohen, D. (2012). Interpersonal synchrony: A survey of evaluation methods across disciplines. IEEE Transactions on Affective Computing, 3(3), 349-365. (available at: http://speapsl.aphp.fr/pdfpublications/2012/2012-12.pdf)
- Girard, Jeffrey M., and Jeffrey F. Cohn. “A primer on observational measurement.” Assessment 23.4 (2016): 404-413. (available at: https://psyarxiv.com/582cy/download?format=pdf)
- Halfon, S., M. Doyran, A.A. Salah, “Multimodal Affect Analysis of Psychodynamic Play Therapy,” Psychotherapy Research, vol. 31, no. 3, pp. 402-417, 2021. (available at: https://webspace.science.uu.nl/~salah006/halfon20multimodalAuthorProof.pdf)
- Mongan, J., Moy, L., & Kahn Jr, C. E. (2020). Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiology: Artificial Intelligence, 2(2), e200029. (may be available at: https://pubs.rsna.org/doi/full/10.1148/ryai.2020200029)
- Schuller, B. (2011). Voice and speech analysis in search of states and traits. In A.A. Salah, T. Gevers (eds) Computer Analysis of Human Behavior (pp. 227-253). Springer, London. (available at: https://mediatum.ub.tum.de/doc/1107295/file.pdf)
- Vinciarelli, A., Esposito, A., André, E., Bonin, F., Chetouani, M., Cohn, J. F., … & Salah, A. A. (2015). Open challenges in modelling, analysis and synthesis of human behaviour in human–human and human–machine interactions. Cognitive Computation, 7(4), 397-413. (available at: https://link.springer.com/article/10.1007/s12559-015-9326-z)
- Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social signal processing: Survey of an emerging domain. Image and vision computing, 27(12), 1743-1759. (available at: http://cs.uwindsor.ca/~xyuan/references/SocialSignal09.pdf)
Contact:
Albert Ali Salah (Utrecht University): a.a.salah@uu.nl
General enquiries to ACII2022 Tutorial Chair:
Youngjun Cho (UCL): youngjun.cho@ucl.ac.uk
Ruud Hortensius (Universiteit Utrecht): r.hortensius@uu.nl