Durable Interaction with Socially Intelligent Robots (iSocioBot)

Research project supported by The Danish Council for Independent Research | Technology and Production Sciences, Ministry of Science, Innovation and Higher Education (in Danish: Det Frie Forskningsråd | Teknologi og Produktion, Uddannelses- og Forskningsministeriet)

Grant DFF – 1335-00162

1 August 2013 – 31 December 2017

A mirror project homepage: http://socialrobot.dk

Project leader:

Zheng-Hua Tan

Department of Electronic Systems, Aalborg University, Denmark

E-mail: zt@es.aau.dk

Project description:

This project develops potentially groundbreaking methods that make service robots socially intelligent and capable of establishing durable relationship with their users. This relies on developing the capabilities to sense and express, which faces grand challenges: the low-quality signals and the poor context-awareness. We first propose a new paradigm called reinforcement fusion, which combines sensor signals in an interactive way: e.g. when a robot detects a sound direction, it turns towards the direction to see better and moves towards it to hear better. Reinforcement fusion is analogous to reinforcement learning, a known term in machine learning. It will dramatically improve robot's sensibility to the context including social behaviours. Secondly we propose a concept of social behaviour entrainment to adapt behaviours. Entrainment is the phenomenon that dialogue partners tend to adapt their speaking style of each other. Our hypothesis is that through reinforcement fusion based tracking and social information extraction, machine learning based context and user modelling, and social behaviour entrainment, durable social interaction between human and robot is achievable. The reinforcement fusion paradigm is applicable to all sorts of systems with steerable sensors.

Project members: Zheng-Hua Tan, Søren Holdt Jensen, Børge Lindberg, Nicolai Bæk Thomsen, Xiaodong Duan, Evgenios Vlachos, Ibrahim Hameed.

Robots: iSocioBot (or SocioBot), Nao, Pioneer 3-DX, TurtleBot, and HOVIS Genie.


    News:

1. Visit and talk by Prof. John H.L. Hansen, The University of Texas at Dallas, U.S.A. Talk title: “Speaker Diarization in Naturalistic Data with Application to Distant based Speech Recognition”, August 29, 2017 at 13:00.
   
2. DR News: p4 radioavisen, p1 radioavisen, March 2, 2017.
3. Our robotic research with healthy elders and elders suffering from dementia at Fremtidens Plejehjem facility was picked up by Aalborg Kommune LinkedIn page, July 15, 2016.
4. International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE 2016), jointly organised by iSocioBot project, was held at Aalborg University, Denmark, July 6-8, 2016.
5. News articles in BiTE (part of BT): Danskudviklet robot er fremtidens sociale hjaelper: Se hvad den kan and Om få år kan du få en robot, der kender dig så godt som dine venner, August 2015.
6. iSocioBot team offered 3 weeks summer school training course in Programming Social Robots for Human Interaction at Aalborg University, July 20-August 7, 2015.
7. iSocioBots and the team were present at the People's Meeting (Folkemødet) in Bornholm, June 2015. The People's Meeting, an annual event in Denmark, featured with 2700 events and attacked ca. 100 000 guests this year.
8. The second iSocioBot was built on the basis of Pioneer 3-DX, June 2015.
9. Visit and talk by Research Scientist Najim Dehak, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology (MIT), U.S.A. Talk title: “I-vector representation based on GMM and DNN for audio classification", May 19, 2015 at 13.00.
10. Nao and Pioneer 3-DX both joined the robot team in January 2015.
11. iSocioBot was present at the Culture Night 2014 in Copenhagen at the Ministry of Higher Education and Science (with about 4500 visitors) and interacted with Sofie Carsten Nielsen, Minister for Higher Education and Science, October 10, 2014.
12. iSocioBot inaugurated 'Safe 7' in Nibe, September 6, 2014.
13. Visit and talk by Prof. Jen-Tzung Chien, National Chiao Tung University. Talk title: "Bayesian Nonparametric Information Processing", August 26, 2014 at 13:30.
14. Our iSocialBot and Presenter Ulla Essendrop presided over the official opening of the Day of Research 2014 in Denmark (Forskningens Døgn 2014), which Crown Princess Mary attended with 250 invited guests on 24 April 2014. Et døgn der varer tre døgn | Nordjyske.dk. A video clip of the opening.
15. Visit and talk by Prof. John H.L. Hansen, The University of Texas at Dallas, U.S.A. Talk title: “Speaker & Noise Variability – Making Speech Systems Robust”, Jan. 29, 2014 at 13:00.
16. News article in Ingeniøren: Aalborg-forskere: Sådan bygger vi den menneskelige robot, 1 November 2013. Google English translation.

    Publications:

1. Zheng-Hua Tan, Achintya kr. Sarkar and Najim Dehak, “rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method,” Computer Speech and Language, vol. 59, pp. 1-21, January 2020.
2. Achintya kr. Sarkar, Zheng-Hua Tan, Hao Tang, Suwon Shon and James Glass, “Time-Contrastive Learning Based Deep Bottleneck Features for Text-Dependent Speaker Verification,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 27, no. 8, pp.1267-1279, August 2019.
3. Xiaodong Duan and Zheng-Hua Tan, “A Spatial Self-Similarity Based Feature Learning Method for Face Recognition under Varying Poses,” Pattern Recognition Letters, vol. 111, pp. 109-116, August 2018.
4. Evgenios Vlachos and Zheng-Hua Tan, “Public Perception of Android Robots: Indications from an Analysis of YouTube Comments,” the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018), Madrid, Spain, 1-5 October 2018.
5. Gabriele Trovato, Renato Paredes, Javier Balvin, Francisco Cuellar, Nicolai Bæk Thomsen, Søren Bech, and Zheng-Hua Tan, “The Sound or Silence: investigating the influence of robot noise on proxemics,” the 27th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2018, Nanjing and Tai’an, China, 27-31 August 2018.
6. Zheng-Hua Tan, Nicolai Bæk Thomsen, Xiaodong Duan, Evgenios Vlachos, Sven Ewan Shepstone, Morten H. Rasmussen and Jesper Lisby Højvang, “iSocioBot – A Multimodal Interactive Social Robot,” International Journal of Social Robotics, vol. 10, no. 1, pp. 5-19, Jan 2018.
7. Achintya Sarkar and Zheng-Hua Tan, “Incorporating Pass-Phrase Dependent Background Models for Text-Dependent Speaker Verification,” Computer Speech & Language, vol. 47, pp 259-271, January 2018.
8. Jen-Tzung Chien, Chao-Hsi Lee and Zheng-Hua Tan, “Latent Dirichlet Mixture Model,” Neurocomputing, vol. 278, pp 12-22, February 2018.
9. Xiaodong Duan, Nicolai B. Thomsen, Zheng-Hua Tan, Børge Lindberg and Søren H. Jensen, “Weighted Score Based Fast Converging CO-training with Application to Audio-Visual Person Identification,” The 29th IEEE International Conference on Tools with Artificial Intelligence (ICTAI2017), Boston, Massachusetts, USA, Nov. 6-8, 2017.
10. Achintya Kr. Sarkar and Zheng-Hua Tan, “Time-Contrastive Learning Based DNN Bottleneck Features for Text-Dependent Speaker Verification,” NIPS 2017 Time Series Workshop, Long Beach, CA, USA, Dec. 8, 2017.
11. Daniel Michelsanti and Zheng-Hua Tan, “Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification,” Interspeech 2017, Stockholm, Sweden, 20-24 August 2017.
12. Achintya Sarkar, Md Sahidullah, Zheng-Hua Tan and Tomi Kinnunen, “Improving Speaker Verification Performance in Presence of Spoofing Attacks Using Out-of-Domain Spoofed Data,” Interspeech 2017, Stockholm, Sweden, 20-24 August 2017.
13. Elizabeth Ann Jochum, Evgenios Vlachos, Sally Grindsted Nielsen, Anja Christoffersen, Ibrahim Hameed and Zheng-Hua Tan, “”Using Theatre to Study Interaction with Care Robots”, International Journal of Social Robotics,  2016, 8(4), 457-470. (Springer).
14. Jen-Tzung Chien, Chao-Hsi Lee and Zheng-Hua Tan, “Dirichlet Mixture Allocation”, the 26th IEEE International Workshop on Machine Learning for Signal Processing (MLSP), Salerno-Italy, 13-16 September 2016.
15. Zheng-Hua Tan, Najim Dehak, Jan Larsen and Zhanyu Ma (eds.), Proceedings of the First International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE 2016), IEEE Press, 2016.
16. Nicolai B. Thomsen, Xiaodong Duan, Zheng-Hua Tan, Børge Lindberg, and Søren Holdt Jensen, “Improving the Convergence of CO-training for Audio-Visual Person Identification,” The International Workshop on Sensing, Processing and Learning for Intelligent Machines (SPLINE2016), July 6-8, 2016, Aalborg, Denmark.
17. Ibrahim A. Hameed, Zheng-Hua Tan, Nicolai B. Thomsen and Xiaodong Duan, “User Acceptance of Social Robots,” The 9th International Conference on Advances in Computer-Human Interactions (ACHI 2016), Venice, Italy, April 24-28, 2016. Best Paper Award.
18. Xiaodong Duan and Zheng-Hua Tan, “Neighbors Based Discriminative Feature Difference Learning for Kinship Verification,” The 11th International Symposium on Visual Computing, December 14-16, 2015
    , Las Vegas, Nevada, USA.
19. Nicolai Thomsen, Zheng-Hua Tan, Børge Lindberg and Søren Holdt Jensen, “A Heuristic Approach for a Social Robot to Navigate to a Person Based on Audio and Range Information,” 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (iROS), Hamburg, Germany, September 28 – October 02, 2015.
20. Sally Grindsted Nielsen, Anja Christoffersen, Elizabeth Jochum and Zheng-Hua Tan, “Robot Future: Using Theatre to Influence Acceptance of Care Robots,” The New Friend 2015 Conference, Almere, The Netherlands, October 22-23, 2015. Best Paper Award Runner-up.
21. Xiaodong Duan and Zheng-Hua Tan, “Local Feature Learning for Face Recognition under Varying Poses,” IEEE International Conference on Image Processing (ICIP 2015), 27-30 September 2015, Quebec City, Canada.
22. Xiaodong Duan and Zheng-Hua Tan, “A Feature Subtraction Method for Image Based Kinship Verification under Uncontrolled Environments,” IEEE International Conference on Image Processing (ICIP 2015), 27-30 September 2015, Quebec City, Canada.
23. Rasmus Lyngby Kristensen, Zheng-Hua Tan, Zhanyu Ma and Jun Guo, “Binary Pattern Flavored Feature Extractors for Facial Expression Recognition: An Overview,” CIS-MIPRO 2015, 25-29 May 2015, Opatija, Croatia.
24. Zheng-Hua Tan, Nicolai Bæk Thomsen and Xiaodong Duan, “Designing and Implementing an Interactive Social Robot from Off-the-shelf Components,”The 3rd IFToMM Symposium on Mechanism Design for Robotics (MEDER2015), June 2-4, 2015, Aalborg, Denmark.
25. Nicolai Bæk Thomsen, Zheng-Hua Tan, Børge Lindberg and Søren Holdt Jensen, “Learning Direction of Attention for a Social Robot in Noisy Environments,” The 3rd AAU Workshop on Robotics (AAUROB2014), Aalborg, Denmark.
26. Nicolai B. Thomsen, Zheng-Hua Tan, Børge Lindberg and Søren Holdt Jensen, “Improving Robustness against Environmental Sounds for Directing Attention of Social Robots,” The 2nd Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction, September 14, 2014, Singapore.