Prof. Tat-Seng CHUA National University of Singapore (NUS) "Towards Relation Inference and Causal Reasoning in Video"

Dr Chua is the KITHCT Chair Professor at the School of Computing, National University of Singapore (NUS). He is also the Distinguished Visiting Professor of Tsinghua University. Dr Chua was the Founding Dean of the School of Computing from 1998-2000. His main research interests include unstructured data analytics, multimedia information retrieval, recommendation and conversation systems, and the emerging applications in E-commerce, wellness and Fintech. Dr Chua is the co-Director of NExT, a joint research Center between NUS and Tsinghua, focusing on Extreme Search.

Dr Chua is the recipient of the 2015 ACM SIGMM Achievements Award for the Outstanding Technical Contributions to Multimedia Computing, Communications and Applications. He is the Chair of steering committee of ACM International Conference on Multimedia Retrieval (ICMR) and Multimedia Modeling (MMM) conference series. He was the General Co-Chair of ACM Multimedia 2005, ACM CIVR (now ACM ICMR) 2005, ACM SIGIR 2008, and ACM Web Science 2015. He serves in the editorial boards of three international journals. Dr. Chua is the co-Founder of two technology startup companies in Singapore. He holds a PhD from the University of Leeds, UK.

Prof. Tat-Seng CHUA National University of Singapore (NUS) "Towards Relation Inference and Causal Reasoning in Video"

Recent advances in personal video recording devices and video sharing platforms, such as the Youtube and TikTok, have resulted in an explosive growth of online videos. This has given rise to demands for various video content understanding tasks, including video search, captioning, question answering (QA), and most importantly, content filtering. It motivates us to research into comprehensive and fine-grained video content analytics. A crux in content understanding is the object relations in video, which capture a wide variety of interactions between the objects. It permits the system to extract relation triplets in the form of <subject, predicate, object> which elevates the extractable semantic of video to that of text, and supports a wide range of higher level video-text fusion tasks such as the multi-perspective video captioning, video QA, and multimodal knowledge graph. It also supports causal reasoning, such as the counterfactual inference.

In this talk, I will present background on object recognition, and discuss key challenges in transforming research from object recognition to that of video relation inference and causal reasoning. I will conclude the talk by outlining several interesting future research directions and applications in social media analytics. As part of this research, we have constructed a large-scale video dataset to facilitate relevant research in this emerging field. The dataset has been used for a grand challenge in the ACM Multimedia 2019 conference.

Prof. Munchurl Kim Korea Advanced Institute of Science and Technology (KAIST) "Deep Video Compression: A Paradigm Shift from Block-based Hybrid Coding to Deep Learning based Neural Computation"

Munchurl Kim is a Professor at the School of Electrical Engineering, Korea Advance Institute of Science and Technology (KAIST). Munchurl Kim received his Ph.D. degrees in Electrical and Computer Engineering from University of Florida, Gainesville, USA, in 996. From 1997 to 2000, he was with Electronics and Telecommunications Research Institute (ETRI) where he led national research projects in object-oriented video coding with MPEG-4 standardization, and in multimedia searching, retrieval with MPEG-7 standardization, and data broadcasting. He had been involved in MPEG standardization activities of ISO/IEC and he served as Head of Korean delegates for MPEG standardization in 2000~2001. Throughout 12 year’s standardization activities, he proposed 146 technical contributions to JCT-VC/MPEG standardizations, and served as a Project Leader and Project Editors for the MPEG-7 and MPEG-A standard developments. He holds 35 HEVC essential patents and holds 96 registered domestic patents, and 50 registered international patents. He has authored and coauthored more than 286 international and domestic journal and conference papers. His research of interest includes deep learning based image restoration and quality enhancement, and deep video compression.

Prof. Munchurl Kim Korea Advanced Institute of Science and Technology (KAIST) "Deep Video Compression: A Paradigm Shift from Block-based Hybrid Coding to Deep Learning based Neural Computation"

After the first invention for DPCM (differential pulse code modulation) in 1952, the video compression technologies have been steadily developed up to date within the frame of DPCM-based motion-compensated transform coding. The coding efficiency improvement has been doubled about every ten years from MPEG-2 Video to HEVC (High Efficiency Video Coding) standard. Currently, the JVET (Joint Video Exports Team) of ISO/IEC MPEG and ITU-T VCEG is collaboratively developing a next generation video coding standard, called versatile video coding (VVC), still pushing to double the coding efficiency compared to HEVC and will finalize the standardization in Oct. 2020. Although many technologies (coding tools) improving coding efficiency have been developed, all the preceding video coding standards are based on block-based hybrid coding structures with transform and motion-compensated prediction. The efforts of coding efficiency improvement have mainly been focused on advancing inter-frame prediction, transforms, and entropy coding to reduce the temporal, spatial and statistical redundancies in video sequences. Beside these block-based hybrid coding technologies, deep learning based image and video compression efforts have been made with increasing interests, and many research works are being introduced through top-tier conferences. The deep learning based image compression began in real earnest based on auto-encoder with deep convolutional layers in 2016. This first attempt showed that such a deep learning approach could outperform the JPEG and WebP in terms of SSIM, and its following recurrent work neural network (RNN) based architecture in 2018 has exceeded to BPG (HEVC Intra Coding) in coding efficiency in terms of MS-SSIM. Other works with entropy learning in 2019 have outperformed the BPG eventually in both PSNR and SSIM with more than 10% coding efficiency improvement. It can be easily expected that upcoming deep learning based image compression methods would soon outperform the VVC Intra coding performance. In deep video compression, an increasing number of works are being presented with various network architectures that are beneficial in learning the compact representation in latent feature spaces.

In this talk, I will retrace the effort of video coding develops and standardization, and discuss about the limitations in the conventional approaches. Then I will introduce the recent deep learning approaches to image and video coding technologies and their some combinations with conventional video codecs. We will show some research outputs for fully deep learning based video compression and hybrid coding approaches in a combinations of a conventional video codec and deep learning technologies with applications for 4K/8K UHD video compression.

Prof. Carlos Busso The University of Texas at Dallas "Multimodal Emotion Analysis and Synthesis: From Audiovisual Interplay to Contextual Information in Dyadic Human Interaction"

Carlos Busso received his BS and MS degrees with high honors in electrical engineering from the University of Chile, Santiago, Chile, in 2000 and 2003, respectively, and his PhD degree (2008) in electrical engineering from the University of Southern California (USC), Los Angeles, in 2008. He is an associate professor at the Electrical Engineering Department of The University of Texas at Dallas (UTD). He was selected by the School of Engineering of Chile as the best electrical engineer graduated in 2003 across Chilean universities. At USC, he received a provost doctoral fellowship from 2003 to 2005 and a fellowship in Digital Scholarship from 2007 to 2008. At UTD, he leads the Multimodal Signal Processing (MSP) laboratory [http://msp.utdallas.edu]. He is a recipient of an NSF CAREER Award. In 2014, he received the ICMI Ten-Year Technical Impact Award. In 2015, his student received the third prize IEEE ITSS Best Dissertation Award (N. Li). He also received the Hewlett Packard Best Paper Award at the IEEE ICME 2011 (with J. Jain), and the Best Paper Award at the AAAC ACII 2017 (with Yannakakis and Cowie). He is the co-author of the winner paper of the Classifier Sub-Challenge event at the Interspeech 2009 emotion challenge. His research interest is in human-centered multimodal machine intelligence and applications. His current research includes the broad areas of affective computing, multimodal human-machine interfaces, nonverbal behaviors for conversational agents, in-vehicle active safety system, and machine learning methods for multimodal processing. His work has direct implication in many practical domains, including national security, health care, entertainment, transportation systems, and education. He has served as general chair of ACII 2017 and ICMI 2021. He is a member of ISCA, AAAC, and ACM, and a senior member of the IEEE.

Prof. Carlos Busso The University of Texas at Dallas "Multimodal Emotion Analysis and Synthesis: From Audiovisual Interplay to Contextual Information in Dyadic Human Interaction"

The verbal and non-verbal channels of human communication are internally and intricately connected. As a result, gestures and speech present high levels of correlation and coordination. This relationship is greatly affected by the linguistic and emotional content of the message being communicated. The interplay is observed across the different communication channels such as various aspects of speech, facial expressions, and movements of the hands, head and body. For example, facial expressions and prosodic speech tend to have a stronger emotional modulation when the vocal tract is physically constrained by the articulation to convey other linguistic communicative goals. This interplay also affects gestures and speech events observed across individuals. Psycholinguistic studies on human communication have shown that during human interaction individuals tend to adapt their behaviors mimicking the spoken style, gestures and expressions of their conversational partners. This synchronization pattern is referred to as entrainment. We study the presence of entrainment at the emotion level in cross-modality settings and its implications on multimodal emotion recognition systems. The analysis explores the relationship between acoustic features of the speaker and facial expressions of the interlocutor during dyadic interactions. The analysis shows a strong mutual influence in their expressive behaviors. We will discuss clear implications of these results for audiovisual emotion recognition, and audiovisual synthesis of expressive conversational agents.