CityU Talk



Information Theory and Multi-Modal Learning


Lizhong Zheng      October, 2020
@ CityU HK
 

What is Multi-Modal Learning?

  • Modalities:
  • Human Senses: 
  • Sensor Measurements:
  • Attributes:
  • Array Elements:
 
  • Applications:
  • Audio-visual speech recognition
  • Multi-media information retrieval
  • Image/Video captioning
  • Emotion recognition
 
  • 5 Core Challenges (Baltrusaitis, Morency’17):
  • Representation
  • Alignment
  • Fusion
  • Translation
  • Co-Learning
 

Why is this an Information Theory Problem?

  • Multi-Variate Dependence

  • To measure the dependence between x{\mathsf x} and y{\mathsf y}: mutual information, HGR correlation
  • Three random variables?



 



  • Core Problem of Multi-Terminal Information Theory
  • Broadcast channel
  • The CEO problem
  • Secret key generation