Lecture by O. L. de Lacalle Semantic text similarity with images

Dr. Oier Lopez de Lacalle is giving a talk next 16:00 June 26th, 2019 – Aula Anfiteatro Ex-Farmacologia Viale Morgagni 65. The talk will focus on “Evaluating Multimodal Representations on Sentence Similarity: The Visual Semantic Textual Similarity Dataset”.

In this talk Dr. Oier Lopez de Lacalle introduces vSTS, a new dataset for measuring textual similarity of sentences using multimodal information. The dataset is comprised by images along with its respectively textual captions and allows to study whether better sentence representations can be built when having access to corresponding images, e.g. a caption and its image, in contrast with having access to the text alone.

We describe the dataset both quantitatively and qualitatively, and claim that it is a valid gold standard for measuring automatic multimodal textual similarity systems.

vSTS extends the existing semantic textual similarity tasks with images, and aims at being a standard dataset to test the contribution of visual information when evaluating sentence representations.

We also describe the initial experiments combining the multimodal information. In the experiments we show that the dataset allows to explore two hypothesis:

  • H1) whether the image representations alone are able to predict caption similarity;
  • H2) whether a combination of image and text representations allow to improve the text-only results on this similarity task.

Biography: Dr. Oier Lopez de Lacalle is a B.Eng in Informatics (2003), received his Ph.D. in Computer Science in 2009 from the University of the Basque Country (UPV / EHU).

He is currently working as a post-doctoral researcher in the MUSTER project in the IXA group of the UPV / EHU. Previously,
he was a researcher hired by Ikerbasque at the University of Edinburgh. In these years he has published more than 40 articles in
journals and international conferences in the area of Natural Language Processing (NLP) and Artificial Intelligence (AI).

His research focuses mainly on semantic processing and information extraction using probabilistic modeling and deep learning algorithms.

The contributions various publications in JCR journals and high-level congresses (CORE A+). He is a regular reviewer in international journals and participates in international congress committee programs such as EACL, NAACL, ACL, EMNLP, and IJCNLP.

ACM Multimedia 2022 Awards

Best Paper & Demo Honorable Mention

Marie Curie

Lamberto Ballan Fellow of the week

Alberto Del Bimbo

Most cited researchers

Alberto Del Bimbo: MICC co-founder and former Director

My Kieu, Andrew D. Bagdanov, Marco Bertini, Alberto del Bimbo, Task-conditioned Domain Adaptation for Pedestrian Detection in Thermal Imagery

ECCV 2020 Paper

Domain Adaptation for Pedestrian Detection