NL
EN

PhD position: Multimodal Alignment with Common Sense

  • Nieuwe Universiteitsgebouw (NU)
  • Amsterdam
  • 1 / 1
  • Faculty of Science
  • PhD
  • MSc
  • Closes on
    19-05-2025
  • Publishing date
    24-04-2025
Apply
Are you a strong AI MSc graduate passionate and knowledgeable about multimodal learning and reasoning? Then keep reading, this fully funded PhD position at Vrije Universiteit (VU) Amsterdam may be for you!

Your function

Vision-language models (VLMs) represent a paradigm shift in artificial intelligence, moving from traditional single-task supervised learning to more versatile general-purpose learning, enabling seamless adaptation to multiple tasks. While VLMs' performance in lab settings has notably improved, concerns about their compositionality, transparency, and adaptability persist. Moreover, those trained via contrastive self-supervised learning, such as CLIP, often exhibit an additional challenge: a persistent modality gap. In essence, multimodal models still struggle to connect different modalities holistically because of a lack of commonsense reasoning capabilities.

The Learning and Reasoning and the Quantitative Data Analytics groups seek a joint PhD student in Multimodal Alignment with Common Sense. This project will explore methods to enhance their robustness, abstraction, and visual grounding abilities to bridge the modality gap and address the grand challenge of commonsense reasoning with multimodal foundation models. The PhD student will develop innovative self-supervised learning frameworks to align vision and text better while seamlessly incorporating commonsense knowledge. The project will validate the hypothesis that integrating commonsense knowledge not only enhances alignment but also improves transparency and explainability in cross-modal inference. The PhD student may explore different representations for reasoning, including neural, natural language, and structural formats, for application domains such as video question answering. To bridge the gap between these two fields, the PhD student will be supported by one supervisor with extensive commonsense AI expertise (Dr. Filip Ilievski) and one with extensive machine learning expertise (Dr. Shujian Yu).

Your duties

  • shape and refine the indicated research topic according to your interests
  • conduct research on the indicated topic using a variety of research methods
  • publish the results as scientific papers in prestigious AI venues
  • attend suitable conferences to present the research
  • support BSc / MSc teaching and supervision activities in the department

Your profile

  • a strong MSc degree in Artificial Intelligence, Data Science, Computer Science, or a related field
  • at least foundational knowledge in both machine learning and reasoning, but more advanced knowledge in one of the two areas is expected
  • solid programming abilities, e.g., in Python
  • excellent communication skills in English (verbal and written)
  • motivated team player who shows initiative to drive their research
  • bonus: experience with foundational multimodal, language, or vision models
  • bonus: experience with commonsense AI
  • bonus: experience with information-theoretic learning
  • bonus: prior publications
  • bonus: industry experience (especially with multimodal AI systems)

As a university, we strive for equal opportunities for all, recognising that diversity takes many forms. We believe that diversity in all its complexity is invaluable for the quality of our teaching, research and service. We are always looking for talent with diverse backgrounds and experiences. This also means that we are committed to creating an inclusive community so that we can use diversity as an asset.

We realise that each individual brings a unique set of skills, expertise and mindset. Therefore we are happy to invite anyone who recognises themselves in the profile to apply, even if you do not meet all the requirements.

What do we offer?

A challenging position in a socially engaged organisation. At VU Amsterdam, you contribute to education, research and service for a better world. And that is valuable. So in return for your efforts, we offer you:

  • a salary of minimum € 2.901,00 (PhD) and maximum € 3.707,00 (PhD) gross per month, on a full-time basis. This is based on UFO profile PhD candidate. The exact salary depends on your education and experience.
  • a position for 1 FTE. Your employment contract will initially last 18 months. After a satisfactory evaluation of the initial appointment, the contract will be extended to a total duration of 4 years.

We also offer you attractive fringe benefits and regulations. Some examples:

  • A full-time 38-hour working week comes with a holiday leave entitlement of 232 hours per year. If you choose to work 40 hours, you have 96 extra holiday leave hours on an annual basis. For part-timers, this is calculated pro rata.
  • 8% holiday allowance and 8.3% end-of-year bonus
  • solid pension scheme (ABP)
  • contribution to commuting expenses
  • optional model for designing a personalized benefits package

About us

Faculty of Science
Researchers and students at VU Amsterdam’s Faculty of Science tackle fundamental and complex scientific problems to help pave the way for a sustainable and healthy future. From forest fires to big data, from obesity to malnutrition, and from molecules to the moon: we cover the full spectrum of the natural sciences. Our teaching and research have a strong experimentally technical, computational and interdisciplinary nature.

We work on new solutions guided by value-driven, interdisciplinary methodologies. We are committed to research, valorisation and training socially engaged citizens of the world who will make valuable contributions to a sustainable, healthy future.

Are you interested in joining the Faculty of Science? You will join undergraduate students, PhD candidates and researchers at the biggest sciences faculty in the Netherlands. You will combine a professional focus with a broad view of the world. We are proud of our collegial working climate, characterised by committed staff, a pragmatic attitude and engagement in the larger whole. The faculty is home to over 11,000 students enrolled in 40 study programmes. It employs over 1,600 professionals spread across 10 academic departments.

Vrije Universiteit Amsterdam
Vrije Universiteit Amsterdam stands for values-driven education and research. We are open-minded experts with the ability to think freely - a broader mind. Maintaining an entrepreneurial perspective and concentrating on diversity, significance and humanity, we work on sustainable solutions with social impact. By joining forces, across the boundaries of disciplines, we work towards a better world for people and planet. Together we create a safe and respectful working and study climate, and an inspiring environment for education and research. Learn more about our codes of conduct

We are located on one physical campus, in the heart of Amsterdam's Zuidas business district, with excellent location and accessibility.  Over 6,150 staff work at the VU and over 31,000 students attend academic education.

Diversity
Diversity is the driving force of VU Amsterdam. VU wants to be accessible and receptive to diversity in disciplines, cultures, ideas, nationalities, beliefs, preferences and worldviews. We believe that trust, respect, interest and differences lead to new insights and innovation, to sharpness and clarity, to excellence and a broader understanding.

We stand for an inclusive community and believe that diversity and internationalisation contribute to the quality of education, research and our services.

Therefore, we are always searching for people whose backgrounds and experience contribute to the diversity of the VU community.

Interested

Are you interested in this position? Then please submit an application by May 19th, 2025 with the following documents: 

  1. A CV that describes relevant experience / expertise for the position
  2. MSc certificate (if already available) and list of (current) MSc grades
  3. Cover letter outlining why you are interested in this position, including the name and email of two people we can contact for a reference
  4. Prior publications, if any

Applications received by e-mail will not be considered.

Acquisition in response to this advertisement is not appreciated.

Your function

Vision-language models (VLMs) represent a paradigm shift in artificial intelligence, moving from traditional single-task supervised learning to more versatile general-purpose learning, enabling seamless adaptation to multiple tasks. While VLMs' performance in lab settings has notably improved, concerns about their compositionality, transparency, and adaptability persist. Moreover, those trained via contrastive self-supervised learning, such as CLIP, often exhibit an additional challenge: a persistent modality gap. In essence, multimodal models still struggle to connect different modalities holistically because of a lack of commonsense reasoning capabilities.

The Learning and Reasoning and the Quantitative Data Analytics groups seek a joint PhD student in Multimodal Alignment with Common Sense. This project will explore methods to enhance their robustness, abstraction, and visual grounding abilities to bridge the modality gap and address the grand challenge of commonsense reasoning with multimodal foundation models. The PhD student will develop innovative self-supervised learning frameworks to align vision and text better while seamlessly incorporating commonsense knowledge. The project will validate the hypothesis that integrating commonsense knowledge not only enhances alignment but also improves transparency and explainability in cross-modal inference. The PhD student may explore different representations for reasoning, including neural, natural language, and structural formats, for application domains such as video question answering. To bridge the gap between these two fields, the PhD student will be supported by one supervisor with extensive commonsense AI expertise (Dr. Filip Ilievski) and one with extensive machine learning expertise (Dr. Shujian Yu).

Your duties

  • shape and refine the indicated research topic according to your interests
  • conduct research on the indicated topic using a variety of research methods
  • publish the results as scientific papers in prestigious AI venues
  • attend suitable conferences to present the research
  • support BSc / MSc teaching and supervision activities in the department

Your profile

  • a strong MSc degree in Artificial Intelligence, Data Science, Computer Science, or a related field
  • at least foundational knowledge in both machine learning and reasoning, but more advanced knowledge in one of the two areas is expected
  • solid programming abilities, e.g., in Python
  • excellent communication skills in English (verbal and written)
  • motivated team player who shows initiative to drive their research
  • bonus: experience with foundational multimodal, language, or vision models
  • bonus: experience with commonsense AI
  • bonus: experience with information-theoretic learning
  • bonus: prior publications
  • bonus: industry experience (especially with multimodal AI systems)

As a university, we strive for equal opportunities for all, recognising that diversity takes many forms. We believe that diversity in all its complexity is invaluable for the quality of our teaching, research and service. We are always looking for talent with diverse backgrounds and experiences. This also means that we are committed to creating an inclusive community so that we can use diversity as an asset.

We realise that each individual brings a unique set of skills, expertise and mindset. Therefore we are happy to invite anyone who recognises themselves in the profile to apply, even if you do not meet all the requirements.

What do we offer?

A challenging position in a socially engaged organisation. At VU Amsterdam, you contribute to education, research and service for a better world. And that is valuable. So in return for your efforts, we offer you:

  • a salary of minimum € 2.901,00 (PhD) and maximum € 3.707,00 (PhD) gross per month, on a full-time basis. This is based on UFO profile PhD candidate. The exact salary depends on your education and experience.
  • a position for 1 FTE. Your employment contract will initially last 18 months. After a satisfactory evaluation of the initial appointment, the contract will be extended to a total duration of 4 years.

We also offer you attractive fringe benefits and regulations. Some examples:

  • A full-time 38-hour working week comes with a holiday leave entitlement of 232 hours per year. If you choose to work 40 hours, you have 96 extra holiday leave hours on an annual basis. For part-timers, this is calculated pro rata.
  • 8% holiday allowance and 8.3% end-of-year bonus
  • solid pension scheme (ABP)
  • contribution to commuting expenses
  • optional model for designing a personalized benefits package

About us

Faculty of Science
Researchers and students at VU Amsterdam’s Faculty of Science tackle fundamental and complex scientific problems to help pave the way for a sustainable and healthy future. From forest fires to big data, from obesity to malnutrition, and from molecules to the moon: we cover the full spectrum of the natural sciences. Our teaching and research have a strong experimentally technical, computational and interdisciplinary nature.

We work on new solutions guided by value-driven, interdisciplinary methodologies. We are committed to research, valorisation and training socially engaged citizens of the world who will make valuable contributions to a sustainable, healthy future.

Are you interested in joining the Faculty of Science? You will join undergraduate students, PhD candidates and researchers at the biggest sciences faculty in the Netherlands. You will combine a professional focus with a broad view of the world. We are proud of our collegial working climate, characterised by committed staff, a pragmatic attitude and engagement in the larger whole. The faculty is home to over 11,000 students enrolled in 40 study programmes. It employs over 1,600 professionals spread across 10 academic departments.

Vrije Universiteit Amsterdam
Vrije Universiteit Amsterdam stands for values-driven education and research. We are open-minded experts with the ability to think freely - a broader mind. Maintaining an entrepreneurial perspective and concentrating on diversity, significance and humanity, we work on sustainable solutions with social impact. By joining forces, across the boundaries of disciplines, we work towards a better world for people and planet. Together we create a safe and respectful working and study climate, and an inspiring environment for education and research. Learn more about our codes of conduct

We are located on one physical campus, in the heart of Amsterdam's Zuidas business district, with excellent location and accessibility.  Over 6,150 staff work at the VU and over 31,000 students attend academic education.

Diversity
Diversity is the driving force of VU Amsterdam. VU wants to be accessible and receptive to diversity in disciplines, cultures, ideas, nationalities, beliefs, preferences and worldviews. We believe that trust, respect, interest and differences lead to new insights and innovation, to sharpness and clarity, to excellence and a broader understanding.

We stand for an inclusive community and believe that diversity and internationalisation contribute to the quality of education, research and our services.

Therefore, we are always searching for people whose backgrounds and experience contribute to the diversity of the VU community.

Questions about the vacancy?

Please contact

Dr. Shujian Yu / Dr. Filip Ilievski

Assistant Professor

Interested

Are you interested in this position? Then please submit an application by May 19th, 2025 with the following documents: 

  1. A CV that describes relevant experience / expertise for the position
  2. MSc certificate (if already available) and list of (current) MSc grades
  3. Cover letter outlining why you are interested in this position, including the name and email of two people we can contact for a reference
  4. Prior publications, if any

Applications received by e-mail will not be considered.

Acquisition in response to this advertisement is not appreciated.

Questions about the vacancy?

Please contact

Dr. Shujian Yu / Dr. Filip Ilievski

Assistant Professor

Application procedure

Other interesting vacancies

Don't miss out on your dream job at VU Amsterdam

This website uses cookies

We, and third parties, use cookies on our website. We use cookies to ensure that our website functions properly, to store your preferences, to gain insight into visitor behavior, but also for marketing and social media purposes (showing personalized advertisements). By clicking 'Accept', you agree to the use of all cookies. In our Cookie Statement. you can read more about the cookies we use and save or change your preferences. By clicking 'Refuse' you only agree to the use of functional cookies.