Understanding Safety through Image-to-Text LLM

The existing safety documentation and standards are replete with descriptive schematics, diagrams, and flow charts that present safety-relevant data. However, the data presented in these figures is often challenging to interpret, and given the fundamental importance of safety, it must be reliable. The objective of this project is to analyse and select appropriate large language models (LLMs) that can interpret figures and extracting useful information, which can then be used as inputs for more complex LLMs. The student will conduct a literature review of key offerings that provide such functionality and perform a comparative study of their usability in an industrial context. The most suitable model identified will be re-evaluated based on a dataset created throughout the project.

Download as PDF


Figure 1: One of the flowcharts from the IEC 61508 standard that should be described in textual form using the developed models

 

Thesis Type:

 

  • Master Thesis / BSc Thesis / Seminar-Project

Goal and Tasks:

  • Conduct a literature review and classification of image-to-text LLMs architectures;
  • Create a safety-specific dataset containing flowcharts from IEC 62061 and ISO 13849-1;
  • Fine-tune the selected LLM on the created dataset and evaluate its performance;
  • Identify and compute use case-specific LLM metrics;

Recommended Prior Knowledge:

  • Basic programming skills, such as Python, C, or C++;
  • Experience or interest in Large Language Models;
  • Interest in Industrial Safety, Functional Safety and Standardization.

Start:

  • a.s.a.p.

Contact: