Reproduce “ctELM: Decoding and Manipulating Embeddings of Clinical Trials with Embedding Language Models” on OpenResearch

About this paper

Brian Ondov, Chia-Hsuan Chang, Yujia Zhou, Mauro Giuffrè, Hua Xu

ctELM introduces an open-source Embedding Language Model framework designed to decode and manipulate text embeddings of clinical trial abstracts. The model generates plausible and coherent hypothetical abstracts from novel embeddings, fooling human experts 44% of the time compared to 2% for previous state-of-the-art methods, and enables controlled modification of clinically meaningful concepts such as subject age and sex.

Key insights

• ELMs can be successfully open-sourced and generalized to complex, sensitive domains like biomedicine, enabling enhanced interpretability and controlled text generation from embeddings.
• Increasing data scale consistently improves ELM performance across tasks, and a single-phase training procedure can be efficient and competitive for large datasets, simplifying ELM development.
• ctELM's generated hypothetical abstracts are highly plausible and coherent, effectively deceiving human experts in nearly half of the evaluations, significantly surpassing previous text inversion methods.
• The model allows for controlled semantic manipulation, demonstrating targeted changes in generated text, such as subject demographics (sex, age), by navigating the embedding space along Concept Activation Vectors.

Summary by alphaXiv.