Read worthy article and papers by Anthropic (already ~4 months old):

Tracing the thoughts of a large language mode
https://www.anthropic.com/research/tracing-thoughts-language-model

Circuit Tracing: Revealing Computational Graphs in Language Models
https://transformer-circuits.pub/2025/attribution-graphs/methods.html

On the Biology of a Large Language Model
https://transformer-circuits.pub/2025/attribution-graphs/biology.html

In context of Tummi the question might be, if we can extract a knowledge graph from LLMs into an expert system and vice versa?