MEMEX jr. – eye square’s LLM-based knowledge tool for customer dialogue

Michael Schiessl, CEO of eye square, comments: "I am impressed with how far the combination of vector databases and LLMs carries. The avatar explains some of the findings very well, such as our unique framework model of neurosemiotics and definitions of key terms like attention, experience, or meaning. It does this at least as well as our senior management, given the obvious fact that much of the externalized knowledge is instantaneous. Unlike a human manager, our new tool is available 24/7, beyond time zones, time of day, schedules, and rest periods. This important step frees up time and resources for our management. The result is scientifically very precise, gives reasonable strategic designs, and overall has great creative power. I believe that customers who want a quick look at overarching issues will definitely benefit from this. MEMEX Jr. is also a great platform for our internal knowledge sharing."

Andreas Thoelke, CTO of eye square, adds: "MEMEX jr. perfectly implements our credo of combining science, technology, and art. It is a proof of concept for our AI strategy, which foresees the use of AI primarily in knowledge management, for easier data analysis to accelerate studies and projects, and finally in "semantic eye tracking". The latter is particularly promising as it integrates key areas of our neurosemiotic model, especially semantics and System 0."

"MEMEX jr." System and Function

"MEMEX jr." is the first eye square project primarily concerned with knowledge management, developed by a team of data scientists and artists (Frederik Neitzel, Daniel Döhringer, Moritz Mattern, Dennis Rudolph, and Dr. Julia Nitschke) that had previously also cooperated in eye square’s extensive augmented reality work.

Utilizing either a keyboard or voice-driven interactions, users can seamlessly engage with the system to capture queries that are then run through an extensive knowledge base. The integration of Retrieval Augmented Generation (RAG) ensures unprecedented precision and relevance in information extraction as semantic retrieval, supported by vector databases, refines the accuracy and richness of answers.

These curated responses then go through another level of optimization by being passed through a large language model. Through the use of nuanced prompt engineering, the LLM was customized to assume the persona of Michael Schiessl, ensuring that responses are both personalized and do not stray into speculative territory. This approach addresses knowledge gaps in LLMs and ensures that up-to-date and contextual information is conveyed. To clarify the experience, feedback is synthesized using a text-to-speech model that reproduces the distinct cadence of Michael’s voice. After preparing the audio data, our Unity-driven VR/AR environment recognizes and retrieves these audio files. The result is that Michael’s avatar articulates the responses and provides the user with an engaging and realistic conversational experience.

"MEMEX Jr." Example chat

Question: What is neurosemiotics?

Answer: "Neurosemiotics is eye square’s integrated analytical model for understanding human experience, especially in relation to brands, advertising, and products. It combines theories and insights from neuromarketing, psychology, and social cognition into a descriptive model that combines qualitative and quantitative methods. Neurosemiotics focuses on the central levels of psychological representation, which include perception, feeling, and thinking. Specific methods and measurement techniques such as eye-tracking, electrodermal activity (EDA), and electroencephalography (EEG) are used at each of these levels. The goal is to gain a comprehensive understanding of consumer preferences and market offerings. For those interested in marketing and market research, neurosemiotics offers a conceptually sound and empirically measurable way to describe the impact of brands, campaigns, and products."

Question: What is System 0?

Answer: "System 0, as described by eye square, refers to perceptual-sensory experience, or “real perception”. It is the most basic and fastest type of perceptive system and is often referred to as "sensory perception. It captures purely physical-neurological perception, in which neither cognitive nor implicit/emotional centers are involved. This perceptual model goes beyond implicit and explicit explanations and relies on extensive attentional data collected by eye square over the past decade. The receptive quality of System 0 may contribute to building longer relationships and is valuable for understanding advertising impact."