Memory 2.0: Building an AI Twin for Our Memories

We live in a digital age, communicating online, utilizing websites and applications, and increasingly interacting with AI agents. Yet, we find ourselves repeatedly entering personal information, responding to similar questions, and recalling details from past conversations. This not only burdens and tires our minds (increasing cognitive load) but also hinders our ability to use technology efficiently. Chinese researchers have developed a framework called Second Me for managing personal memory. Let’s explore how it functions.

Current solutions, such as browser autofill or Single Sign-On (SSO) systems, offer limited assistance. They serve as static data storage without understanding context or adapting to user needs. Consequently, users still have to monitor, verify, and often manage this information manually. With the emergence of large language models (LLMs), there’s an opportunity for a fundamentally new AI-based approach to personal memory management. This is precisely what the authors propose in their article «AI-native Memory 2.0: Second Me.»

The primary objective of this work is to develop the Second Me framework: a sophisticated system for “offloading” and managing personal memory. This system is envisioned to function not merely as a repository but as a dynamic AI-driven mediator in user interactions with the external world, including other people, services, and AI systems.

Key tasks for Second Me include:

Storing and organizing user knowledge in a structured format by leveraging LLM capabilities;

Dynamically utilizing this knowledge for automated response generation, form autofilling, and context maintenance in dialogues;

Reducing cognitive burden and simplifying user interactions with digital systems;

Providing context to enrich user queries directed at external systems (such as expert AIs) with pertinent information about the user;

Creating and testing a fully automated process (pipeline) for developing and refining personal models based on user data.

Essentially, the authors aim to create a user’s «second self» in the digital realm, absorbing the routine aspects of utilizing personal information and proactively operating based on the user’s context and needs.

To implement Second Me, the authors build upon their previous concept of Large Personal Model (LPM) 1.0 and introduce an enhanced three-tier hybrid architecture:

L0: Raw Data Layer: Unstructured user data (documents, logs, etc.) that can use approaches like RAG (Retrieval-Augmented Generation).

L1: Natural Language Memory Layer: Structured or semi-structured data in natural language (such as a brief biography, list of preferences, key facts).

L2: AI-Native Memory Layer: Knowledge assimilated and organized directly within the LLM parameters. This model (L2) serves as the core of the system.

Key improvements in Second Me compared to LPM 1.0 include:

Enhanced layer integration: L0 and L1 provide more context to the L2 model.

New role for L2: The L2 model now acts less as an executor and more as a coordinator (orchestrator). It manages interactions with external expert models and resources, always acting from the user’s perspective.

Fully automated training process: The authors have developed a pipeline involving:

Data generation: Automatically creating training data from raw user data using LLM (including Multi-agent and Chain-of-Thought (CoT) strategies);

Data selection: Multi-tier filtering to choose high-quality examples;

Training: Employing Parameter-Efficient Fine-Tuning (PEFT) for effective personalization of the base LLM (Qwen2.5-7B-Instruct), followed by Direct Preference Optimization (DPO) for fine-tuning based on user preferences;

Evaluation: Automated quality assessment of the model using LLM critics (judges) through specially designed metrics and tasks.

Particular attention has been given to Chain-of-Thought (CoT) strategies. Three strategies (Weak, Multi-step, Strong) were compared to improve the model’s reasoning ability and the quality of its responses. The impact of DPO on enhancing the model’s alignment with user preferences was also examined.

To assess model effectiveness, three critical tasks were devised:

Memory Q&A: Evaluating the ability to retrieve and utilize information from the user’s memory (both for the user and when presenting the user to others);

Context Enhancement: Assessing the model’s ability to supplement a user’s query to an external service with necessary details from their memory;

Context Critic: Evaluating the model’s capability to adjust interactions with external agents, considering the user’s preferences and context.

Data style matters: The use of data generated in a «Strong CoT» style (with clear reasoning and response structure) significantly enhances the model’s performance across all tasks compared to weaker CoT variants;

DPO improves personalization: Applying Direct Preference Optimization (DPO) after Supervised Fine-Tuning (SFT) noticeably enhances quality, with the model better reflecting user preferences and effectively utilizing relevant information from their records (evident in specific examples);

Automated evaluation has limitations: While automated assessments using LLM allow for rapid model evaluations, they do not always accurately reflect real quality; for instance, a critic might prefer longer responses.

Overall, the findings confirm that the proposed approach is effective and that the automated pipeline is useful for creating personal AI memory assistants.

The article presents Second Me as a promising framework for developing next-generation personal AI systems. These systems could act as a «second self» for users, augmenting their cognitive functions and streamlining their interactions with the digital world.

However, the current approach is based on single-step interactions. For further advancements, a more complex generation of multi-step dialogues is necessary. There are also limitations regarding automated evaluations, along with the need to gather extensive feedback from actual users. Additionally, the current work focuses on text, but for a comprehensive coverage of human experience, it is crucial to incorporate other modalities (images, audio, etc.).

Nevertheless, the work on «AI-native Memory 2.0: Second Me» marks a significant step toward the creation of truly personalized and beneficial AI assistants. The proposed hybrid architecture and automated training process appear promising. The open-source nature of the project is a significant advantage for further development and adaptation by the community.

***

If you are interested in AI, subscribe to my Telegram channel — where I regularly share insights about AI implementation in business, launching AI startups, and explain how all these AI wonders work.