AI Agent Vector Memory System

AI Agent Vector Memory System

Overview

The AI Agent includes a semantic memory system that tracks conversations, actions, and context across sessions using vector embeddings for intelligent recall.

Architecture

Components

1. VectorStore (vector-store.js)
   • Low-level vector operations
   • Embedding generation (128D character-based)
   • Cosine similarity search
   • Persistence and pruning
2. AgentMemory (memory.js)
   • High-level memory manager
   • Conversation tracking
   • Action logging
   • Context storage
   • Session management
3. Memory Tools (integrated in ai-agent.js)
   • recall_context - Search memory semantically
   • remember_decision - Store important decisions
   • get_file_history - View file edit history
   • get_context_summary - Session overview

Memory Types

1. Conversations

• User and assistant messages
• Importance: 0.6 (user), 0.5 (assistant)
• Searchable by content
• Maintains conversation buffer (last 20 turns)

2. Actions

• File edits (write operations)
• Command executions
• Importance: 0.7
• Tracks files and outcomes

3. Context

• User preferences
• Project decisions
• Key information
• Importance: configurable (0-1)

Storage

ai-agent-simple/memory/
├── vectors.json      # Vector embeddings and metadata
├── sessions/         # Session-specific data (future)
└── backups/          # Automatic backups before pruning

Embedding Strategy

Character-Based Embeddings (128D)

• Frequency-based character distribution
• Normalized to unit vectors
• Consistent with search indexer
• Fast and deterministic
• Upgradeable to sentence-transformers

// Example embedding
const text = "Edit the about page";
const embedding = await vectorStore.embed(text);
// => [0.23, 0.15, ..., 0.08] (128 dimensions)

Performance

• Search: <100ms for 1000 entries
• Storage: ~1KB per entry
• Pruning: Automatic at maxEntries
• Persistence: Auto-save after each operation

Usage

Automatic Memory

The agent automatically stores:

• Every conversation turn
• File write operations
• Command executions

Manual Memory Tools

Recall Context

// Agent uses internally
recall_context({
  query: "previous changes to homepage",
  type: "action",
  limit: 5
})

Remember Decision

// Agent stores decisions
remember_decision({
  key: "site_structure",
  value: "Using Jekyll with GitHub Pages",
  importance: 0.8
})

File History

// Check what was done to a file
get_file_history({
  filepath: "index.html",
  limit: 5
})

Context Summary

// Get session overview
get_context_summary()
// Returns: session ID, recent actions, stored context, conversation turns

Configuration

const memory = new AgentMemory({
  storagePath: './memory/vectors.json',  // Storage location
  maxEntries: 1000,                       // Maximum entries before pruning
  maxBufferSize: 20,                      // Conversation buffer size
  sessionId: 'custom-session-id'          // Optional session ID
});

Pruning Strategy

When entries exceed maxEntries:

1. Create automatic backup
2. Sort by importance and recency
3. Keep high-importance entries
4. Ensure at least 1 entry per session
5. Maintain recent context

Memory Statistics

const stats = await memory.getStats();
// {
//   totalEntries: 847,
//   maxEntries: 1000,
//   types: { conversation: 423, action: 198, context: 226 },
//   sessions: 12,
//   currentSession: "session_1733072400000_abc123",
//   conversationBufferSize: 14
// }

Export/Import

// Export for backup
const data = await memory.export();
await fs.writeFile('backup.json', JSON.stringify(data));

// Import from backup
const data = JSON.parse(await fs.readFile('backup.json'));
await memory.import(data);

Session Management

Current Session

const context = await memory.getSessionContext();
// { sessionId, conversationBuffer, bufferSize, startTime }

Clear Session

await memory.clearSession();
// Removes session-specific memories, starts new session

Decision History

const history = await memory.getDecisionHistory('site_structure', 5);
// Returns: historical values, timestamps, importance

Smart Recall

const results = await memory.smartRecall("homepage changes", { k: 10 });
// {
//   query: "homepage changes",
//   results: [...],
//   grouped: { conversation: [...], action: [...], context: [...] },
//   sessionContext: {...},
//   totalRelevant: 8
// }

Future Enhancements

Planned Features

1. Advanced Embeddings: Sentence-transformers via API
2. Session Persistence: Save/restore full sessions
3. Memory Graphs: Relationship tracking between memories
4. Importance Decay: Auto-adjust importance over time
5. Selective Recall: Filter by date ranges, file patterns
6. Memory Insights: Pattern detection and suggestions

Upgrade Path

The embedding function is abstracted and can be replaced:

async embed(text) {
  // Future: Call sentence-transformer API
  // Current: Character-based (fast, local)
}

API Reference

See memory.js for full method documentation:

• storeConversation(role, content)
• storeAction(action, files, outcome)
• storeContext(key, value, importance)
• recall(query, options)
• getFileHistory(filepath, limit)
• getContextSummary()
• smartRecall(query, options)
• export() / import(data)

Best Practices

1. High Importance: Use 0.8-1.0 for critical decisions
2. Search Queries: Be specific for better results
3. Pruning: Run periodically for large datasets
4. Backups: Export before major changes
5. Sessions: Clear sessions when starting new topics

Troubleshooting

No Results from Recall

• Query too specific
• Increase limit parameter
• Lower minScore threshold
• Check memory statistics

Memory Growing Too Large

• Reduce maxEntries
• Run prune() manually
• Adjust importance scores
• Clear old sessions

Slow Search

• Check entry count (stats)
• Prune to <1000 entries
• Consider batch operations

Examples

Example 1: Remembering User Preferences

await memory.storeContext(
  'preferred_style',
  'Minimalist design with dark mode',
  0.9
);

// Later...
const prefs = await memory.recall('user style preference', { type: 'context' });

Example 2: Tracking File Changes

// Automatic on write_file
await memory.storeAction('write', ['about.html'], 'Updated bio section');

// Retrieve history
const history = await memory.getFileHistory('about.html');
// [{ action: 'write', outcome: 'Updated bio section', timestamp: ... }]

Example 3: Conversation Context

// Automatic on each message
await memory.storeConversation('user', 'Make the homepage more modern');
await memory.storeConversation('assistant', 'I updated index.html with...');

// Find similar past conversations
const similar = await memory.findSimilarConversations('homepage changes', 3);

Performance Metrics

• Embedding: ~1ms per text
• Storage: ~10ms per entry
• Search: ~50ms for 500 entries
• Prune: ~200ms for 1000 entries
• Memory Usage: ~1MB per 1000 entries

License

Part of the AI Agent Simple project.