# Token-Aware Dynamic Lorebook

#### **Gemini wrote most of this document. I'm too lazy to write docs.**  

This is forked from [Tydorius's script](https://janitorai.com/characters/1411c045-6bc2-434e-b8f6-42998e8c74fc_character-token-count-conscious-lorebook-template) and reworked for my use cases.

<details>

<summary><strong>THE DYNAMIC LOREBOOK CODE</strong></summary>

```javascript
/**
 * ============================================================================
 * ENGINE: NOVATORE'S DYNAMIC LOREBOOK
 * ============================================================================
* Details here: novatore.gitbook.io/novatore-docs 
* CORE MECHANICS:
 * 1. Hybrid Routing: Scenario lore prepends to the top. Personality lore uses 
 * targeted Regex injection based on `character_name` and `xml_gate`.
 * 2. Universal Personality: If character_name == 'universal', prepends to top.
 * 3. Dynamic Scrubbing: Instantly cleans <dynamic_memory> and standard gates.
 * 4. Lazy Writer Bridge: Array-driven location prediction.
 * ============================================================================
 */

// ==========================================
// 1. TOP-LEVEL ENGINE CONFIGURATION
// ==========================================
const ENGINE_CONFIG = {
    TARGET_BUDGET: 5300,
    STRETCH_CAP: 6100,
    LOCATION_TRIGGERS: [ // These are example entries, pls change them.
        { id: "hq", keywords: ["hq", "headquarters", "corpo plaza", "maxtac", "precinct", "station", "armory", "briefing", "base"] },
        { id: "penthouse", keywords: ["penthouse", "charter hill", "safehouse", "apartment"] },
        { id: "pacifica", keywords: ["pacifica", "construction", "resort", "coast", "project", "combat zone"] }
    ]
};

// ==========================================
// 2. DATABASE ARRAYS
// ==========================================
// Replace/insert these arrays with your lorebook entries.
const personalityLoreDatabase = [];
const worldLoreDatabase = [];
const castLoreDatabase = [];
const locationLoreDatabase = [];

// ==========================================
// 3. DATABASE REGISTRY (Routing & Rules)
// ==========================================
const DB_REGISTRY = [
    { name: 'personality', data: personalityLoreDatabase, defaultGate: 'universal_lore', targetField: 'personality', isImmune: true, injectContext: 'personality' },
    { name: 'world', data: worldLoreDatabase, defaultGate: 'world_lore', targetField: 'scenario', isImmune: false, injectContext: 'scenario' },
    { name: 'cast', data: castLoreDatabase, defaultGate: 'cast_lore', targetField: 'scenario', isImmune: false, injectContext: 'scenario' },
    { name: 'location', data: locationLoreDatabase, defaultGate: 'location_lore', targetField: 'scenario', isImmune: false, injectContext: 'scenario' }
];

try {
    // ==========================================
    // 4. DYNAMIC SCRUB & BASELINE COSTING
    // ==========================================
    var baseContexts = {
        personality: context.character.personality || "",
        scenario: context.character.scenario || ""
    };

    // A. Clean internal character injections (Zero-Token Exact String Scrubbing)
    var pDb = DB_REGISTRY.find(function(r) { return r.name === 'personality'; }).data;
    for (var i = 0; i < pDb.length; i++) {
        var entry = pDb[i];
        var tiers = ['full', 'summary', 'bullet'];
        for (var t = 0; t < tiers.length; t++) {
            var txt = entry[tiers[t]] ? entry[tiers[t]].personality : "";
            if (txt) {
                // split.join is a fast, safe global replace for raw text
                baseContexts.personality = baseContexts.personality.split(txt).join("");
            }
        }
    }
    // Clean up residual empty lines left by the string scrubber
    baseContexts.personality = baseContexts.personality.replace(/\n{3,}/g, '\n\n');

    // B. Clean top-level prepended gates (Scenario & Universal)
    var topGates = ['universal_lore', 'world_lore', 'cast_lore', 'location_lore'];
    for (var g = 0; g < topGates.length; g++) {
        var scrubRegex = new RegExp("<" + topGates[g] + ">[\\s\\S]*?<\\/" + topGates[g] + ">\\n*", "gi");
        baseContexts.personality = baseContexts.personality.replace(scrubRegex, "");
        baseContexts.scenario = baseContexts.scenario.replace(scrubRegex, "");
    }

    baseContexts.personality = baseContexts.personality.trim();
    baseContexts.scenario = baseContexts.scenario.trim();

    function estimateTokens(text) { return text ? Math.ceil(text.length / 4) : 0; }
    var currentTokens = estimateTokens(baseContexts.personality) + estimateTokens(baseContexts.scenario);

    // ==========================================
    // 5. STATE RESOLUTION & LAZY WRITER BRIDGE
    // ==========================================
    var botLastMessageStr = "";
    var botHistoryStr = ""; 
    
    var rawUserMessage = (context && context.chat && context.chat.last_message) ? String(context.chat.last_message) : "";
    var userLastMessageStr = rawUserMessage.toLowerCase();

    // RESTORED: The actual working JanitorAI API path
    if (context && context.chat && context.chat.last_messages) {
        var lm = context.chat.last_messages;
        var validBots = [];
        for (var i = 0; i < lm.length; i++) {
            if (lm[i] && lm[i].is_bot === true && typeof lm[i].message === "string") {
                validBots.push(lm[i].message);
            }
        }
        if (validBots.length > 0) {
            botLastMessageStr = validBots[validBots.length - 1];
            botHistoryStr = validBots.slice(-3).join(" ");
        }
    }

    function detectLocation(text) {
        if (!text) return "neutral";
        text = text.toLowerCase();
        for (var i = 0; i < ENGINE_CONFIG.LOCATION_TRIGGERS.length; i++) {
            var loc = ENGINE_CONFIG.LOCATION_TRIGGERS[i];
            for (var k = 0; k < loc.keywords.length; k++) {
                if (text.indexOf(loc.keywords[k]) !== -1) return loc.id;
            }
        }
        return "neutral";
    }

    // SLUGLINE PARSER: Checks ONLY the most recent message
    var botLocation = "neutral";
    var locMatch = botLastMessageStr.match(/Location\s*[:|]\s*([^|\n]+)/i);
    if (locMatch) botLocation = detectLocation(locMatch[1]);

    var userLocation = detectLocation(userLastMessageStr);
    var targetLocation = (userLocation !== "neutral") ? userLocation : botLocation;
    
    // THE BRIDGE: Uses the full 3-turn bot history + user's current message
    var searchTarget = (botHistoryStr + " " + rawUserMessage).replace(/[`*]/g, '').toLowerCase();

    function countMentions(keywords, text) {
        var count = 0;
        if (!keywords) return count;
        for (var i = 0; i < keywords.length; i++) {
            if (text.indexOf(keywords[i].toLowerCase()) !== -1) count++;
        }
        return count;
    }

    // ==========================================
    // 6. CANDIDATE POOLING & BASE COSTING
    // ==========================================
    var allCandidates = [];

    for (var d = 0; d < DB_REGISTRY.length; d++) {
        var registryItem = DB_REGISTRY[d];
        var db = registryItem.data;

        for (var i = 0; i < db.length; i++) {
            var entry = db[i];
            if (!entry) continue;

            var mentions = countMentions(entry.keywords, searchTarget);
            var score = entry.importance || 0;
            if (mentions > 0) score += (mentions * 100);

            var isCritical = false;
            
            // Location critical check
            if (registryItem.name === 'location' && targetLocation !== "neutral") {
                if (entry.id.indexOf(targetLocation) !== -1 || (entry.keywords && entry.keywords.indexOf(targetLocation) !== -1)) {
                    score += 2000;
                    isCritical = true;
                }
            }
            
            // RESTORED: Any actively mentioned cast/lore entry gets flagged as critical
            if (mentions > 0) {
                isCritical = true;
            }

            var tokensBullet = estimateTokens(entry.bullet[registryItem.targetField] || "");
            var tokensSummary = estimateTokens(entry.summary[registryItem.targetField] || "");
            var tokensFull = estimateTokens(entry.full[registryItem.targetField] || "");

            currentTokens += tokensBullet;

            allCandidates.push({
                registryItem: registryItem,
                entry: entry,
                score: score,
                isCritical: isCritical,
                tier: 'bullet',
                tokens: { bullet: tokensBullet, summary: tokensSummary, full: tokensFull }
            });
        }
    }

    // ==========================================
    // 7. THE BULLET PRUNE & ELASTIC UPGRADE
    // ==========================================
    if (currentTokens > ENGINE_CONFIG.TARGET_BUDGET) {
        allCandidates.sort(function(a, b) { return a.score - b.score; });
        for (var p = 0; p < allCandidates.length; p++) {
            var pItem = allCandidates[p];
            if (pItem.registryItem.isImmune) continue; 
            if (currentTokens > ENGINE_CONFIG.TARGET_BUDGET) {
                currentTokens -= pItem.tokens.bullet;
                pItem.tier = 'none';
            } else break;
        }
    }

    var upgradeCandidates = allCandidates.filter(function(c) { return c.tier === 'bullet' && c.score > c.entry.importance; });
    upgradeCandidates.sort(function(a, b) { return b.score - a.score; });

    for (var u = 0; u < upgradeCandidates.length; u++) {
        var uItem = upgradeCandidates[u];
        var costToSummary = uItem.tokens.summary - uItem.tokens.bullet;
        var costToFull = uItem.tokens.full - uItem.tokens.bullet;

        // RESTORED: isCritical bypasses TARGET_BUDGET and accesses STRETCH_CAP
        if (currentTokens + costToFull <= ENGINE_CONFIG.TARGET_BUDGET || (uItem.isCritical && currentTokens + costToFull <= ENGINE_CONFIG.STRETCH_CAP)) {
            currentTokens += costToFull; uItem.tier = 'full';
        } else if (currentTokens + costToSummary <= ENGINE_CONFIG.TARGET_BUDGET || (uItem.isCritical && currentTokens + costToSummary <= ENGINE_CONFIG.STRETCH_CAP)) {
            currentTokens += costToSummary; uItem.tier = 'summary';
        }
    }

    // ==========================================
    // 8. HYBRID PAYLOAD ROUTING
    // ==========================================
    var scenarioInjects = { world_lore: [], cast_lore: [], location_lore: [] };
    var personalityInjects = { universal: [], characters: {} };

    for (var c = 0; c < allCandidates.length; c++) {
        var finalItem = allCandidates[c];
        if (finalItem.tier === 'none') continue;
        
        var content = finalItem.entry[finalItem.tier][finalItem.registryItem.targetField];
        if (!content) continue;

        // SCENARIO ROUTING (Standard Prepend)
        if (finalItem.registryItem.injectContext === 'scenario') {
            var sGate = finalItem.registryItem.defaultGate;
            scenarioInjects[sGate].push(content);
        } 
        // PERSONALITY ROUTING (Regex vs Universal)
        else {
            var charName = finalItem.entry.character_name || "universal";
            if (charName === "universal") {
                personalityInjects.universal.push(content);
            } else {
                // If xml_gate is blank/missing, flag it with a fallback string
                var pGate = finalItem.entry.xml_gate || "CHAR_FALLBACK";
                if (!personalityInjects.characters[charName]) personalityInjects.characters[charName] = {};
                if (!personalityInjects.characters[charName][pGate]) personalityInjects.characters[charName][pGate] = [];
                personalityInjects.characters[charName][pGate].push(content);
            }
        }
    }

    // ==========================================
    // 9. INJECTION EXECUTION & FAILSAFES
    // ==========================================
    
    // A. Inject Scenario Lore (Top-Level Prepend)
    var finalScenarioBlock = "";
    for (var key in scenarioInjects) {
        if (scenarioInjects[key].length > 0) {
            finalScenarioBlock += "<" + key + ">\n" + scenarioInjects[key].join("\n") + "\n</" + key + ">\n\n";
        }
    }
    context.character.scenario = finalScenarioBlock + baseContexts.scenario;

    // B. Inject Personality Lore (Internal Raw Regex with Fallbacks)
    var finalPersonality = baseContexts.personality;
    for (var cName in personalityInjects.characters) {
        var charRegex = new RegExp("<" + cName + ">([\\s\\S]*?)</" + cName + ">", "i");
        var charMatch = finalPersonality.match(charRegex);
        
        if (charMatch) {
            var charBlock = charMatch[0];
            var gateMap = personalityInjects.characters[cName];
            
            for (var targetGate in gateMap) {
                var payloadText = gateMap[targetGate].join("\n");
                var injected = false;
                
                // Attempt to inject raw text strictly inside the child gate (e.g., </cam_lore>)
                if (targetGate !== "CHAR_FALLBACK") {
                    var gateRegex = new RegExp("</" + targetGate + ">", "i");
                    if (charBlock.match(gateRegex)) {
                        charBlock = charBlock.replace(gateRegex, "\n" + payloadText + "\n</" + targetGate + ">");
                        injected = true;
                    }
                }
                
                // FAILSAFE: If xml_gate was missing, or tag wasn't found in text
                // Inject raw text at the very bottom, right above the character's closing tag (</cam>)
                if (!injected) {
                    var charEndRegex = new RegExp("</" + cName + ">", "i");
                    charBlock = charBlock.replace(charEndRegex, "\n" + payloadText + "\n</" + cName + ">");
                }
            }
            finalPersonality = finalPersonality.replace(charRegex, charBlock);
        }
    }

    // C. Inject Universal Personality (Top-Level Prepend)
    if (personalityInjects.universal.length > 0) {
        var uniBlock = "<universal_lore>\n" + personalityInjects.universal.join("\n") + "\n</universal_lore>\n\n";
        finalPersonality = uniBlock + finalPersonality;
    }
    context.character.personality = finalPersonality;
    
} catch(e) {
    // Failsafe execution block to ensure API doesn't crash on syntax error
    // Optionally append error to personality for debugging:
    context.character.personality = (context.character.personality || "") + "\n\n[CRITICAL ENGINE ERROR: " + e.message + "]\n\n";
}
```

</details>

Some examples of the script in practice, but heavily modified:

* [Used as an engine to juggle/reorder multiple character context](https://janitorai.com/scripts/f744c069-9273-44dc-b823-163e29044b3b)
* [Used as an engine to track dual identities](https://janitorai.com/scripts/2d83f962-145e-4a8c-abaa-fb9faf491ed1)
* [Just a plain ol' lorebook](https://janitorai.com/scripts/6854f263-687d-4ccd-abd0-e284d9f4ceff)

NOTE: When debugging on JAI, check the **CHANGES** tab, not the **CONSOLE**.

***

## Concept & Configuration

This engine is an automated memory manager designed to solve token bloat, context overflow, and LLM hallucination in long-term roleplays.

#### The Core Mechanics: Elastic Memory

Language models have strict memory limits. Dumping massive lorebooks into the context window causes the AI to forget the scene and break character. The script solves this by dynamically scaling text to fit your exact budget.

You must write three sizes for every lore entry: `full`, `summary`, and a 1-sentence `bullet`.

* The Baseline: When triggered, the engine loads only the 1-sentence `bullets`. This establishes a cheap, unbreakable baseline of truth to prevent the AI from hallucinating facts.
* The Upgrade Phase: If you have token budget remaining, the engine automatically upgrades the highest-priority entries into their `summary` or `full` sizes.
* The Prune (Survival Mode): If a scene triggers too much data and blows the budget, the engine deletes the lowest-priority bullets to save memory and prevent a crash.
* Personality Immunity: Personality entries are immune to pruning. A character's core psychology will always survive the cut.

***

#### The Architecture: Why This Script is Different

1\. Surgical Personality Injection

Standard lorebooks dump character history into general world memory, causing the AI to read its own trauma or habits like a Wikipedia article. The script splits the data. Environment facts (World, Cast, Location) are sent to general memory, but Personality lore is injected directly inside the character's static XML tags. This forces the LLM to internalize the dynamic lore as its own core identity.

2\. Predictive Location Tracking

Standard lorebooks are reactive—they wait for the bot to arrive at a location, causing the AI to hallucinate the environment for one turn before the lore loads. This script uses a centralized `LOCATION_TRIGGERS` array to read the user's chat and anticipate movement. If the user types, *"Let's head to the armory,"* the engine catches the keyword and pre-loads the HQ layout *before* the bot generates its reply.

***

### Setting Up Your Token Budget

At the very top of the Engine code, you will see a section called Top-Level Engine Configuration. This is the most important part of the setup.

```javascript
const ENGINE_CONFIG = {
    TARGET_BUDGET: 5300,
    STRETCH_CAP: 6100,
// ...
```

These two numbers dictate exactly how much data the engine is allowed to inject into the bot's memory on any given turn.

#### 1. How is the Budget Calculated?

The engine calculates its current "weight" by adding two things together:

* Your Base Character Sheet: The permanent tokens taken up by your bot's static definition: global scenario and personality boxes.
* The Dynamic Injections: The estimated token weight of the specific lore entries the engine is trying to load for the current turn.

*(Note: The engine does not include the active chat history in this calculation. This budget is strictly for the character sheet + the injected lore).*

#### 2. The `TARGET_BUDGET` (The Safe Zone)

This is your ideal maximum token limit.

* When the engine wakes up, it gathers all relevant lore and initially loads them as bullets (the smallest, cheapest version).
* If the total token count (Base Sheet + Bullets) is *under* the `TARGET_BUDGET`, the engine starts upgrading the most important lore entries into their `summary` or `full` versions until it hits this limit.
* If the total token count is *over* the `TARGET_BUDGET`, the engine starts pruning (deleting) the least important bullets entirely to save memory.

#### 3. The `STRETCH_CAP` (The Emergency Override)

Sometimes, the user walks into a specific location (like the "MaxTac HQ"), and the engine *must* load the full layout of that location so the AI doesn't hallucinate the environment.

* If a piece of lore is flagged as Location Critical, the engine is allowed to ignore the `TARGET_BUDGET` and upgrade that specific entry to its `full` size.
* However, it will only do this if the total tokens stay under the `STRETCH_CAP`. This is your absolute hard limit to prevent the bot from crashing due to context overflow.

> Best Practice for Creators: Look at your static Character Sheet's token size. Set your `TARGET_BUDGET` about 1000–2500 tokens higher than your base sheet to give the engine plenty of room to breathe and inject high-quality lore!

***

### Tracking Locations (The Header System)

To load the correct environment lore (like the layout of a specific building), the engine needs to know where the characters currently are. It does this by reading the Header (or Slugline) of the bot's messages, as well as predicting movement based on the user's chat.

Right below the budget in the `ENGINE_CONFIG`, you will see the Location Triggers array:

```json
    LOCATION_TRIGGERS: [
        { id: "hq", keywords: ["hq", "headquarters", "corpo plaza", "maxtac", "precinct", "station", "armory", "briefing", "base"] },
        { id: "penthouse", keywords: ["penthouse", "charter hill", "safehouse", "apartment"] },
        { id: "pacifica", keywords: ["pacifica", "construction", "resort", "coast", "project", "combat zone"] }
    ]
```

#### How It Works:

1. The Bot's Header: The engine scans the bot's last message for a specific format: `Location: [Somewhere]`.
   * *Example:* If the bot writes `Location: MaxTac Headquarters`, the engine sees the word "Headquarters", matches it to the `"hq"` ID in your config, and immediately loads your HQ location lore.
2. The User Prediction (Lazy Writer Bridge): The engine also reads the *user's* last reply. If your bot is currently at the Penthouse, but the user types, *"Let's drive down to the Pacifica construction site,"* the engine detects "Pacifica" and "construction." It preemptively switches the target location to `"pacifica"` so the bot has the correct lore ready *before* it even replies!

#### How to Set This Up for Your Bot:

* Teach Your Bot: In your bot's system prompt or first message, ensure it is formatted to always include a header like `Location: [Current Place]` at the top of its replies.
* Customize Your Triggers: You can change the `LOCATION_TRIGGERS` in the code to match your specific world.
  * `id`: The internal name the engine uses (keep this lowercase, no spaces).
  * `keywords`: A list of words the engine will look for in the Header or the user's chat to trigger this location.

*(Note: When you write your Location Database JSONs later, you will use the `id` you set here to link the lore to the location!)*

***

## JSON Database Guide

You don't need to know how to code to use this system. You just need to know how to organize your bot's lore into simple data blocks (JSONs).

The engine uses four distinct databases to manage your bot's memory: World, Cast, Location, and Personality.

This guide will show you exactly how to format your entries for each one.

***

### The Core Elements

Every entry you write, regardless of the database, uses a "Tiered" system. Because AI has a limited memory budget, the engine automatically shrinks or expands your lore based on how much space is left. You must provide three versions of your lore for every entry:

* `full`: The complete, highly detailed description.
* `summary`: A condensed version of the details.
* `bullet`: The bare-minimum, single-sentence fact.

***

### 1. The Scenario Databases (World, Cast, Location)

These three databases handle the environment around your character. The engine automatically takes whatever is triggered here and neatly stacks it at the very top of the AI's memory.

Use this format for `worldLoreDatabase`, `castLoreDatabase`, and `locationLoreDatabase`:

```json
[
  {
    "id": "militech_corp_brief",
    "keywords": ["militech", "corp", "soldier", "nusa"],
    "importance": 85,
    "full": {
      "scenario": "- [Write your massive, detailed lore block here. Include all the gritty details.]"
    },
    "summary": {
      "scenario": "- [Write a shorter, 1-2 sentence summary here.]"
    },
    "bullet": {
      "scenario": "- [Write a single, strict bullet point here.]"
    }
  }
]
```

#### What do these fields mean?

* `id`: A unique, lowercase name for this entry (use underscores instead of spaces).
* `keywords`: The specific words that will trigger this memory. If the user or the bot types one of these words, the engine wakes this entry up.
* `importance`: A priority score from 1 to 100. If the AI is running out of memory, it will delete entries with lower scores first to save the high-score ones.
* `scenario`: This tells the engine to push the text into the general world-building memory, rather than the character's direct personality.

***

### 2. The Personality Database

This database is special. It handles the internal thoughts, secrets, and behaviors of your characters.

Instead of dumping the text at the top of the memory, the engine acts like a surgeon. It hunts down the specific character in your static definition and slips the dynamic lore right inside their profile.

Use this format for `personalityLoreDatabase`:

```json
[
  {
    "id": "cam_combat_tactics",
    "character_name": "cam",
    "xml_gate": "cam_lore",
    "keywords": ["gun", "shoot", "combat", "fight"],
    "importance": 90,
    "full": {
      "personality": "- [Write Cam's detailed combat behavior here.]"
    },
    "summary": {
      "personality": "- [Write the summary version here.]"
    },
    "bullet": {
      "personality": "- [Write the core combat rule here.]"
    }
  }
]
```

#### The Special Fields:

* `character_name`: The exact name of the character's parent tag in your character sheet (e.g., if you use `<cam>`, write `"cam"`).
* `xml_gate`: The exact name of the child tag where you want the text to go. If you want it inside `</cam_lore>`, write `"cam_lore"`. *(Note: If you forget to put `<cam_lore>` in your character sheet, the engine is smart enough to just drop the text at the bottom of the `<cam>` block safely!)*
* `personality`: This ensures the engine treats this text as core character logic, making it immune to being deleted when memory gets tight.

#### The "Universal" Override

Sometimes, you want a rule to apply to the entire roleplay, not just one specific character. To do this, change the `character_name` to `"universal"`.

```json
[
  {
    "id": "global_injury_logic",
    "character_name": "universal",
    "xml_gate": "", 
    "keywords": ["blood", "wound", "medic", "pain"],
    "importance": 95,
    "full": {
      "personality": "- [Universal rule: Characters must realistically react to pain and bleeding.]"
    },
    "summary": {
      "personality": "- [Shorter universal rule.]"
    },
    "bullet": {
      "personality": "- [Shortest universal rule.]"
    }
  }
]
```

*When you use `"universal"`, you can leave `xml_gate` blank. The engine will automatically pin this rule to the absolute top of the personality memory so the AI never forgets it.*


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://novatore.gitbook.io/novatore-docs/other-resources/token-aware-dynamic-lorebook.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.