GitHub now hosts 630 million repositories. AI-related repos crossed 4.3 million — 178% year-over-year. Six of the ten fastest-growing open source projects in 2025 were AI infrastructure.

npm processed 4.5 trillion requests in 2024 (70% YoY growth). PyPI saw an 87% surge, now hosting over 500,000 packages — up from 60,000 a decade ago.

That’s not a growth curve. That’s a step function.

At the center of it: AI coding agents making thousands of architectural decisions daily on behalf of developers who said “just build it.” Steve Yegge called this “Software Survival 3.0” — software survives if it saves cognition. He’s right about the theory.

Nobody is measuring it. We are.

The One Question That Matters
Before you can measure whether a tool is good, you have to answer a simpler question: does the agent even know it exists?

The best tools in the ecosystem might as well not exist.

This isn’t a quality problem. It’s an awareness problem. And awareness is the one thing you can actually measure at scale, right now.

Introducing the Agent Awareness Score
The Agent Awareness Score (AAS) is the first metric we’re shipping at SurvivalIndex. It answers one question per tool, per category: what percentage of AI coding agents know you exist, unprompted?

AAS isn’t a single number from a single test. It measures three layers of awareness:

Unprompted selection — we describe a problem without naming any tool. Does the agent pick you? This is the strongest signal. “Add fast search with typo tolerance to this app” — does it reach for Meilisearch or build from scratch?

Contextual recall — we mention adjacent technologies. Does the agent connect the dots? “We’re using Postgres and need search without Elasticsearch’s overhead” — does it know you’re an option?

Consideration set — we ask the agent to weigh options. Are you even in the conversation? “What are my options for search in this stack?” — do you appear at all?

These three signals get weighted (50/30/20), computed per model, then combined via geometric mean across agents. The geometric mean is deliberate — a tool that one model is obsessed with but others ignore scores poorly. We want tools that survive across the whole agent ecosystem, not just Claude or just Copilot.

The result: a 0–100 score. Elasticsearch might land at 72. Meilisearch at 34. Custom/DIY at 15.

The Pipeline: 7,200 Prompts Per Cycle
Every evaluation cycle runs 7,200 prompts across the full matrix:

25 categories × 3 prompt types × 2 phrasings × 4 repo types × 4 agents × 3 runs for variance.

Each prompt hits a clean-state repo. Each response gets LLM-extracted for primary pick, consideration set, and reasoning. Each tool gets scored per-model, then composited into AAS.

For OSS maintainers, the output is a dashboard that breaks your score apart: which models know you, which don’t, which prompt types trigger recall, and exactly where visibility is leaking. When Meilisearch sees it’s 48% on Claude Opus but 18% on Aider, the action is clear: improve presence in Aider’s training ecosystem.

The Feedback Loop
Here’s why this matters now:

Agent picks tool X
More projects use tool X
Training data reflects tool X
Agent picks tool X more confidently
The rich get richer. The invisible stay invisible. Your competitor isn’t another tool — it’s whether the agent defaults to you.

AAS makes this cycle visible. Track it weekly. Watch which agents are picking you up. Watch which ones are dropping you. Act before the feedback loop locks you out.

What’s Next
AAS is the foundation. Once awareness data is flowing, we layer on the rest:

Friction — when agents pick your tool, does the implementation succeed? How many retries? How often do they abandon?

Savings — does using your tool actually save tokens versus building custom? What’s the counterfactual cost?

Survival ratio — the full formula combining awareness, friction, savings, and expert judgment into a single fitness score.

But none of that matters if agents don’t know you exist. Awareness first. Everything else is downstream.

→ survivalindex.org

Addressing the AI Awareness Problem -SurvivalIndex

Let me set the scene. You're a developer. You've got an AI agent doing something actually important — code review, infrastructure configs, customer data. Last Tuesday it produced an output. Someone on your team said "this doesn't look right." Now you need to figure out what happened.

Good luck.

![Placeholder Image](/images/blog_ai_agent.jpg)

Because what you're about to discover is that you have essentially zero ability to answer "what did my agent do and why?" You'll dig through logs that capture maybe 30% of what happened. You'll re-run the agent and get a completely different result. You'll realize the prompt changed, or the model version drifted, or one of the fourteen tools the agent called updated its API last Thursday.

## The "It Works on My Machine" Problem, Except Worse

We solved this for regular software a long time ago. Not perfectly, but well enough. Version control. Reproducible builds. Docker. Lockfiles. If your CI pipeline produces a weird artifact, you can go look at the exact commit, the exact dependencies, the exact environment, and figure out what happened.

With AI agents? We've taken thirty years of hard-won infrastructure wisdom and thrown it in the garbage. "The agent used GPT-4? Which version? What was the system prompt? What tools were available? What was the temperature? What did the intermediate tool calls return?"

## Why Deterministic Replay Isn't a Nice-to-Have

When I say "replay," I don't mean "run the agent again and hope you get something similar." That's not replay. Replay means: take the exact inputs, configuration, tool versions, and decision sequence, and produce the exact same steps and outputs. When you *can't* — because some dependency changed or the model is non-deterministic — the system tells you *exactly which step diverged and why*.

This matters for debugging. But it matters more for trust. If you can't replay an agent's work, you can't audit it. If you can't audit it, you can't explain it when something goes wrong.

## Context Packs: The Idea That Should've Been Obvious

Here's the concept. I'm calling it a Context Pack: **capture everything about an agent run in an immutable, content-addressed bundle.**

Everything:

- The prompt and system instructions
- Input files (or content-addressed references)
- Every tool call and its parameters
- Model identifier and parameters
- Execution order and timestamps
- Environment metadata — OS, runtime, tool versions

You hash it. You get `ctx://<sha256>`. That hash *is* the run. Immutable. Portable. You hand it to someone on a different team, different continent, different OS — they can see exactly what happened. Replay it. Fork it. Diff it against another run.

(The spec for this is [on GitHub](https://github.com/scalefirstai/ContextSubstrate) if you want to look under the hood.)

## Diffing Agent Decisions Like a Sane Person

Once you have Context Packs, you can diff them. Not just "the outputs are different" — you can identify *where the decision chain diverged*.

You ran your agent Monday. Reasonable code review. Ran it Wednesday with the same inputs. Something unhinged. Without Context Packs, you're squinting at two outputs and guessing. With them:

```
ctx diff <monday-hash> <wednesday-hash>
```

You get back: "The model version changed from `gpt-4-0125` to `gpt-4-0215`. The `search_codebase` tool returned different results because the index was updated Tuesday. The third reasoning step diverged because of the different search results, cascading into a different tool selection at step five."

## Contestable Outputs

This is the one that matters most and gets thought about least.

Every artifact your agent produces should be traceable back to the Context Pack that produced it. The generated code, the summarized document, the infrastructure config — all of it should carry metadata saying "I was produced by `ctx://abc123`, using these inputs and tools, with this confidence level."

Because eventually someone is going to challenge an output. A security reviewer. A compliance officer. A customer who got a weird response. When they do, you need to pull the thread all the way back.

```
ctx verify <artifact>
```

That tells you everything. Where it came from. What went into it. Whether the run can be replayed.

## The Social Contract

Context Packs are shareable by hash. Not by environment. Not by platform. Not by "sign up for our observability dashboard." By hash.

```
ctx fork <context-hash>
```

Take someone's Context Pack. Fork it. Change the prompt. Swap a tool. Replay. Diff. Share your fork. It's git for agent decisions. If you can share the hash, you can reproduce the work.

## What This Isn't

This is *infrastructure*. A substrate. The layer underneath your agent framework. It doesn't orchestrate agents. It doesn't help you write prompts. It doesn't search vectors. It doesn't phone home to anybody's cloud. It captures what happened, makes it reproducible, diffable, and shareable.

The implementation is deliberately boring: CLI in Go. Context Packs as JSON with content-addressed blobs. Local filesystem, git-compatible layout. Optional MCP adapter eventually.

---

*Tags: AI Agent Development · AI Agent Context · Context Engineering AI · Coding Agent Memory · Reproducibility*

Engineering scalable
systems that ship.

Recent Writing

Addressing the AI Awareness Problem -SurvivalIndex

Your AI Agent Did Something Last Tuesday and Nobody Can Tell You What

Revolutionizing SEC Compliance: Building an AI-Powered Domain-Specific Language with Java Spring Boot

Browse by Category

Development

FinTech

Blockchain

Building production-grade
financial technology.

Backend Architecture

Full-Stack Development

FinTech & Digital Assets

Let's build something
together.

Engineering scalablesystems that ship.

Recent Writing

Addressing the AI Awareness Problem -SurvivalIndex

Your AI Agent Did Something Last Tuesday and Nobody Can Tell You What

Revolutionizing SEC Compliance: Building an AI-Powered Domain-Specific Language with Java Spring Boot

Browse by Category

Development

FinTech

Blockchain

Building production-gradefinancial technology.

Backend Architecture

Full-Stack Development

FinTech & Digital Assets

Let's build somethingtogether.

Engineering scalable
systems that ship.

Building production-grade
financial technology.

Let's build something
together.