Protect Pipeline (fetch-new-docs)

Overview

The protect cron job selects documents that need a new proof anchored. Selection runs a MongoDB aggregation: documents newer than START_TIME, without an existing proof of the target type, joined via documentRef.

Document timestamp field

Canonical field: createdDate.

Source	Field written	Picked by protect?
Legacy backend / IP Magic Box manual	createdDate (+ legacy created)	Yes (since 1.0.0)
Neo-backend / Tower v2 agent ingest	createdDate only	Yes (since 1.0.0)
Very old docs	created only (rare)	No — migrate if needed

Before 1.0.0, the pipeline matched created (Mongoose legacy), so neo-ingested documents were invisible to protect (admin showed 0/0).

Aggregation pipeline

Stages in fetch-new-docs.ts:

1. $match — createdDate: { $gt: new Date(startTime) }
2. $project — hash, denomination, createdDate
3. $lookup — proofs collection where documentRef = _id and type = ps
4. $match — _pa array empty (no proof yet)

Configuration

Env var	Role
START_TIME	ISO timestamp — only documents after this date are eligible
DOCS_PER_RUN	Optional cap on documents processed per protect run

Key files

• src/database/requests/fetch-new-docs.ts — pipeline definition
• src/database/requests/fetch-new-docs.spec.ts — unit tests on pipeline stages
• src/app.service.ts — protect job orchestration

Related

• Backend createdDate proxy for legacy API consumers
• Neo-backend writes createdDate on agent deposits