feat-document-extract
✨ New Features
Document extract and Tower deposit
Goal: A Tower deposit stays invisible until the file is actually stored. A later deposit of the same file still stores a new copy. Extract reads that copy and writes the business fields on the document.
Solution: Tower resolves its storagePath template and asks neo for a draft. Neo creates the remote file with that key, or the next free -2, -3 suffix, and returns the key. Tower uploads with its own credentials. Confirm makes the document visible. Extract watches every company that has the feature enabled, through one agent token.
Decisions:
- Tower is the first caller. Web is parked. The Box migrates later, on the same pair of calls.
- A draft is not a document in the account. Lists, extract, download and the proof ignore it. It becomes visible only at confirm, once the file is stored.
- Hash idempotence is removed. The same hash creates a new document. While the file uploads, neo looks up other visible documents with that hash and sets
olderDocumentIdto the oldest match. The lookup does not block the draft or the confirm. A match is the same user, or two users who share a company. The index ondocuments.hashis created in Mongo.noDuplicateon the request is the only case that does not create a document: it still looks only inside the company and returns that older document. Tower does not send it. Confirm itself stays idempotent: calling it again returns the live document and does not create a second remote file. - The object key lives on
remotefiles.remote. Neo creates that remote file with the draft and setsfileId.targetstays the storage kind from the extract/upload config (AMAZON_S3,FTP, …).tower: truemarks that Tower owns the object for Box/web download. The document has nostorageKey. Confirm only setsdraftto false. The response still returns the key Tower must upload to. POST /documentscreates an invisible draft and returns the document id and the final object key. The proof does not start. The document does not appear in the account.- The key is the source
storagePathonce Tower has replaced the#tags. No random id.#file_directory/#file_name_with_hashonscan.jpg/8f3a21/INBOXisINBOX/scan_8f3a21.jpg. - Neo is the only writer of
remotefiles. The remote file is the pointer. Astorage-pathattribute is not. - If a document or a draft already holds that key, neo appends
-2,-3, before the extension, and returns that key. An abandoned draft leaves a gap in the numbering. The next deposit still proceeds. - Tower uploads the bytes to the returned key with its own credentials. Push only. Neo does not sign the upload and does not read the bucket.
HeadObjectis not used: on S3 it requiress3:GetObject, and Tower only sendsPutObject. - A failed upload, or a bad storage configuration, leaves the draft invisible. The mail cursor does not advance.
- Confirm makes the document visible. The proof can start. The mail cursor advances.
- If the upload succeeded and confirm failed, Tower calls confirm again and does not send the bytes again.
- Two mails with the same file are two drafts. The second takes the next suffix. Both are uploaded, both become visible, both get a proof.
- A file deleted from the bucket breaks the documents that already point at it. The next deposit takes the next suffix and uploads for real. Neo does not skip the upload, does not reuse the old key, and does not relink the new deposit to the oldest document.
- A script replay passes
noDuplicatefor that link. Without the flag, neo creates the document and resolvesolderDocumentIdin the background. - While a deposit has no remote file, extract may rebuild the key from
flags.extract.storagePath(the same Tower template) and signGetObjectwith the named upload config. That bridge goes away once confirm writes the remote file. flags.extract.storageConfigis the name of theuploadConfigsentry extract reads. Empty falls back to the active Box config, thendefault. The Box active target is left unchanged. The entry always hassettings.endpoint, plus bucket, region, and the secret.- Extract uses an agent token. Neo returns every company with
flags.extract.enabled. A company token returns only that company. The watch list is that flag. It is not a company list in the container, notPLATFORM_ADMIN_TOKEN, and not a super-admin user. - Prompt and field set live on the enrich folder (
extract.prompt,displayedCodes).flags.extractis opt-in plus storage read. The worker image only carries the response contract. - Extract and download ignore drafts.
Already on this branch:
- Named storage config, template key, and presigned
GetObjectfor deposits that have no remote file yet. - Company and folder metadata as the model context.
npm run extract:load-companywrites the formula flag, the named S3 config, the attribute catalogue, and the Formules folder. An empty secret does not wipe a secret already stored.- Draft deposit:
POST /documentscreates the remote file and an invisible document.olderDocumentIdis written in the background during the upload.POST /documents/:id/confirmreveals the draft.noDuplicateis the only hash dedup that skips creation.
Not built yet:
- Eligible listing across companies for the extract agent token.
- The worker looping those companies. Today one process, one token, one company, then it exits.
Files Modified:
-
src/domains/company/extract*.ts— config, promote, eligible, batches,storageConfig,storagePath -
src/domains/company/company.controller.ts/company.service.ts/company.module.ts— extract routes -
src/domains/document/document.service.ts/document.controller.ts— draft deposit, confirm, download session (logs file_key / bucket / endpoint host),PATCH /documents/:id/extract -
src/domains/document/deposit-key.ts—-2/-3suffix -
src/domains/document/older-document.ts— hash link during the upload -
src/domains/document/templates/*— lists hide drafts -
src/domains/document/extract-storage.ts— template key,#file_directoryfromftp_source-pathoremail_mailbox, S3 presign -
src/scripts/load-extract-company.ts— formula company loader; Formules matches FTPtest-extract-pr -
Extract worker: Huginn (formerly
docs/next/extract-worker/) -
GET /documents/:idlets Tower check access in one call and returnscompanyIdplusfileId.tower. -
Box and web use
tower: trueon the remotefile (storagetargetstaysAMAZON_S3/FTP/ …) to choose the Tower download path.