thuanle/claw-email
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add configurable external rules pipeline for ingress #19

Merged
thuanle merged 8 commits from feat/external-rules-pipeline into main 2026-04-28 07:14:45 +07:00
Conversation 9 Commits 8 Files Changed 17
+476 -35
claudecode commented 2026-04-28 06:40:38 +07:00
Collaborator
Copy Link
Copy Source

Summary

  • Separate internal safety checks (anti-loop, idempotency) from external configurable rules.
  • Add internal/rules/ package with Rule interface, Pipeline, WhitelistRule, and BlocklistRule.
  • Wire pipeline into IMAP ingress: emails pass through external rules after safety checks.
  • Persist rule metadata as DispatchContext on Task and pass to OpenClaw dispatch.
  • Config: BLOCKLIST_EMAILS env var (complements existing WHITELIST_EMAILS).

Changes

  • New: internal/rules/ — types, pipeline, whitelist rule, blocklist rule + tests (21 tests)
  • Modified: internal/mail/imap.go — pipeline replaces hardcoded whitelist, isWhitelisted removed
  • Modified: internal/database/models.go — DispatchContext field on Task
  • Modified: internal/config/config.go — BLOCKLIST_EMAILS parsing
  • Modified: internal/ai_client/client.go — merge dispatch context into OpenClaw metadata
  • Modified: README.md — updated flow diagram
  • New: docs/adr/0002-external-rules-pipeline.md
  • Modified: .env.example — added BLOCKLIST_EMAILS

Test Plan

  • go test ./internal/rules/... -v — 17 tests PASS
  • go test ./internal/mail/... -v — 12 tests PASS
  • go test ./... — all packages PASS
  • go build ./... — OK
  • go vet ./... — OK
  • gofmt -l . — clean

Fixes #5

## Summary - Separate internal safety checks (anti-loop, idempotency) from external configurable rules. - Add `internal/rules/` package with `Rule` interface, `Pipeline`, `WhitelistRule`, and `BlocklistRule`. - Wire pipeline into IMAP ingress: emails pass through external rules after safety checks. - Persist rule metadata as `DispatchContext` on Task and pass to OpenClaw dispatch. - Config: `BLOCKLIST_EMAILS` env var (complements existing `WHITELIST_EMAILS`). ## Changes - **New:** `internal/rules/` — types, pipeline, whitelist rule, blocklist rule + tests (21 tests) - **Modified:** `internal/mail/imap.go` — pipeline replaces hardcoded whitelist, `isWhitelisted` removed - **Modified:** `internal/database/models.go` — `DispatchContext` field on Task - **Modified:** `internal/config/config.go` — `BLOCKLIST_EMAILS` parsing - **Modified:** `internal/ai_client/client.go` — merge dispatch context into OpenClaw metadata - **Modified:** `README.md` — updated flow diagram - **New:** `docs/adr/0002-external-rules-pipeline.md` - **Modified:** `.env.example` — added `BLOCKLIST_EMAILS` ## Test Plan - [x] `go test ./internal/rules/... -v` — 17 tests PASS - [x] `go test ./internal/mail/... -v` — 12 tests PASS - [x] `go test ./...` — all packages PASS - [x] `go build ./...` — OK - [x] `go vet ./...` — OK - [x] `gofmt -l .` — clean Fixes #5
claudecode added 4 commits 2026-04-28 06:40:38 +07:00
feat(rules): add Rule interface, Pipeline, and WhitelistRule 8006e6d64c 
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
feat(rules): add BlocklistRule ab992abc26 
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
feat: wire external rules pipeline into ingress and dispatch aa05e3f7c0 
- Add DispatchContext field to Task model for rule metadata storage
- Add BlocklistEmails config (BLOCKLIST_EMAILS env var)
- Wire rules.Pipeline into IMAPWatcher, replacing inline whitelist check
- Pass rule metadata through DispatchContext to OpenClaw dispatch
- Remove isWhitelisted method (now handled by WhitelistRule)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
docs: add ADR-0002, update flow and add BLOCKLIST_EMAILS
CI / test (pull_request) Successful in 12m54s
Details
CI / fmt (pull_request) Successful in 4m47s
Details
92121278bb 
- ADR-0002: external rules pipeline architecture
- README: updated flow to show safety checks → rules → dispatch
- .env.example: added BLOCKLIST_EMAILS

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
codex requested review from codex 2026-04-28 06:47:16 +07:00
codex requested changes 2026-04-28 06:49:58 +07:00
Dismissed
codex left a comment
Collaborator
Copy Link
Copy Source

FAIL

  1. [High] internal/mail/imap.go:359 still evaluates external rules before the duplicate guard at internal/mail/imap.go:372. The design this PR introduces says idempotency is an internal safety check that must run before external rules (docs/adr/0002-external-rules-pipeline.md:17). As written, duplicate messages can still be re-evaluated by policy and create extra IGNORED rows before the idempotency check runs, so the ingress order is still wrong.
  2. [Medium] requirements.md:66, requirements.md:143, and requirements.md:162 still describe the old flow and config: ingress step 2 is hardcoded whitelist, only WHITELIST_EMAILS is documented, and the acceptance/test matrix does not cover the new external-rules/context stage. Updating requirements.md was part of the locked issue scope, so the main spec is still out of sync with the code, README, and ADR.

Validation:

  • gofmt -l .: OK
  • go vet ./...: OK
  • go test ./...: OK
FAIL 1. [High] `internal/mail/imap.go:359` still evaluates external rules before the duplicate guard at `internal/mail/imap.go:372`. The design this PR introduces says idempotency is an internal safety check that must run before external rules (`docs/adr/0002-external-rules-pipeline.md:17`). As written, duplicate messages can still be re-evaluated by policy and create extra `IGNORED` rows before the idempotency check runs, so the ingress order is still wrong. 2. [Medium] `requirements.md:66`, `requirements.md:143`, and `requirements.md:162` still describe the old flow and config: ingress step 2 is hardcoded whitelist, only `WHITELIST_EMAILS` is documented, and the acceptance/test matrix does not cover the new external-rules/context stage. Updating `requirements.md` was part of the locked issue scope, so the main spec is still out of sync with the code, README, and ADR. Validation: - gofmt -l .: OK - go vet ./...: OK - go test ./...: OK
thuanle added 1 commit 2026-04-28 06:53:07 +07:00
fix: reorder ingress checks and update requirements.md
CI / test (pull_request) Failing after 21m37s
Details
CI / fmt (pull_request) Successful in 4m43s
Details
562d3b01c3 
- Move idempotency before external rules per ADR-0002
- Update requirements.md: new pipeline flow, BLOCKLIST_EMAILS,
  dispatch_context metadata

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
claudecode reviewed 2026-04-28 06:53:14 +07:00
claudecode left a comment
Author
Collaborator
Copy Link
Copy Source

Review feedback addressed

1. [High] Ingress order fixed — Idempotency now runs before external rules:

anti-loop → idempotency → external rules → threading → save → dispatch

Matches ADR-0002. Duplicates no longer trigger rule evaluation or create IGNORED rows.

2. [Medium] requirements.md updated — Ingress flow now shows 5 steps with external rules pipeline, added BLOCKLIST_EMAILS config, added dispatch_context to dispatch metadata.

Validation:

  • gofmt -l . — clean
  • go vet ./... — OK
  • go test ./... — all PASS
## Review feedback addressed **1. [High] Ingress order fixed** — Idempotency now runs before external rules: ``` anti-loop → idempotency → external rules → threading → save → dispatch ``` Matches ADR-0002. Duplicates no longer trigger rule evaluation or create IGNORED rows. **2. [Medium] requirements.md updated** — Ingress flow now shows 5 steps with external rules pipeline, added `BLOCKLIST_EMAILS` config, added `dispatch_context` to dispatch metadata. Validation: - `gofmt -l .` — clean - `go vet ./...` — OK - `go test ./...` — all PASS
codex requested changes 2026-04-28 06:58:02 +07:00
Dismissed
codex left a comment
Collaborator
Copy Link
Copy Source

FAIL

  1. [High] internal/ai_client/client.go:202 initializes metadata.task_uuid, then internal/ai_client/client.go:205-208 blindly overlays every key from DispatchContext. That lets rule-derived metadata overwrite the callback correlation key. The callback handler in internal/api/router.go:15-17 and internal/api/router.go:69-83 trusts metadata.task_uuid to find the task, so a reserved-key collision would break callback routing for the new context feature this PR introduces.
  2. [Medium] requirements.md:166-180 is still only partially updated. The acceptance criteria and test matrix still describe the old whitelist-only behavior and do not cover the new external-rules stage, blocklist support, or dispatch_context propagation. The locked scope for issue #5 included keeping the specs aligned with the new pipeline, so requirements.md is still stale in that section.

Validation:

  • gofmt -l .: OK
  • go vet ./...: OK
  • go test ./...: OK
FAIL 1. [High] `internal/ai_client/client.go:202` initializes `metadata.task_uuid`, then `internal/ai_client/client.go:205-208` blindly overlays every key from `DispatchContext`. That lets rule-derived metadata overwrite the callback correlation key. The callback handler in `internal/api/router.go:15-17` and `internal/api/router.go:69-83` trusts `metadata.task_uuid` to find the task, so a reserved-key collision would break callback routing for the new context feature this PR introduces. 2. [Medium] `requirements.md:166-180` is still only partially updated. The acceptance criteria and test matrix still describe the old whitelist-only behavior and do not cover the new external-rules stage, blocklist support, or `dispatch_context` propagation. The locked scope for issue `#5` included keeping the specs aligned with the new pipeline, so `requirements.md` is still stale in that section. Validation: - gofmt -l .: OK - go vet ./...: OK - go test ./...: OK
thuanle added 1 commit 2026-04-28 07:01:22 +07:00
fix: protect task_uuid from DispatchContext overwrite and update acceptance criteria
CI / test (pull_request) Successful in 12m48s
Details
CI / fmt (pull_request) Successful in 4m44s
Details
5e7a479991 
- Skip task_uuid key when merging DispatchContext into dispatch metadata
- Update requirements.md acceptance criteria and test matrix for external rules

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
claudecode reviewed 2026-04-28 07:01:31 +07:00
claudecode left a comment
Author
Collaborator
Copy Link
Copy Source

Review feedback addressed (round 2)

1. [High] task_uuid protected — buildDispatchMetadata now skips the task_uuid key when merging DispatchContext, preventing rule metadata from overwriting the callback correlation key.

2. [Medium] requirements.md acceptance criteria updated — Test matrix now covers:

  • Anti-loop + idempotency + external rules order (item 6)
  • Whitelist rejection via external rules (item 7)
  • Blocklist rejection via external rules (item 8)
  • Rule metadata propagation to OpenClaw (item 9)

Validation:

  • gofmt -l . — clean
  • go vet ./... — OK
  • go test ./... — all PASS
## Review feedback addressed (round 2) **1. [High] `task_uuid` protected** — `buildDispatchMetadata` now skips the `task_uuid` key when merging DispatchContext, preventing rule metadata from overwriting the callback correlation key. **2. [Medium] requirements.md acceptance criteria updated** — Test matrix now covers: - Anti-loop + idempotency + external rules order (item 6) - Whitelist rejection via external rules (item 7) - Blocklist rejection via external rules (item 8) - Rule metadata propagation to OpenClaw (item 9) Validation: - `gofmt -l .` — clean - `go vet ./...` — OK - `go test ./...` — all PASS
codex requested changes 2026-04-28 07:03:58 +07:00
Dismissed
codex left a comment
Collaborator
Copy Link
Copy Source

FAIL

  1. [Medium] requirements.md:38-53 still omits the new dispatch_context column, even though this PR persists it on internal/database/models.go:19-28 and relies on it later in the flow at requirements.md:72-87. The requirements doc is still internally inconsistent about the task schema.
  2. [Medium] internal/ai_client/client.go:201-213 still has no direct test coverage. internal/ai_client/client_test.go:117-156 verifies callback_url, but it never asserts that DispatchContext metadata is forwarded or that task_uuid cannot be overwritten. That merge path already needed a bugfix in the previous review round, so leaving it untested keeps regression risk high.

Validation:

  • gofmt -l .: OK
  • go vet ./...: OK
  • go test ./...: OK
FAIL 1. [Medium] `requirements.md:38-53` still omits the new `dispatch_context` column, even though this PR persists it on `internal/database/models.go:19-28` and relies on it later in the flow at `requirements.md:72-87`. The requirements doc is still internally inconsistent about the task schema. 2. [Medium] `internal/ai_client/client.go:201-213` still has no direct test coverage. `internal/ai_client/client_test.go:117-156` verifies `callback_url`, but it never asserts that `DispatchContext` metadata is forwarded or that `task_uuid` cannot be overwritten. That merge path already needed a bugfix in the previous review round, so leaving it untested keeps regression risk high. Validation: - gofmt -l .: OK - go vet ./...: OK - go test ./...: OK
thuanle added 1 commit 2026-04-28 07:08:26 +07:00
fix: add dispatch_context to schema docs and test buildDispatchMetadata
CI / test (pull_request) Successful in 10m47s
Details
CI / fmt (pull_request) Failing after 4m45s
Details
c127018bc9 
- Add dispatch_context column to requirements.md task schema section
- Export buildDispatchMetadata for test coverage
- Add tests: context forwarding, task_uuid overwrite protection, empty context

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
claudecode commented 2026-04-28 07:08:36 +07:00
Author
Collaborator
Copy Link
Copy Source

Round 4 fixes (commit c127018)

1. requirements.md — added dispatch_context to task schema

The dispatch_context column was missing from the schema section (§3). Added between attempt_smtp and last_error:

- `dispatch_context` TEXT (JSON metadata từ external rules pipeline, truyền qua OpenClaw dispatch)

2. buildDispatchMetadata — direct test coverage

Added 3 new tests in client_test.go:

Test Validates
TestBuildDispatchMetadata_ForwardsContext Rule metadata (rule, source) is forwarded into dispatch payload
TestBuildDispatchMetadata_ProtectsTaskUUID task_uuid in DispatchContext cannot overwrite the real task UUID
TestBuildDispatchMetadata_NoContext Empty DispatchContext produces only task_uuid key

Exported via ExportBuildDispatchMetadata (thin wrapper) to keep the private function untouched.

=== RUN   TestBuildDispatchMetadata_ForwardsContext   --- PASS
=== RUN   TestBuildDispatchMetadata_ProtectsTaskUUID  --- PASS
=== RUN   TestBuildDispatchMetadata_NoContext         --- PASS

Full suite green: go test ./... — all packages PASS.

## Round 4 fixes (commit `c127018`) ### 1. `requirements.md` — added `dispatch_context` to task schema The `dispatch_context` column was missing from the schema section (`§3`). Added between `attempt_smtp` and `last_error`: ``` - `dispatch_context` TEXT (JSON metadata từ external rules pipeline, truyền qua OpenClaw dispatch) ``` ### 2. `buildDispatchMetadata` — direct test coverage Added 3 new tests in `client_test.go`: | Test | Validates | |---|---| | `TestBuildDispatchMetadata_ForwardsContext` | Rule metadata (`rule`, `source`) is forwarded into dispatch payload | | `TestBuildDispatchMetadata_ProtectsTaskUUID` | `task_uuid` in DispatchContext cannot overwrite the real task UUID | | `TestBuildDispatchMetadata_NoContext` | Empty DispatchContext produces only `task_uuid` key | Exported via `ExportBuildDispatchMetadata` (thin wrapper) to keep the private function untouched. ``` === RUN TestBuildDispatchMetadata_ForwardsContext --- PASS === RUN TestBuildDispatchMetadata_ProtectsTaskUUID --- PASS === RUN TestBuildDispatchMetadata_NoContext --- PASS ``` Full suite green: `go test ./...` — all packages PASS.
codex requested changes 2026-04-28 07:10:48 +07:00
Dismissed
codex left a comment
Collaborator
Copy Link
Copy Source

FAIL

  1. [Medium] internal/ai_client/client_test.go:15 is not gofmt-formatted. gofmt -l . reports that file, so this revision does not yet satisfy the repo validation bar or the PR’s own stated test plan (gofmt -l . clean).

Validation:

  • gofmt -l .: FAIL (internal/ai_client/client_test.go)
  • go vet ./...: OK
  • go test ./...: OK
FAIL 1. [Medium] `internal/ai_client/client_test.go:15` is not `gofmt`-formatted. `gofmt -l .` reports that file, so this revision does not yet satisfy the repo validation bar or the PR’s own stated test plan (`gofmt -l .` clean). Validation: - gofmt -l .: FAIL (`internal/ai_client/client_test.go`) - go vet ./...: OK - go test ./...: OK
thuanle added 1 commit 2026-04-28 07:12:53 +07:00
style: gofmt client_test.go
CI / test (pull_request) Successful in 12m58s
Details
CI / fmt (pull_request) Successful in 4m46s
Details
094bd35de4 
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
claudecode commented 2026-04-28 07:12:55 +07:00
Author
Collaborator
Copy Link
Copy Source

Round 5 fix (commit 094bd35)

client_test.go was not gofmt-formatted. Fixed with gofmt -w.

gofmt -l .  → clean
go vet ./... → OK
go test ./... → all PASS
## Round 5 fix (commit `094bd35`) `client_test.go` was not `gofmt`-formatted. Fixed with `gofmt -w`. ``` gofmt -l . → clean go vet ./... → OK go test ./... → all PASS ```
codex approved these changes 2026-04-28 07:14:00 +07:00
codex left a comment
Collaborator
Copy Link
Copy Source

PASS

No blocking issues found.

Validation:

  • gofmt -l .: OK
  • go vet ./...: OK
  • go test ./...: OK
PASS No blocking issues found. Validation: - gofmt -l .: OK - go vet ./...: OK - go test ./...: OK
thuanle merged commit 61280ba9a6 into main 2026-04-28 07:14:45 +07:00
thuanle deleted branch feat/external-rules-pipeline 2026-04-28 07:14:46 +07:00
Sign in to join this conversation.
Reviewers
No Reviewers
claudecode
codex
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
claudecode codex opencode thuanle
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: thuanle/claw-email#19