Hinata Wiki

Appearance

Local Testing Guide — Mail Poller on Mac ​

Before deploying to Z2, test mail-poller locally on your Mac to verify:

  • Credential loading
  • IMAP and Graph API connectivity
  • State persistence
  • Archive format

Prerequisites ​

  • Python 3.11+
  • Credentials available (from ct103 or local copies)
  • Network access to Gmail IMAP and Microsoft Graph API

Step 1: Prepare Test Environment ​

Create a local testing directory:

bash
mkdir -p /tmp/mail-poller-test/{archive,logs}

# Copy credentials
cp -r /path/to/itachi-credentials /tmp/mail-poller-test/credentials
chmod 600 /tmp/mail-poller-test/credentials/*.json

# Copy script
cp projects/brain/mail-poller-z2/mail-poller.py /tmp/mail-poller-test/

Step 2: Modify Script for Testing (temporary) ​

Edit /tmp/mail-poller-test/mail-poller.py to use test directories:

python
# Line 50: Change to test paths
STATE_DIR = Path("/tmp/mail-poller-test")
ARCHIVE_DIR = STATE_DIR / "archive"
CREDENTIALS_PATH = "/tmp/mail-poller-test/credentials"

Or use environment variables (preferred):

bash
# Modify script to read from env:
# CREDENTIALS_PATH = os.getenv("CREDENTIALS_PATH", "/opt/itachi/credentials")
# STATE_DIR = Path(os.getenv("STATE_DIR", "/opt/hinata/mail-poller"))

Step 3: Test Individual Accounts ​

Test Gmail (IMAP) ​

Dry-run (no writes):

bash
cd /tmp/mail-poller-test
python3 mail-poller.py \
  --account gmail \
  --dry-run \
  --verbose

Expected output:

[...] [INFO] mail-poller: [gmail] Starting IMAP poll...
[...] [INFO] mail-poller: [gmail] Connected to imap.gmail.com
[...] [INFO] mail-poller: [gmail/INBOX] Found N new messages
[...] [INFO] mail-poller: [gmail] Poll complete: N new messages

Full run (with writes):

bash
cd /tmp/mail-poller-test
python3 mail-poller.py --account gmail --verbose

Check output:

bash
# Should create state.json
cat state.json | python3 -m json.tool

# Should create archive structure
find archive/gmail -name "*.json" | head -5

# Sample message
find archive/gmail -name "*.json" | head -1 | xargs cat | python3 -m json.tool

Test Outlook (Graph API) ​

Dry-run:

bash
python3 mail-poller.py \
  --account hotmail-michael-asolo \
  --dry-run \
  --verbose

Expected output:

[...] [INFO] mail-poller: [hotmail-michael-asolo] Starting Graph API poll...
[...] [INFO] mail-poller: [hotmail-michael-asolo] Token refreshed
[...] [INFO] mail-poller: [hotmail-michael-asolo] Found N new messages
[...] [INFO] mail-poller: [hotmail-michael-asolo] Poll complete: N new messages

Check if token was updated:

bash
# Should have 'access_token', 'refresh_token', 'updated'
cat /tmp/mail-poller-test/credentials/outlook-tokens-hotmail-michael-asolo.json | python3 -m json.tool

Step 4: Test All Accounts (Full Run) ​

bash
python3 mail-poller.py --verbose

Expected output:

[...] [INFO] mail-poller: Starting mail poller (dry_run=False)
[...] [INFO] mail-poller: [gmail] Starting IMAP poll...
[...] [INFO] mail-poller: [gmail] Poll complete: N new messages
[...] [INFO] mail-poller: [hotmail-michael-asolo] Starting Graph API poll...
[...] [INFO] mail-poller: [hotmail-michael-asolo] Poll complete: N new messages
[...] [INFO] mail-poller: [outlook-michael-nnamah] Starting Graph API poll...
[...] [INFO] mail-poller: [outlook-michael-nnamah] Poll complete: N new messages
[...] [INFO] mail-poller: [outlook-n-nnamah] Starting Graph API poll...
[...] [INFO] mail-poller: [outlook-n-nnamah] Poll complete: N new messages
[...] [INFO] mail-poller: Archived M messages
[...] [INFO] mail-poller: Mail poller complete

Step 5: Verify State Persistence ​

Run again (should have zero new messages, since cursors were just updated):

bash
python3 mail-poller.py --verbose

Expected output:

[...] [INFO] mail-poller: [gmail/INBOX] No new messages
[...] [INFO] mail-poller: [gmail] Poll complete: 0 new messages
[...] [INFO] mail-poller: [hotmail-michael-asolo] Poll complete: 0 new messages
[...] [INFO] mail-poller: No new messages
[...] [INFO] mail-poller: Mail poller complete

If you got new messages on second run, the state cursor is not working. Check:

bash
jq .gmail.folders /tmp/mail-poller-test/state.json
jq .hotmail-michael-asolo.last_received /tmp/mail-poller-test/state.json

Step 6: Verify Archive Format ​

bash
# Count messages per account
find /tmp/mail-poller-test/archive -type f -name "*.json" | \
  sed 's/.*archive\///;s/\/202.*//' | \
  sort | uniq -c

# Verify all messages have required fields
python3 << 'PYTHON'
import json
from pathlib import Path

archive_dir = Path("/tmp/mail-poller-test/archive")
required_fields = {"account", "email", "message_id", "message_hash", "date", "subject", "from", "to", "body_text", "year_month"}

for json_file in archive_dir.rglob("*.json"):
    with open(json_file) as f:
        msg = json.load(f)
        missing = required_fields - set(msg.keys())
        if missing:
            print(f"MISSING FIELDS in {json_file}: {missing}")
        else:
            print(f"OK: {json_file.name}")
PYTHON

Sample valid message:

json
{
  "account": "gmail",
  "email": "michael.asolo1@gmail.com",
  "message_id": "<abc123@gmail.com>",
  "message_hash": "abcd1234efgh5678",
  "date": "2026-06-05T10:30:45+00:00",
  "subject": "Test Subject",
  "from": "sender@example.com",
  "to": "michael.asolo1@gmail.com",
  "body_text": "Plain text body...",
  "body_html": "",
  "year_month": "2026/06"
}

Step 7: Test Error Handling ​

Simulate Invalid Credential ​

Edit a credential file with bad password:

bash
python3 << 'PYTHON'
import json

cred = json.load(open("/tmp/mail-poller-test/credentials/mail_imap_credential.json"))
cred["password"] = "invalid"
json.dump(cred, open("/tmp/mail-poller-test/credentials/mail_imap_credential.json", "w"))
PYTHON

python3 mail-poller.py --account gmail --verbose

Expected output:

[...] [ERROR] [gmail] IMAP poll failed: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'
[...] [INFO] mail-poller: Mail poller complete

Script should continue and try other accounts (error gracefully handled).

Simulate Missing Credential File ​

bash
rm /tmp/mail-poller-test/credentials/mail_imap_credential.json

python3 mail-poller.py --account gmail --verbose

Expected output:

[...] [ERROR] [gmail] Credential file not found: /tmp/mail-poller-test/credentials/mail_imap_credential.json
[...] [ERROR] [gmail] IMAP poll failed: ...

Step 8: Performance Check ​

Time a full run:

bash
time python3 mail-poller.py --verbose

Expected performance:

  • Total time: 1–5 seconds (depending on account size)
  • IMAP polling: ~300ms per account
  • Graph API polling: ~500ms per account
  • Archive I/O: ~100ms

If significantly slower, check:

  • Network latency (ping imap.gmail.com, graph.microsoft.com)
  • Disk I/O (check /tmp is not on slow network mount)
  • Python startup time

Step 9: Cleanup ​

After testing, remove temporary files:

bash
rm -rf /tmp/mail-poller-test

Or keep for repeated testing (state.json will be preserved).

Common Issues ​

"urllib.error.HTTPError: 401 Client Error: Unauthorized" ​

Microsoft token expired or invalid. Regenerate tokens (see INSTALLATION.md Step 1c).

"imaplib.IMAP4.error: [AUTHENTICATIONFAILED] Invalid credentials" ​

Gmail app password incorrect or expired. Regenerate at https://myaccount.google.com/apppasswords.

"ConnectionRefusedError: [Errno 61] Connection refused" ​

Cannot reach Gmail IMAP or Microsoft Graph servers. Check:

  • Network connectivity: ping imap.gmail.com
  • Firewall rules: nc -zv imap.gmail.com 993

"state.json persists old cursor (fetches old emails)" ​

State file not being updated. Check:

  • File permissions: ls -la state.json (should be writable)
  • Disk space: df /tmp
  • Script errors in logs (verbose mode)

Reset cursor:

bash
rm /tmp/mail-poller-test/state.json
python3 mail-poller.py  # Will refetch everything

Next Steps ​

Once local testing passes:

  1. Copy mail-poller.py to vault (no modifications)
  2. Deploy to Z2: ./deploy.sh
  3. Run manual test on ct102: ssh z2 'pct exec 102 -- /opt/hinata/mail-poller/mail-poller.py --verbose'
  4. Verify systemd timer: ssh z2 'pct exec 102 -- systemctl list-timers'