Background Jobs & Cron

What this module does

Some operations are too slow or too broad to run inside a web request:

• Syncing thousands of emails across all cases
• Running overnight reports across all 40+ agencies
• Sending scheduled notifications

Two separate systems handle this work: the Huey job queue (orchid_jobs) for on-demand async tasks, and cronapp.py for scheduled jobs.

Key files

• cronapp.py — 42 KB scheduled job runner
• Directoryorchid_jobs/

  • __init__.py — app factory, Huey configuration
  • Directoryapp/

    • app.py — Huey setup, Flask app context
  • Directoryjobs/

    • email_sync.py — email sync jobs (periodic + on-demand)
    • email_matching.py — email-to-contact matching job
    • example.py — reference implementations
  • Directorymodels/

    • job.py — Job tracking model
    • error.py — Job error logging
  • Directoryhelpers/

    • email.py — email helpers for jobs

The two background systems

orchid_jobs — Huey task queue

An async task queue built on Huey. It runs as a separate process from flaskapp.py.

How it works:

1. flaskapp.py enqueues a task by calling a Huey task function
2. The task is stored in the queue backend (memory, SQLite, or Redis depending on environment)
3. A Huey worker process picks up the task and executes it
4. The worker runs inside its own Flask app context (from orchid_jobs/__init__.py)

Available jobs:

Job	When it runs	What it does
EMAIL_SYNC_PERIODIC	Periodically	Syncs email for all agencies
EMAIL_SYNC_AGENCY	On-demand	Syncs email for one specific agency
EMAIL_SYNC_ADMIN	On-demand	Syncs email for one specific admin
EMAIL_SYNC_DISCONNECT	On-demand	Disconnects email sync for an admin
EMAIL_MATCHING	After sync	Matches emails to cases/contacts

Huey backends by environment:

• MemoryHuey — local development (tasks lost on restart)
• SqliteHuey — staging (tasks persisted in SQLite)
• RedisHuey — production (tasks persisted in Redis)
• BlackHoleHuey — testing (tasks discarded immediately)

cronapp.py — the scheduled runner

A separate Flask application that runs on a cron schedule (configured in AWS). Unlike flaskapp.py, which connects to one agency’s database, cronapp.py iterates all agency databases.

How it works:

# Pseudocode of cronapp.py's main pattern:
for db_name in get_all_agency_databases():
    engine = create_dynamic_engine(db_name)
    with app.app_context():
        # run job for this agency
        run_notifications(engine)
        run_health_checks(engine)
        # etc.

What cron jobs do:

• Send scheduled notifications (reminders, upcoming events)
• Run health checks on agency data integrity
• Process overnight aggregations
• Clean up expired sessions and tokens

Job context — carrying Flask state into background jobs

Background jobs run outside the normal request lifecycle, so they cannot access g or session. The JMSJobContext class in orchid_jobs is used to carry necessary context (agency ID, admin ID, etc.) into job functions:

# Enqueue a job from a web request:
context = JMSJobContext.from_request()
email_sync_job(context.serialize())

# Inside the job:
def email_sync_job(context_data):
    context = JMSJobContext.deserialize(context_data)
    with app.app_context():
        # use context.agency_id, context.admin_id, etc.

Gotchas

orchid_jobs is a completely separate application

orchid_jobs has its own Flask app, its own models (job tracking, error logging), and its own database connection. It cannot directly import models from flaskapp.py’s context. If you add a model to the main app that a job needs, you must also make it accessible from the orchid_jobs context.

Cron jobs hit ALL agencies

Any bug in cronapp.py affects every single agency, simultaneously. Always test cron logic with a single agency first and add explicit guards against running in unexpected environments.

MemoryHuey loses tasks on restart

In local development, MemoryHuey stores tasks in memory. If the Huey worker crashes or restarts, all queued tasks are lost. This is expected behavior for local dev. Do not rely on task persistence locally.

Debugging jobs

When a job fails silently: (1) Check the job error model (orchid_jobs/models/error.py) for stored exceptions. (2) Check the Huey worker logs. (3) Check EmailSync.status if it’s an email sync issue. There are no automatic alerts on job failures.