Processing 50,000 Hashes in 40 Seconds - Memory-Mapped IO and Sharded Queues

When designing a cryptographic reverse lookup system, single point lookups represent an optimized low-latency path: an HTTP request arrives, hits an in-memory hot cache or a partition-aware index, and returns in ~0.48ms.

However, real-world forensic investigations and daily banking reconciliation runs do not operate one record at a time. Enterprise security operations centers (SOCs) and accounting teams frequently ingest dumps containing 10,000 to 50,000 hashes that must be resolved against a 100-million record dataset.

Handling an array of 50,000 64-character hexadecimal strings over standard JSON endpoints will cripple conventional web frameworks due to:

• Excessive JSON serialization/deserialization memory allocation.
• HTTP connection timeouts (gateways terminating connections after 30 seconds).
• Thread starvation and head-of-line blocking in database connection pools.

Here is how the LOOKUP API asynchronous batch worker engine resolves 50,000 hashes in under 42 seconds with zero impact on low-latency point query SLAs.

1. Asynchronous Job State Machine

To prevent slow client timeouts and gateway 504 errors, any batch payload exceeding 500 items is processed through an asynchronous job queue:

[Client POST /v1/jobs/batch] ── (Payload: 50,000 hashes)
            │
            ▼
[Edge Gateway: Hash Validation & Chunking]
            │ ── (Allocates UUID job_id)
            │ ── (Stores state: "queued" in In-Memory Registry)
            │ ── (Returns HTTP 202 Accepted in <15ms)
            │
            ▼
[Distributed Background Worker Pool]
       ├── Worker 1: Shard Chunk [0..9999]
       ├── Worker 2: Shard Chunk [10000..19999]
       ├── Worker 3: Shard Chunk [20000..29999]
       ├── Worker 4: Shard Chunk [30000..39999]
       └── Worker 5: Shard Chunk [40000..49999]
            │
            ▼
[Job Status Registry: Atomic Progress Increments]
            │
            ▼
[Webhook Dispatcher / Secure Download S3 Presigned URL]

Fast Acceptance Phase (<15ms)

When a client submits POST /v1/jobs/batch:

1. The gateway checks the client's API key tier (verifying production eligibility).
2. The incoming stream is validated for hex format (^[a-fA-F0-9]{64}$).
3. An internal state record is written to a low-latency key-value cluster with status: "queued", total: 50000, and processed: 0.
4. The client immediately receives HTTP 202 Accepted with a tracking job_id.

2. Zero-Copy Sharding and Concurrent Lookups

A 50,000-hash JSON body is approximately 3.6 megabytes of raw text. Parsing this into standard language objects creates tens of thousands of heap allocations.

Instead, the worker engine parses hashes as fixed 32-byte binary slices without intermediate string copies.

Partition Chunking

The 50,000 hashes are split into concurrent chunks of 10,000 items each. Each worker thread processes its partition chunk using pipelined multi-key batch requests directly against the underlying storage cluster:

• Rather than issuing 50,000 individual queries, the worker bundles 500 hashes per storage partition query.
• Network round-trips drop from 50,000 down to 100 batch round-trips.
• With 5 parallel workers, all 100 partition queries execute concurrently across the cluster.

3. Real-Time Telemetry and Webhook Delivery

While the job executes, clients have two options for monitoring and retrieval:

Option A: Polling Job Status

Clients poll GET /v1/jobs/{job_id} at 2-second intervals:

{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "processing",
  "total_hashes": 50000,
  "processed_hashes": 34500,
  "found_count": 34120,
  "missing_count": 380,
  "progress_percentage": 69.0
}

Option B: Automated Webhook Dispatch

If the client supplied a webhook_url in the original submission, the worker engine signs the completed payload with an X-Signature-SHA256 HMAC header and posts the results directly to the client's listener:

POST https://api.yourfintech.com/webhooks/lookup-complete
X-Signature-SHA256: 7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069
Content-Type: application/json
 
{
  "job_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d",
  "status": "completed",
  "execution_time_seconds": 41.8,
  "total_requested": 50000,
  "found_count": 49821,
  "missing_count": 179
}

Benchmark: Synchronous vs. Asynchronous Performance

Getting Started with 50K Async Jobs

To submit background batch jobs, register for an API key and explore our interactive job monitor:

• Launch Async Workbench: Interactive Jobs Console
• Review OpenAPI Schemas: API Explorer
• Get Production Access: Developer Keys