SQS Data Feed

SQS means AWS Simple Queue Service. Use it when the broker needs a buffered, retryable way to push high-volume trading facts or aggregate activity into Returning.AI.

What SQS proves

A successful SQS send only proves the message was accepted by the queue. Reward calculation, deduplication, and user matching happen downstream. Keep a clear idempotency key so retries do not double credit a trader. A successful send returns HTTP 200 plus a MessageId, which means queued, not processed.

When to use SQS

Use SQS for	Do not use SQS for
Hourly or daily trading aggregates	Initial widget page authentication
Backfills and replayable trading facts	Events that must update UI instantly
Large broker exports that need chunking or manifests	Raw secrets, MT passwords, or personally unnecessary fields

Message shape

The data-ingest surface uses snake_case. Send a fixed trigger_id for the integration, a unique metadata.batch_id for each distinct payload, and rows that can upsert by user_id plus date.

trading-activity-message.json

{
  "trigger_id": "664f1c2ab3d4e5f6a7b8c9d0",
  "metadata": {
    "source": "broker-dwh",
    "batch_id": "2026-06-11-009"
  },
  "data": [
    {
      "user_id": "BRK-100234",
      "date": "2026-06-11",
      "deposit_total": 2500.00,
      "closed_lots": 14.25
    }
  ]
}

Large payloads and CSV exports

Direct SQS messages have a hard size limit of about 256 KB. Larger batches use the file-ingest endpoint as multipart upload; the service stores the file in S3, queues a small S3 pointer, and chunks rows in groups of about 500. Treat 10k records as a throughput rule of thumb, not a byte limit.

file-ingest.sh

curl -X POST \
  "https://data-workflow.returning.ai/apis/v1/data-workflows/file-ingests?trigger_id=664f1c2ab3d4e5f6a7b8c9d0" \
  -H "x-api-key: YOUR_DATA_WORKFLOW_API_KEY" \
  -F "data=@hour-09.csv;type=text/csv" \
  -F 'metadata={"source":"broker-dwh","batch_id":"2026-06-11-009"}'

Dedupe and values

trigger_id is a fixed MongoDB ObjectId for the integration. Send it unchanged on every request.
batch_id is request-level dedupe. Use yyyy-MM-dd-HHH; resending the same batch is ignored, so corrected payloads need a new batch ID.
user_id + date is the row upsert key. Prefer overwrite-with-all-time snapshot totals for deposits and closed trades instead of additive deltas.
Send base 100% values. Returning.AI applies tier multipliers server-side. Do not promise record-level cross-batch dedupe unless it is re-verified.
Send hourly trade batches about 30 minutes past the hour so the broker trade database has time to replicate.

Operational checks

Agree whether the feed sends raw deals, per-account aggregates, or campaign-specific metrics.
Confirm the timezone and close-time semantics for deal windows.
Document whether bonus credit, rebate credit, and real deposit are separate fields.
Monitor queue depth, dead-letter count, and oldest message age.
Keep sample payloads and replay scripts for broker UAT.