SQS data feed

AWS Simple Queue Service (SQS) is a buffered transport for trading facts, approved aggregates, backfills, and replayable exports. It is a delivery step, not a reward result. The queue, Event Ingest trigger, workflow, and user readback must each be checked.

When to use SQS

Direct SQS requires a provisioned queue and AWS permissions. Some brokers instead receive a provisioned HTTP or multipart file endpoint. Choose the transport that the integration owner has actually provisioned.

Message shape

Use the root Event Ingest envelope only when the configured trigger expects it. The trigger_id is a fixed ObjectId for that configured integration, not a new value for each message. If batch dedupe is enabled, the configured path must be checked before sending. In the documented contract, metadata.batch_id is at the root, whiledata contains complete rows.

{
  "trigger_id": "<FIXED_EVENT_INGEST_TRIGGER_OBJECT_ID>",
  "metadata": {
    "source": "broker-dwh",
    "batch_id": "trades-2026-09-07-001"
  },
  "data": [
    {
      "customer_id": "CUSTOMER_1001",
      "source_event_id": "deal-2001-fx-001",
      "occurred_at": "2026-09-07T10:15:00.000Z",
      "event_type": "trade.closed",
      "platform": "mt5",
      "server": "broker-live-1",
      "account_login": "ACCOUNT_2001",
      "raw_symbol": "EURUSD.a",
      "normalized_volume": 1.2,
      "volume_unit": "standard_lot"
    }
  ]
}

The row names above are a synthetic example mapping, not a universal Returning.AI trading schema. Confirm the workflow's fields, required record ID path, asset-class mapping, and unit conversion with the setup owner. Keep account lineage and source IDs even when the delivered row is an aggregate.

Sender setup

1. Store the fixed trigger reference and provisioned transport settings on the server. Keep AWS IAM credentials or a data-ingest key out of browser code and logs.
2. Build the complete serialized envelope before measuring it. Count UTF-8 bytes, including root metadata, row data, and any chunk identity fields.
3. Send one candidate or an AWS batch only after it fits the agreed budget. For SendMessageBatch, inspect every successful and failed entry even when the HTTP response is 200.
4. Record the logical batch ID, transport chunk ID, source record IDs, and original event times for reconciliation. Do not log credentials or full customer exports.

The repository example is a no-network packaging helper. It prepares JSON candidates and prints their sizes; it never sends to AWS or a Returning.AI endpoint.

Download the no-network data-feed example. Requires Node.js 20 or newer. This is a tested packaging example, not a live sender.

unzip data-feed.zip
cd data-feed
npm ci
npm test
npm run demo

Large payloads and byte budgets

AWS SQS currently allows up to 1,048,576 bytes (1 MiB) per message. SendMessageBatch allows up to 10 messages and a 1 MiB aggregate payload ceiling. These AWS limits are separate from the lower limit of a provisioned queue, HTTP gateway, file route, or sender choice.

See the verified AWS references for the message quotas and SendMessageBatch request.

File option: provisioned multipart CSV

Use a file only when Returning.AI has provisioned the URL, credentials, effective edge limit, and a multi-chunk identity and dedupe contract for the configured trigger. The command below intentionally uses named values, not a guessed public route.

curl --fail-with-body -sS -X POST \
  "<PROVISIONED_FILE_INGEST_URL>?trigger_id=<FIXED_EVENT_INGEST_TRIGGER_OBJECT_ID>" \
  -H "x-api-key: <PROVISIONED_DATA_WORKFLOW_KEY>" \
  -F "data=@trades-2026-09-07.csv;type=text/csv" \
  -F 'metadata={"source":"broker-dwh","batch_id":"file-2026-09-07-001"}'

A response such as queued: true means the file job was accepted by the application path. It does not mean that all rows passed validation or that a reward was applied. Check the persisted ingest result and downstream workflow status.

Dedupe and retry identities

Dedupe has more than one boundary. Confirm which paths are configured on the actual trigger before promising replay behavior.

• Retry an identical logical batch with the same configured identity only when the goal is to suppress an exact duplicate. A missing configured identity is not proof that a retry is safe.
• Persist each exact serialized body and its chunk plan. Do not change record order, metadata, or byte budget and reuse the same logical batch ID: an existing chunk ID could then refer to different records.
• A changed payload needs an approved correction policy. A new batch ID can create a second additive reward; it is not a generic correction method.
• Preserve the original occurred_at and source record ID on a retry. Do not replace business event time with retry time.
• Do not send both raw deals and their derived user aggregate to the same additive reward workflow unless one is explicitly non-crediting lineage.

Operational checks

Follow the delivery in order. If a checkpoint is missing, ask the owner for the named diagnostic rather than sending the same payload again.

Next step

If the input is trading activity, define the canonical-user aggregation and units in the trading volume and rewards guide. If the input is registration, KYC, or another lifecycle event, use the workflow webhook guide instead. For the browser result, return to the widget quickstart only after the saved user state is confirmed.