Large Data Loads
An initial migration, a full resync or a large file import sends far more events than an integration's everyday traffic. The Integration Toolkit recognizes such a load by itself and processes it on capacity set aside for it, so that it does not slow down other organizations and is not slowed down by them.
What this means for you
- Everyday traffic is processed promptly. Another organization's large load does not slow it down.
- Large loads need no configuration. They are recognized automatically. Nothing is rejected in the normal case, nothing is lost, and the results are the same as for any other event.
- Large loads take longer. The first part is processed at once. After a short pause, the rest follows at a steady pace, which can take hours for a very large migration.
- Interactive inbound is unaffected. Pushes an end customer is waiting for keep their own lane.
- Order is kept. Events for the same entity are processed in the order in which they were accepted, also when a large load starts and when it ends. See Order of Events.
- Two responses to handle. Rarely, a request is answered with
503and aRetry-Afterheader: with the codeBULK_CAPACITY_EXHAUSTEDwhen the capacity for large loads is in use, or withINGEST_THROTTLEDwhen ingestion is busy for a moment. Wait for theRetry-Afterinterval and send the same request again. See When Capacity Is Exhausted and When Ingestion Is Busy. - A limit on the size of one event and one request. An event may produce at most 5,000 updates, and a request at most 25,000. See Size Limits.
- You can book a window for a planned load. In the Integration Hub or through the API, so that a migration runs on its own capacity from its first event. See Bulk Import Windows.
Everyday Traffic and Large Loadsโ
The inbound events of all organizations are processed together. Each organization has an allowance for that: how many events it can send in a short time and still count as everyday traffic. Single changes, delta syncs and occasional batches stay within it.
An organization that sends more than its allowance covers is treated as running a large load. A migration or a full resync does that: it sends thousands of events within minutes. From then on, the organization's events are processed on capacity set aside for the load:
- At a pace that follows the health of the services behind it, not as fast as the events arrive. A load moves faster when the platform is quiet and slower when it is busy, so that it does not degrade everyday traffic.
- With the same ordering rules as everyday traffic. What the inbound API says about the order of events, under
group_idin the API reference, applies to a large load as well. The order is also kept between the events sent before the load was recognized and the load itself; see Order of Events. - Until the load has finished and the organization has paused: for 15 minutes without new events, or for 3 minutes when the load has run for at least 30 minutes and only a trickle of events follows it (fewer than about one event every two seconds). After that, its events count as everyday traffic again.
Your large load does not slow down other organizations, and their large loads do not slow down your everyday traffic.
What Countsโ
- Every event sent to the inbound API, on every version of the endpoint.
- Every row of a file import.
- Every replayed event.
- Every inbound event that a managed call produces.
Events processed with interactive inbound do not count.
Events that produce very many updates. An event that turns into very many entity updates counts for more than one that writes a single entity. Its size is known only when it is processed, so it is counted then, not when it is received. If such an event takes your organization over its allowance, the request that carried it is not refused and is still processed as everyday traffic. The requests you send after it are processed as part of the large load.
What a Large Load Looks Likeโ
From the outside, a large load runs in three steps:
- The beginning is processed at once, like everyday traffic, up to the allowance.
- A short pause. Once the load is recognized, the rest of it waits until the events already queued ahead of it have been processed. This usually takes a few minutes. Your requests are accepted as usual during the pause.
- A steady pace. The rest of the load is processed steadily until it is done. For a very large migration, this can take hours.
The results are the same as for everyday traffic: the same mappings, the same entities and the same monitoring codes. Follow the progress in the Monitoring tab, where the events appear as they are processed.
Your own everyday traffic joins the load. While a load runs, everything else the organization sends through the inbound API and file imports is processed as part of it, at the load's pace. A delta sync sent during a migration therefore takes longer than usual. Send what an end customer is waiting for with interactive inbound, which a large load does not delay.
There are no fixed processing times. How long a load takes depends on its size, on the work each event causes and on how busy the platform is.
Order of Eventsโ
Events for the same entity are processed in the order in which they were accepted. Moving an organization's events to the capacity for large loads does not change that:
- When a load starts. The rest of the load waits until the events accepted before it have been processed. That is the short pause of step 2.
- When a load ends. The organization's events count as everyday traffic again only after the load has been processed completely.
- If epilot switches this way of processing off while a load runs, the load is finished where it is, and the events that follow it are processed after it.
There is one exception. The pause of step 2 has a time limit of 30 minutes, and of 45 minutes for meter readings. If the events accepted before the load are still not processed by then, the load starts anyway, so that one stuck event cannot hold it back for ever. In that case an earlier event can be processed after a later one for the same entity. This is rare, and epilot is notified when it happens.
When Capacity Is Exhaustedโ
The capacity for large loads that run at the same time is limited. If all of it is in use at the moment your organization goes over its allowance, the request that goes over it is refused. This is rare. Requests within the allowance are never refused for this reason, and neither are interactive events.
Two more things are never refused for lack of capacity:
- Managed calls. The call to the external system has already been made when its result is queued, so the inbound event of a managed call is always accepted. The same holds when the allowance cannot be checked.
- File imports. An import waits for capacity instead of failing; see File Imports.
The Responseโ
A refused request is answered with HTTP 503, the header Retry-After: 60 and the error code BULK_CAPACITY_EXHAUSTED. This applies to POST /v3/erp/updates/events, to the deprecated v1 and v2 endpoints, and to replays.
HTTP/1.1 503 Service Unavailable
Retry-After: 60
Content-Type: application/json
{
"code": "BULK_CAPACITY_EXHAUSTED",
"message": "The capacity for large data loads is in use, and this organization is over its allowance for everyday traffic. Nothing of this request was accepted. Send the request again after the Retry-After interval."
}
The body is the error alone: it carries no correlation_id and no results. Nothing of the request was queued, and its deduplication ids were not used up, so the same request sent again is processed, not skipped as a duplicate.
What to Doโ
- Wait for the
Retry-Afterinterval, in seconds: 60 forBULK_CAPACITY_EXHAUSTED, 2 forINGEST_THROTTLED. Read it from the header rather than using a fixed delay. Retrying sooner does not help. - Send the same request again, unchanged.
- Hold back the requests that follow it until the retry is accepted, so that your events keep the order in which you send them.
- If the retry is refused again, wait again. Capacity becomes free as other loads finish. If refusals continue for a long time, contact epilot support.
Give every event a deduplication_id. A request refused whole can be sent again either way, and the retry creates no duplicates, because nothing of it was accepted. With deduplication ids, every retry, whether after a 503, a timeout or a lost response, is recognized and never processed twice.
These steps apply to both 503 responses, BULK_CAPACITY_EXHAUSTED and INGEST_THROTTLED.
Requests That Mix Interactive and Other Eventsโ
Interactive events do not count towards the allowance and are never refused for lack of capacity. When a request carries interactive events and others, for example interactive: true on the request while one of its events belongs to a use case that does not allow interactive processing, only the other events are refused. The response is still a 503 with Retry-After, and it lists every event:
{
"code": "BULK_CAPACITY_EXHAUSTED",
"message": "Some events of this request were not accepted; see results",
"correlation_id": "018f8e9b-5a1b-7c4e-9b2a-4f0f6c1d7a21",
"results": [
{
"event_id": "0199c3f2-8a41-7d0e-b6c7-2f1a9d3e4b55",
"status": "success",
"interactive": true,
"lane": "interactive"
},
{
"event_id": "0199c3f2-8a42-7a10-9c3e-5b7d2e1f0a66",
"status": "error",
"message": "This event was not accepted: the capacity for large data loads is in use, and this organization is over its allowance for everyday traffic. Send the event again after the Retry-After interval."
}
]
}
resultssays which events were accepted. Events withsuccesswere accepted. Events witherrorand a message that starts with "This event was not accepted" were refused.- Retry the refused events after the
Retry-Afterinterval. You can also send the whole request again if every event carries adeduplication_id: the interactive events that were accepted are then recognized as duplicates and come backskipped. - A close is not applied. A
correlation_complete: trueon such a request is ignored with the reasonretry_requested, as the response'scorrelation_closesays. The retry closes the correlation; see What Became of the Close.
Events that asked for interactive processing but run as everyday traffic, because the organization is over its interactive rate limit, count as everyday traffic here as well and can be refused like any other.
When Ingestion Is Busyโ
A second 503 has nothing to do with the size of your allowance. It is answered with the header Retry-After: 2 and the error code INGEST_THROTTLED in two cases:
- The service that checks your organization's allowance is busy and cannot answer. The request is then not accepted, rather than accepted unchecked: accepting it unchecked could put the events of a running large load out of order.
- The queue that receives the events throttled the request.
Both are transient. This response replaces the 500 with the message Unknown API error that the inbound endpoints returned when the queue throttled a request.
HTTP/1.1 503 Service Unavailable
Retry-After: 2
Content-Type: application/json
{
"code": "INGEST_THROTTLED",
"message": "The service that checks the allowance of this organization for incoming events is busy and could not answer. Nothing of this request was accepted. Send the request again after the Retry-After interval."
}
When the body is code and message alone, nothing of the request was accepted or queued, and its deduplication ids were not used up. Send the same request again after the stated time. The retry is accepted and creates no duplicates.
When some events of the request were accepted, the body also carries the message Some events of this request were not accepted; see results, the correlation_id and results, as for a mixed request:
- Events with
successare queued. Do not send them again. - Events with
errorwere not accepted. Send them again after theRetry-Afterinterval. - With a
deduplication_idon every event you can send the whole request again: the accepted events come backskipped. - Events leave in the order of the request, and the send stops at the first event that is not accepted. An event that was not accepted is never followed by an accepted one on the same queue.
- A
correlation_complete: trueon such a request is not applied. The retry carries it again.
This happens when the queue took a part of a large request before it throttled, and when the allowance could not be checked for a request that also carries interactive events, which are never counted and are accepted as usual.
A replay is answered in the same way when nothing of it was queued. A replay that was throttled part of the way is answered with 200 and a Retry-After header; see Replaying Events.
Size Limitsโ
One event can turn into many entity updates, for example when its mapping creates one entity per item of a list in the payload. Two limits bound that:
| Limit | Value | Response when exceeded |
|---|---|---|
| Updates one event may produce | 5,000 | 422. The event's result has status: "error" and code: "EVENT_TOO_WIDE". The other events of the request are processed. |
| Updates the events of one request may produce together | 25,000 | 422 with code: "REQUEST_TOO_WIDE". The whole request is refused and nothing of it is queued. |
An update is one entity that the event's mapping writes. An event that produces exactly 5,000 updates is accepted; 5,001 are refused. Meter readings are not counted, and file imports are not limited this way.
{
"correlation_id": "018f8e9b-5a1b-7c4e-9b2a-4f0f6c1d7a21",
"results": [
{
"event_id": "0199c3f2-8a41-7d0e-b6c7-2f1a9d3e4b55",
"status": "success"
},
{
"event_id": "0199c3f2-8a42-7a10-9c3e-5b7d2e1f0a66",
"status": "error",
"code": "EVENT_TOO_WIDE",
"message": "This event would produce 5,001 updates, more than the maximum of 5,000 updates per event. Split it into several events."
}
]
}
{
"code": "REQUEST_TOO_WIDE",
"message": "This request would produce at least 30,000 updates, more than the maximum of 25,000 updates per request (and 5,000 per event). Nothing of it was accepted. Split it into several requests."
}
What to do:
EVENT_TOO_WIDE: send the data of the event again as several smaller events. Nothing of the event was queued. The event also appears in monitoring with the codeEVENT_TOO_WIDE.REQUEST_TOO_WIDE: send the events again in several smaller requests. The deduplication ids of the request were not used up. The count in the message is a lower bound, because the events after the one that brought the request over the limit are not counted.- Do not retry unchanged. Unlike a
503, a422for size is answered the same way every time.
A result can also carry the code EVENT_EVALUATION_LIMIT: the mapping of the event ran for more than 25 seconds, or built a generated sequence of more than 4 million items, and was stopped before its updates could be counted. Nothing of the event was queued. Send its data in several smaller events.
The inbound event of a managed call cannot be refused with a 422, because the call has already answered. If it produces more than 5,000 updates, it is not processed, and it appears in monitoring with the code EVENT_TOO_WIDE. Change the call or its mapping and run the call again.
File Importsโ
A file import counts as a whole. A large import is recognized as a large load when it starts, and its rows are processed as described above.
If the capacity for large loads is in use when an import needs it, the import does not fail. It waits and tries again by itself every 60 seconds, for up to 6 hours in total, and nothing needs to be done. It waits in the same way when the organization's allowance cannot be checked for a moment.
- Its status stays
PROCESSING. - While it waits, its
progresscarries awaiting_for_capacityobject. epilot's import screens show such an import as Waiting for capacity. - Rows already sent stay sent. When capacity is free, the import continues where it stopped, and no row is sent twice.
- You can abort the import while it waits.
{
"import_id": "7c1e4b2a-93d5-4f08-b6a1-2e9d0c4f5a37",
"status": "PROCESSING",
"progress": {
"processed_rows": 0,
"total_rows": 250000,
"waiting_for_capacity": {
"since": "2026-11-16T08:12:04Z",
"retry_at": "2026-11-16T08:47:05Z",
"gives_up_at": "2026-11-16T14:12:04Z"
}
}
}
| Field | Meaning |
|---|---|
since | When this wait began. It does not change while the same wait continues. |
retry_at | When the import tries again, if known. The import tries again about every minute. |
gives_up_at | The earliest time at which the import fails if capacity is still not free: the end of the 6 hours the import waits in total, counted across all of its waits. It can fail a little later, never earlier. |
waiting_for_capacity is present only while the import waits. It is gone as soon as the import continues, when it is aborted, and when it ends. An import that never had to wait does not carry it. The field is the same on the single-import and list endpoints of both import APIs, /v2/erp/imports and /v2/imports.
Only if no capacity becomes free within the 6 hours does the import fail, with the error code BULK_CAPACITY_EXHAUSTED and this message:
the capacity for large data loads stayed in use for the whole waiting time and the import was stopped. Rows already sent stay sent, please run the import again later
Run the import again later: validate it, then execute it. The rows that were already sent are sent again and update the same entities.
Bulk Import Windowsโ
A large load is recognized by itself, but only after it has used up the organization's allowance: its first events are processed as everyday traffic, and the rest waits for the short pause of step 2. If you know when a migration or a full resync will run, book a bulk import window for it. During the window, your organization's events run on their own capacity from the first event, and a pause between two batches of the load does not end it.
A window applies to the whole organization, to every integration and file import, not to a single integration.
In the Integration Hubโ
- Open an integration and go to the Monitoring tab.
- In the Bulk import window card, choose Plan a bulk import.
- Pick the start date and time (in your time zone), the duration in hours, and add a note on what the load is, for example "Initial migration of all contracts".
- Save. The card shows the window as Scheduled, and as Active while it runs.
You can edit a window until it starts, and cancel it at any time. Cancelling an active window does not stop or slow down what is already running: the load continues and ends by the normal rules.
While your organization's events run on their own capacity, whether in a window or because a load was recognized, the card says so, with the number of updates still waiting:
- Getting your lane ready: earlier updates are finishing first; this is the short pause of step 2.
- Your updates run on their own lane: the load is being processed.
Booking, editing and cancelling need the permission to manage integrations. Everyone who can view integrations sees the card.
Rules for a Windowโ
| Rule | Value |
|---|---|
| Length | at most 24 hours |
| Start | at most 30 days ahead, not in the past |
| Note | required, at most 500 characters |
| Windows per organization | one at a time; booking a new one replaces a window that has not started |
The capacity for windows is shared by all organizations. If too many windows overlap, a booking is refused; choose another time, or contact epilot support for a very large migration.
Through the APIโ
The same window can be read, booked and cancelled with the Integration Toolkit API. The organization is the one of the token.
PUT /v1/bulk-imports/window
Content-Type: application/json
{
"starts_at": "2026-11-24T06:00:00Z",
"ends_at": "2026-11-24T18:00:00Z",
"note": "Initial migration of all contracts"
}
GET /v1/bulk-imports/windowreturns the window, if there is one, with itsstatus(scheduled,activeorended), and the organization's current processing inlane.DELETE /v1/bulk-imports/windowcancels the window.- A refused booking is answered with
409and anerror_code, for examplewindow-active(an active window cannot be replaced, only cancelled),window-too-longortoo-many-windows.
See the API reference for the full schema.
Planning a Migrationโ
- Book a bulk import window for the migration. See Bulk Import Windows. For an initial load of a whole customer base, also tell your epilot contact or epilot support ahead of time.
- Verify the mapping on a subset first. Use the mapping simulation endpoint and a small batch. A mapping error in a large load repeats for every event.
- Handle both
503responses before you start, and give every event adeduplication_id. See What to Do. - Keep events and requests within the size limits. At most 5,000 updates per event and 25,000 per request. See Size Limits.
- Plan for hours, not minutes. Allow for a very large migration to take hours, and schedule cut-overs and go-live dates accordingly.
- Keep end customers' flows on interactive inbound. Registrations and contract additions that happen during a migration are not delayed by it. Do not set
interactive: trueon the migration itself; see Common Mistakes. - Rehearse in a sandbox with a subset, and run the full volume in production. Large loads in a sandbox are processed at a deliberately low rate; see Sandbox Organizations.
Next Stepsโ
- Sandbox Organizations โ how large loads behave in a sandbox
- Inbound Getting Started โ the inbound API and its responses
- Interactive Inbound โ pushes an end customer is waiting for
- Investigating Events โ traces and replay