# Integration Toolkit

Introduction to the Integration Toolkit for integrating ERP systems with epilot

The Integration Toolkit is epilot's opinionated integration platform for building, monitoring, and maintaining ERP integrations. It combines **standardized entity and event schemas**, **JSONata-based mapping**, **a dedicated inbound ingestion API**, **configuration APIs**, **central monitoring**, and **packaged Blueprints/Apps** into a single, cohesive system.

![Integration Toolkit High Level](/img/erp-toolkit-high-level.jpg)

The toolkit is optimized for energy-sector use cases: end-customer self-service via portals, meter reading propagation, contract lifecycle management, and billing account synchronization.

> **Info: The Integration Toolkit does not remove the need for middleware. It standardizes the interface between epilot and your middle layer, radically reducing the complexity compared to direct API integration.**

## Components

The Integration Toolkit is composed of the following components. Each plays a specific role in the integration lifecycle.

| Component | Description | Status |
|-----------|-------------|--------|
| **[Integration Hub](#integration-hub)** | Admin UI in epilot 360 to configure and monitor integrations | Stable |
| **[ERP Integration API](#erp-integration-api)** | CRUD API to manage integrations, use cases, and mappings | Stable |
| **[ERP Inbound API](#inbound-api)** | Dedicated API to receive and simulate inbound ERP events | Stable |
| **[Use Cases](./use-cases.md)** | Documented integration flows with testing support | Stable |
| **[Interactive Registration](./interactive-registration.md)** | Correlation-id contract and a dedicated lane for inbound pushes an end customer is waiting for | Stable |
| **[Core Entities](/docs/entities/core-entities)** | Standardized entity schemas for mapping targets | Stable |
| **[Changesets](/docs/entities/changesets)** | Pending attribute updates that wait for ERP confirmation or human approval | Stable |
| **[Core Events](/docs/integrations/core-events)** | Standardized event payloads for outbound notifications | Stable |
| **[Webhooks](/docs/integrations/webhooks)** | Push events from epilot to ERPs via core events | Stable |
| **[Pollable Outbound](./pollable-outbound.md)** | Pull-based outbound delivery — ERPs poll a queue instead of receiving webhooks | Stable |
| **[JSONata Mapping](#jsonata-mapping)** | Transformation language for inbound and outbound data | Stable |
| **[File Proxy](./file-proxy.md)** | Serve files from external archives on demand without migrating them into epilot | Stable |
| **[Outbound File Delivery](./outbound-file-delivery.md)** | Deliver files referenced by epilot events to external document APIs | Stable |
| **[Managed Calls](#managed-calls)** | Synchronous external API calls with JSONata mapping via connector integrations | Stable |
| **[Secure Proxy](#secure-proxy)** | Route HTTP requests through epilot's secure proxy for static IP egress or VPN access | Stable |
| **[Monitoring](./monitoring/overview.md)** | Central event logging, stats, cross-system traces, replay, and the code reference | Stable |
| **[Alerting & Notifications](./monitoring/alerting.md)** | Per-integration alert rules, anomaly baselines, digests, and delivery to email/in-app | Stable |
| **[ACK Tracking](./monitoring/acks.md)** | ERPs acknowledge processed events, closing end-to-end delivery visibility | Stable |
| **[Blueprints](https://marketplace.epilot.cloud/en/blueprints)** | Packaged, installable integration setups | Stable |
| **[Apps](https://marketplace.epilot.cloud/en/apps)** | Custom automation actions and portal extensions for ERP logic | In progress |

### Integration Hub

The Integration Hub is the administrative UI within epilot 360 (accessible at `/app/integrations`). It provides a guided wizard for activating ERP integrations, selecting self-service use cases, configuring mappings, running integration tests, and monitoring health.

### ERP Integration API

The `/v2/integrations` CRUD API centralizes all integration configuration in one place:

- **Environments** -- API URLs, tenant IDs, secrets
- **API tokens** with scoped roles and permissions
- **Inbound use cases** with entity mappings
- **Outbound use cases** with event mappings
- **Managed call use cases** for synchronous external API calls (connector integrations)
- **Secure proxy use cases** for HTTP requests routed through a static IP or VPN
- **Associated Apps and portal extensions**

Integrations support two types: `erp` (default, for standard ERP flows) and `connector` (for complex proxy integrations with external APIs).

See the [Configuration Guide](./configuration.md) for API details.

### Inbound API

Two dedicated endpoints for receiving ERP data:

| Endpoint | Purpose |
|----------|---------|
| `/v3/erp/updates/events` | Receive raw ERP data as events for processing |
| `/v2/erp/updates/mapping_simulation` | Dry-run events to verify mappings before going live |

See the [Inbound Integration Guide](./inbound/getting-started.md) for setup instructions.

### Use Cases

A **use case** is a documented ERP integration flow: a description of how epilot and the ERP work together to implement a specific scenario (e.g., "Keep Contract In Sync" or "Meter Reading Submission"). Each use case defines:

- Which core entities need to be mapped
- Which core events are sent or received
- Expected ERP behavior and response
- Testing via simulated events or webhook test triggers with ACK confirmation

See the [Use Cases](./use-cases.md) page for a complete list of inbound and outbound integration flows.

### File Proxy

The [File Proxy](./file-proxy.md) enables epilot to serve files from external document systems (e.g., ERP archives, DMS) on demand. Instead of migrating file content during inbound sync, file entities are created with a `custom_download_url` pointing to the proxy. When a user views the file, the proxy fetches the document from the external system in real time using a declarative, multi-step HTTP configuration.

The upload direction is documented separately as [Outbound File Delivery](./outbound-file-delivery.md). It subscribes to Event Catalog events, fans one event out into per-file deliveries, fetches the referenced epilot files, and runs a declarative external HTTP workflow with independent retries and monitoring.

### Managed Calls

Managed Calls enable synchronous API calls to external partner systems with built-in authentication, JSONata mapping, and optional inbound routing. They are configured as `managed_call` use cases within `connector`-type integrations.

Key capabilities:
- **Authentication** — OAuth2 client credentials, API key, or bearer token with automatic token management
- **JSONata mapping** — Transform request and response payloads using JSONata expressions
- **Inbound routing** — Optionally queue the response to the inbound pipeline for async entity processing
- **Secure proxy support** — Route calls through a static IP or VPN connection when needed

See the [Configuration Guide](./configuration.md#managed-call-use-cases) for setup details.

### Secure Proxy

The Secure Proxy routes HTTP requests through epilot's dedicated proxy infrastructure, providing either **static IP egress** (for IP-allowlisted external APIs) or **VPN access** (for customer private networks). It acts as the single authenticated gateway between epilot and customer networks.

- **Static IP mode** — Routes outbound requests from a fixed, allowlistable IP address
- **Secure Link mode** — Routes through a VPN connection for access to private networks
- Domain whitelisting and CIDR-based IP allowlisting enforce strict access control

See the [Configuration Guide](./configuration.md#secure-proxy-use-cases) for setup details.

### JSONata Mapping

[JSONata](https://jsonata.org/) is the core transformation language for defining mappings between epilot's standardized entity/event schemas and ERP-specific data models. It is used across:

- Inbound event processing (ERP to epilot entity mapping)
- Outbound webhook payloads (epilot event to ERP format)
- Pollable outbound payloads, transformed at enqueue time ([Payload Mapping](./pollable-outbound.md#payload-mapping))
- The Map Data flow building block

### Monitoring and Alerting

Every event the toolkit processes — inbound, outbound, file proxy, managed call and
secure proxy alike — is recorded as a monitoring event and surfaced in the Integration
Hub's **Monitoring** tab, with a matching API for automation.

- **[Event stream and stats](./monitoring/overview.md)** — filter by level, code, use
  case or correlation id; success rates and time series per lane
- **[Code reference](./monitoring/codes.md)** — every monitoring code, what it means
  and what to do about it
- **[Traces and replay](./monitoring/investigating.md)** — follow one business
  operation across systems, inspect the captured request and response, and reprocess
  events
- **[Alerting](./monitoring/alerting.md)** — per-integration rules with anomaly
  baselines, plus scheduled digests, delivered to email and in-app
- **[ACK tracking](./monitoring/acks.md)** — ERPs confirm they processed an event,
  closing end-to-end visibility
- **[External monitoring events](./monitoring/external-events.md)** — push your
  middleware's own processing steps in, so one trace spans both systems

## Architecture

![Integration Toolkit Architecture](/img/integrations/overview.svg)

## Integration Directions

### Inbound (ERP to epilot)

Push data from your ERP into epilot. Typical flows:

- Synchronize customer master data (contacts, accounts)
- Import contracts and subscriptions
- Submit meter readings and consumption data
- Update billing and payment information

[Inbound Integration Guide](./inbound/getting-started.md)

Inbound ingestion is asynchronous. For the flows where an end customer is waiting for the push to land — portal self-registration, adding a contract — use [Interactive Registration](./interactive-registration.md): your middleware sends the portal's correlation id with the push, the events run on a dedicated lane, and the portal waits until exactly that data has been written.

### Outbound (epilot to ERP)

Push epilot events to your ERP via webhooks. Typical flows:

- New customer registrations and portal sign-ups
- Contract changes (move-in, move-out, termination)
- Self-service requests (IBAN changes, installment adjustments)
- Meter reading submissions

Outbound events use [Core Events](/docs/integrations/core-events). They can be delivered through [Webhooks](/docs/integrations/webhooks), made available through [Pollable Outbound](./pollable-outbound.md), or used to [deliver referenced files](./outbound-file-delivery.md) to an external API. JSONata transforms simplify event and file payloads before delivery. The same events can also start [Automation Flows](/docs/automation/event-catalog-trigger) inside epilot, for example to notify a service team or update related entities alongside the ERP delivery.

For ERPs that cannot receive webhooks (firewalled, on-prem, or batch-oriented systems), [Pollable Outbound](./pollable-outbound.md) offers a pull-based alternative: your middleware polls a queue inside the ERP Integration API on its own schedule and acknowledges items once consumed.

For self-service flows that need ERP confirmation before the local value is changed (IBAN updates, address changes, payment method changes, etc.), use [Changesets](/docs/entities/changesets). Set `edit_mode: external` on the attribute, configure a matcher, and enable `apply_changesets: true` on the outbound webhook — the ERP receives the proposed new value directly on the entity fields, and the changeset auto-clears when the ERP confirms via `?direct=true` inbound sync.

## Operational Model

The Integration Toolkit and your middle layer have clearly separated responsibilities.

| Toolkit (epilot) | Middle Layer (yours) |
|-------------------|---------------------|
| Stable schemas (core entities/events) and versioning | ERP connectivity (on-prem, VPN, ERP API calls) |
| Deduplication and idempotency | Domain-specific validation required by ERP |
| Inbound ingestion at scale with retries and mapping execution | ACK back to epilot for "processed" state |
| Central monitoring, replay, and retention | |
| Configuration APIs and UI | |

> **Tip: Most integrations involve a middleware (e.g., SAP CPI) that handles ERP-specific connectivity and validation. The toolkit standardizes the epilot side so your middleware only needs to speak one protocol.**

## Integration Patterns

Two primary patterns exist. The right choice depends on your ERP's capabilities.

![Pull vs Event-Driven Integration Patterns](/img/integrations/pull-vs-event-driven.svg)

### Event-Driven (Direct)

Your ERP or middleware sends events directly to the Integration Toolkit when data changes.

**Best for:** systems that support webhooks, need real-time sync, and require minimal transformation beyond mapping configuration.

### Pull-Based (Delta Sync with Middleware)

A middleware layer periodically queries your ERP for changes and pushes them to the Integration Toolkit.

**Best for:** ERPs without webhook support, complex multi-source aggregation, or scenarios requiring custom validation and retry logic.

<details>
<summary>Comparison table</summary>

| Requirement | Event-Driven | Pull-Based |
|-------------|--------------|------------|
| ERP supports webhooks | Required | Not required |
| Real-time sync needed | Best fit | Depends on polling interval |
| Complex transformations | Limited to mapping config | Full flexibility |
| Custom validation logic | Limited | Full flexibility |
| Aggregating multiple sources | Not supported | Supported |
| Minimal infrastructure | Best fit | Requires middleware |

</details>

> **Tip: Many integrations use a **hybrid approach**: event-driven for simple, real-time updates (e.g., contact changes) and pull-based for complex scenarios (e.g., aggregating invoice data from multiple systems).**

## Getting Started

1. **Create an integration** -- register your ERP connection via the [Configuration API](./configuration.md) or Integration Hub
2. **Configure use cases** -- define inbound/outbound data flows with entity mappings
3. **Test mappings** -- use the mapping simulation endpoint to validate before going live. See the [Mapping Examples](./mapping-examples.md) repo for a TDD approach
4. **Send events** -- push data from your ERP to the inbound API
5. **Monitor** -- track processing status, errors, and ACKs in the Integration Hub

Continue to the [Inbound Integration Guide](./inbound/getting-started.md) for step-by-step setup.