When Monte Carlo doesn't natively cover your orchestrator, you don't wait for a connector. Build a real Monte Carlo integration for it from two composable building blocks, configured per capability, landing on the same core product surface as native.
Custom Integrations is the umbrella for any source Monte Carlo doesn't ship a native connector for. For ETL and orchestration, it lets you stand up a real integration against a tool we don't cover yet, with the same jobs, tasks, run history, lineage, and alerting you'd get from a native source. It's in Private Preview and available on Enterprise plans.
Two building blocks sit underneath it: the Push Ingest API and a Custom connector. They're not rival products. A custom connector is built on push; it's the batteries-included way to use it.
The custom connector is the AI-first path: a guided conversation with Claude generates the scaffolding, tests, and deployable artifacts, and every step has a documented, code-based path if you'd rather build by hand. Push is the direct path: your own code, posting events to the API.
You can lean on one block or mix both, and you wire each capability on the source independently. The rest of this guide walks the choices: push versus connector, augment versus net-new, how to mix and match the parts, what each Monte Carlo feature needs, the API surface, and the Claude workflow that builds it.
01 The two building blocks
Both paths end the same way: canonical events arrive at Monte Carlo and resolve into one model of jobs, tasks, and runs. The difference is how much you build and operate yourself, and who runs the collection.
Your code extracts run and metadata events from the source and posts them to the Push Ingest API, from wherever it runs. Maximum flexibility, minimum footprint. No agent required.
// you build the collector, you invoke it
A guided Claude conversation scaffolds a connector, packages it as an image on the Generic Agent (Monte Carlo's container-based collection runtime, deployed in your environment), and registers it as a first-class connection. Under the hood it still pushes the same events, but you get structure, tests, and Monte Carlo runs collection on a schedule for you.
// Claude (or by hand) → image on the Generic Agent → registered connection
Why the connector exists. Pure push is powerful but open-ended: you figure out hosting, scheduling, and what to send. The custom connector is the prescriptive version. It's generated for you (by Claude, or built by hand), it deploys on the Generic Agent, it registers like a native source, and Monte Carlo drives its schedule. Same events on the wire either way.
02 What you're building
Custom Integrations covers two situations. The building blocks are identical; what differs is whether you're filling a gap or standing something up from scratch.
Your source is already integrated, natively or otherwise, but some coverage is missing: a slice of lineage, a metadata field, a run signal the native path doesn't capture.
Push only what's missing. Everything the existing integration already does keeps working untouched; the pushed data layers on top of it.
Monte Carlo has no integration for the tool yet. You're creating the whole thing: declaring its jobs and tasks, sending its runs, wiring its lineage.
Use push directly, or generate a custom connector for the batteries-included path. Either way the source becomes a first-class integration.
Both cascade to the same options. Whether you're augmenting or starting fresh, the parts available to you are the same: push the data directly, tag queries to build lineage, or route webhooks. The next two sections are how those parts fit together.
03 What you send
An ETL integration is fed by two event types. They're independent: send one, the other, or both, and capabilities light up accordingly.
Declares the things that exist: your jobs, their tasks, and an optional grouping layer (a project, folder, or workspace). Sent on a routine interval. This is what makes assets appear in the catalog, and what carries declaration-time lineage and the task graph.
One event per run transition: start, success, failure, retry. Carries status, timing, and the assets the run read and wrote. This is what populates run history and performance, and what turns a failure into an alert.
Metadata is the foundation. Declaring your jobs and tasks is what puts them in the catalog and on the map, and it's the natural place to start. From there it's additive: run events layer on status, run history, performance, and failure alerts; a run's inputs and outputs, or query tagging, add lineage. Send what you have. Each capability lights up as its inputs arrive, and a gap leaves that one feature empty rather than breaking the integration.
04 Composition
Lineage, run events, and webhooks are separate levers. You pick the combination that fits how your tool exposes data.
The direct path is to include the assets a run touched, its inputs and outputs, on the run event.
If your collector can't get at those, there's a second path: query tagging. Tag the queries your jobs run with the job's ID (mcd_job_id), and Monte Carlo's query parser resolves them back to the right job, exactly as it does for a native ETL source. You can also include the connection's resource UUID (mcd_resource_id) to make resolution deterministic when the same job ID spans more than one connection. The two paths are complementary, you can do both, and the metadata declaration still has to exist for tagged queries to attach to.
Polling already catches failures on its own. The agent collects run details about once an hour, so a failure surfaces within roughly that window. For near-real-time alerting, point your tool's webhook at Monte Carlo when a run finishes. For a custom ETL integration the webhook is a lightweight trigger, there's no separate webhook feature to configure:
Point your tool's webhook at Monte Carlo with just enough to identify the source. That triggers Monte Carlo to invoke your connector on the Generic Agent, which fetches the full run details and pushes the complete event, all inside your own network.
If your tool can't send the request exactly as Monte Carlo needs (Matillion's Webhook Post component, for example), a small forwarding proxy in front bridges the gap. Either way the webhook only triggers collection; the connector still produces the event.
Native and push aren't exclusive either. A source can be collected natively for some capabilities and topped up with push for others, for instance native query parsing for lineage alongside pushed run events. The unified model doesn't care which path produced a given row.
05 Capabilities
Every capability below works for a custom ETL integration; what populates depends on what you can send. None of it is all-or-nothing, each capability lights up on its own as its inputs arrive. The one exception is metadata: it's the foundation everything else attaches to, so it's where you start.
| Capability | What it shows | What it needs |
|---|---|---|
| Catalog & search | Jobs and tasks as first-class, searchable assets | Metadata declaration |
| Job & task pages | Per-job detail: metadata, owner, schedule, the tasks inside | Metadata; richer as you add owner, schedule, description |
| Run history & performance | Per-run status and duration, at job and task grain | Run events, with timing for duration |
| Job-failure alerts | An alert when a run fails | A run event reporting a terminal (failed) status |
| Table lineage | Cross-system lineage to and from the assets a run touches | Run inputs/outputs, or query tagging |
| Task lineage | The dependency graph of tasks within a job | Task-level dependency data; available for Airflow today, not yet for other ETL sources. |
| Tagging | Monte Carlo tags applied to ETL assets, like any other asset | Nothing from the source; you apply tags in Monte Carlo |
The mental model: capabilities are additive and decoupled. Start with metadata so your jobs are on the map, then layer on run events for status, run history, performance, and failure alerts, and add lineage by emitting a run's inputs and outputs or by tagging queries. A gap in one area never degrades another, it just leaves that one feature empty until you send what it needs.
What it doesn't cover yet. Task-level lineage is available for Airflow today, not yet for other ETL sources.
06 The API surface
Push is a small REST surface, authenticated with an integration key. Two endpoints back ETL integrations, mirroring the two event types.
POST /ingest/v1/etl/runs
One call per run state change. Required fields: a job identifier, a run identifier, a status, and an event time. Task events additionally link to their parent run, and runs can carry their input/output assets for lineage.
POST /ingest/v1/etl/metadata
Definition-time declaration of jobs, tasks, and optional grouping, sent on an interval. Required: a job identifier and a name. The container it belongs to is set when you connect the integration; schedule, owner, and description are optional enrichers.
The pycarlo SDK wraps both so you work with Python objects instead of raw HTTP. Its IngestionService exposes send_etl_metadata(...) and send_etl_runs(...). Authentication is your integration key, and events are sent in batches.
from pycarlo.core import Client from pycarlo.features.ingestion import IngestionService svc = IngestionService(client=Client()) # 1) declare the job once (definition-time metadata) svc.send_etl_metadata([{ "job_source_id": "nightly_load", "name": "Nightly load", "schedule": {"kind": "CRON", "cron_expression": "0 2 * * *"}, }]) # 2) emit a run event on every state change svc.send_etl_runs([{ "job_source_id": "nightly_load", "run_source_id": "2026-06-03T02:00", "status": "failed", # a failed status drives a failure alert "event_time": "2026-06-03T02:14:11Z", "outputs": [{ # optional: lineage "fully_qualified_name": "analytics.public.orders", "asset_type": "TABLE", "role": "OUTPUT", }], }])
Read the field names as shape, not contract. Exact signatures, the full status set, and batching limits are pinned in the integration docs and the published JSON Schemas. The point here: declaring a job and emitting a run is a handful of fields, and everything optional maps directly to a capability in the table above.
07 AI-first
The expected way to build a custom connector is a guided conversation with Claude, using a packaged skill. AI is how the integration gets built, not a layer bolted on afterward. The same artifacts can be produced by hand, every step is documented, but the AI-first path is the default. Claude runs at build time only, generating the connector code; your pipeline data and run events never go to the model when the integration runs.
Point Claude at the source. It walks a short interview: what the tool is, how it authenticates, which capabilities you want, how its runs and tasks are shaped.
Claude scaffolds the connector and leaves you two small methods to implement: fetch metadata and fetch run details from your tool. Their output is what gets pushed through pycarlo.
A unit-test harness runs your events through the contract and reports which Monte Carlo capabilities they'll populate, and which they won't, like a checklist. Missing lineage? Claude points you at query tagging. Want more coverage? Prompt it to get the next test passing.
The connector is built into an image on the Generic Agent base and deployed in your environment. Register it, and it behaves like any other integration.
Monte Carlo runs collection on a schedule; add a webhook to trigger it in near-real-time. The same connector supports both.
Build it once, by conversation. Reference integrations ship as worked examples, public and reproducible, to read, fork, or hand back to Claude as a starting point for your own source. And because the skill understands the capability model, it actively steers you toward the right composition, declare metadata for the catalog, tag queries when direct lineage isn't reachable, rather than leaving you to discover it.
08 In short
Push when you want control or the source is unreachable from our side. Generate a custom connector when you want the batteries-included, AI-first path with Monte Carlo running the schedule. Augment an integration that's missing coverage, or stand up a net-new one.
Emit lineage or tag for it, route a webhook to trigger collection and let the agent fetch the rest, collect natively and top up with push. Start from metadata so your jobs are on the map, then send whatever else you have, and each capability lights up as its inputs arrive. However you assemble it, the result is a real Monte Carlo integration, monitored, alerting, and connected to everything downstream.
Learn more in the docs →