Job runs & lineage

Every job run and every column flow, attached to the data product it feeds.

FluenBox collects job execution statuses and lineage from Airflow, PySpark and Python, resolves each one to its data product, and turns failures and gaps into findings with an owner.

Signals in

Three ways your pipelines run. One place where their status lands.

Each run reports its status, timings, volumes and the tables it read and wrote. A binding table resolves the DAG task, the Spark job or the script to the data product it serves.

AirflowDAG and task runs, retries, durations, SLA misses
PySparkJobs and stages, input and output tables, row counts
PythonScripts and services reporting runs and lineage through the API
FluenBox Binding table customer_360_daily → Customer 360 enrich_customers → Customer 360 load_crm.py → Customer 360 orders_enriched → Order history
Run health per product

The business sees whether its product ran, not which DAG failed.

Every job that feeds a product shows its last runs, freshness and volume on the product page. A failure becomes a finding assigned to the Responsible role, and the score’s operations dimension reflects it.

Customer 360 · jobs feeding this productLast 14 days
customer_360_daily Airflow
Fresh · 06:14
enrich_customers PySpark
Failed · stage 3
load_crm.py Python
1.2M rows · normal
Finding opened: enrich_customers failed at stage 3Assigned to the data steward (R) · consumers informed (I) · Oregon AI: “schema change in the CRM source, column segment removed”
Lineage

From source to dashboard, down to the column.

Lineage reported by the runs is joined into one graph per data product. Column-level paths show where personal data goes and where it is masked, which is evidence for the GDPR controls.

CRM databaseemail · segment Web eventsKafka topic Billingon-premise enrich_customersPySpark customers_cleanemail → email_hash orders_enrichedtable Customer 360Python · score 4.6 Sales BI Churn model
SourceJobTableData productColumn path: email, hashed before Customer 360
What it keeps current

Per data product, without anyone filling in a form.

Run health

Success, failure, retries and duration of every job that feeds the product.

Freshness and volume

When the product was last refreshed and whether the row counts look normal.

Column lineage

Where every column comes from and where it goes, including masking of personal data.

Data quality results

Rule results recorded against the run that produced the data.

Cost per run

Joined with the FinOps hub, so a heavy job shows up in the product’s cost.

Findings with an owner

A failed run or a broken lineage path becomes a finding routed by RACI role.

The FinOps hub

Connect your own pipelines.

A 30-day pilot on a real part of your business, with your Airflow, Spark and Python jobs reporting from the first week.

Book a 30-day pilot See how it works