A Jira reporting pipeline outgrew AWS Lambda's 15-minute limit because the API could only return full data. How we moved heavy loads to one scheduled Glue job and kept Lambda for incremental work.
In a 16-domain AWS data mesh I built for a UK FinTech, one of the first pipelines pulled Jira data to feed Power BI reporting. It looked like a simple job for AWS Lambda. It was not, and the way we fixed it shaped how we split work between Lambda and Glue across the whole platform.
The problem: an API with no partial loads
We started with a Lambda function to extract the full Jira dataset. The third-party API we used returned the complete data on every run. There was no way to request only what had changed.
As the data grew, the run took longer than 15 minutes. That is the hard maximum runtime for an AWS Lambda function, so the job could not finish. No amount of tuning inside Lambda fixes a limit like that.
Moving to Glue, and the cost lesson
We moved the full load to AWS Glue, which has no 15-minute ceiling and is built for heavier data processing. That solved the runtime problem.
Then we looked at what a single Glue job cost. Glue is billed for the compute time each job uses, so every extra job, and every case where Lambda and Glue both process the same data, adds to the bill.
The fix: one scheduled Glue job for the heavy work
Instead of creating a Glue job per process, we combined all of the long-running, frequently scheduled processes into one Glue job that runs on a schedule. Nothing heavy runs twice, and nothing is paid for in both Lambda and Glue.
The split we run today
- Glue: full-refresh loads and long-running processes, combined into one scheduled job.
- Lambda: short and incremental work, such as change data capture (CDC) and slowly changing dimension (SCD Type 2) history.
One example on the Lambda side tracks team membership history as SCD Type 2. It updates its data in place incrementally, so each run is small and finishes well within Lambda's limits.
Making the consolidated job reliable
Putting several processes into one job raises the stakes: if it fails, more data goes stale at once. These are the patterns that keep it dependable:
- Concurrent threading for API calls inside the job, to keep total runtime down.
- Parquet compaction, so each dataset folder holds one compacted file instead of thousands of small ones.
- Write first, delete second: the new compacted file is written before old files are removed, so a crash never leaves a folder empty.
- Incremental datasets use a high-watermark read from Athena, with a bootstrap mode for the first full load.
Monitoring, so failures are never silent
- An EventBridge rule alerts on any Glue job that fails, times out or stops.
- CloudWatch alarms watch Lambda errors.
- A daily freshness monitor checks 27 Athena tables and flags any that are stale, critically stale or empty.
- All alerts route to one SNS topic, so one place tells the team what needs attention.
When to use Lambda, and when to use Glue
- If a run can exceed 15 minutes, or the source only returns full data, use Glue.
- If the work is short, incremental or event-driven, use Lambda.
- Do not process the same dataset in both.
- Group heavy work into scheduled Glue jobs rather than creating one job per process.
Most pipeline cost and reliability problems are design decisions, not tool problems. Choosing where each workload runs, and monitoring it from day one, is the core of how we build data platforms at EXCO.