{"id":109233,"date":"2026-04-16T14:13:00","date_gmt":"2026-04-16T14:13:00","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=109233"},"modified":"2026-04-08T12:56:35","modified_gmt":"2026-04-08T12:56:35","slug":"when-and-when-not-to-use-llms-in-your-data-pipeline","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/ai\/when-and-when-not-to-use-llms-in-your-data-pipeline\/","title":{"rendered":"When, and when not, to use LLMs in your data pipeline"},"content":{"rendered":"\n

\u201cLet\u2019s add LLMs to the pipeline\u201d has become a familiar refrain in modern data teams, but turning that idea into production reality is where things often go wrong. While large language models can unlock powerful capabilities, they\u2019re just as likely to introduce unnecessary cost, latency, and complexity when misapplied. Drawing on real-world experience building large-scale ML systems, this guide breaks down exactly where LLMs belong in your data pipeline and, just as importantly, where they don\u2019t.<\/strong><\/p>\n\n\n\n

Someone on your team just got back from a conference<\/a>. Or maybe they spent 20 minutes on LinkedIn. Either way, the message lands in Slack: \u201cWe should be using LLMs in our pipeline.\u201d<\/p>\n\n\n\n

I\u2019ve watched this play out more times than I can count – and I get it. LLMs (large language models)<\/a> are genuinely exciting. However, what usually follows that Slack message is not so exciting. It’s a weeks-long detour that ends with an infrastructure bill that makes the VP of Engineering visibly uncomfortable. A pipeline that\u2019s three times slower than before. And a compliance team asking questions nobody prepared for.<\/p>\n\n\n\n

I\u2019ve spent the last few years building production ML (machine learning)<\/a> systems that touch hundreds of millions of users – customer journey optimization, support analytics, behavioral data<\/a> at petabyte scale. The clearest lesson from all of it: the teams that get LLMs right aren\u2019t the ones who use them everywhere<\/em>. They\u2019re the ones who are ruthlessly specific about exactly where they use them.<\/p>\n\n\n\n

This article is my attempt to give you a concrete framework for making that call – not in theory, but instead in the kind of production environment where things break at 2am and someone has to explain the AWS<\/a> bill.<\/p>\n\n\n\n

Why LLMs are being over-applied<\/h2>\n\n\n\n

The appeal of LLMs is obvious. It can read a messy support ticket, figure out what the customer actually wants, classify it, summarize it, and draft a reply – all in one API<\/a> call. Three years ago that would\u2019ve been four separate models and a custom NLP (natural language processing) pipeline<\/a>. Now, it\u2019s a weekend prototype.<\/p>\n\n\n\n

The problem, however, shows up later. Token costs<\/a> scale with volume, and latency is real. The output is probabilistic (meaning identical inputs don\u2019t always produce identical outputs), and when something goes wrong, debugging an LLM call is nothing like debugging a SQL transformation<\/a>. There\u2019s no stack trace for \u201cthe model decided to phrase it differently this time.\u201d<\/p>\n\n\n\n

The trap springs during the proof-of-concept phase. The demo looks great. The prototype handles edge cases, tolerates messy input and produces readable output…but the demo doesn\u2019t run on ten million rows. It doesn\u2019t have a 50-millisecond latency budget. And nobody from compliance is watching when you build it.<\/p>\n\n\n\n

That gap between demo performance and production fitness is where most LLM projects run into trouble.<\/p>\n\n\n\n

Where do LLMs genuinely belong in a data pipeline?<\/h2>\n\n\n\n

Let\u2019s start with where LLMs actually earn their keep…<\/p>\n\n\n\n

1. Unstructured text parsing and enrichment<\/h3>\n\n\n\n

If your pipeline ingests free-form text<\/a> (support tickets, reviews, medical notes, legal documents, call transcripts), LLMs are doing something rule-based systems<\/a> genuinely cannot.<\/p>\n\n\n\n

A regex<\/a> extracts a phone number. It doesn\u2019t extract the emotional tone of a complaint, the buried risk clause in paragraph 14 of a contract, or the intent behind \u201cthis thing keeps doing the thing again.\u201d In one large-scale support analytics deployment I worked on, LLM-based enrichment of call transcripts gave us classification accuracy that would\u2019ve taken hundreds of hand-crafted rules to approximate – rules that would have needed updating every time a new product launched. Instead, we simply updated a prompt.<\/p>\n\n\n\n

2. Semantic search and RAG<\/h3>\n\n\n\n

This is the clearest LLM win in data pipelines, full stop. When users need to query a knowledge base or documentation corpus using natural language – and when keyword matching falls short – RAG architectures<\/a> are purpose-built for the job.<\/p>\n\n\n\n

RAG isn’t even just a fallback option – for retrieval tasks<\/a>, it\u2019s the architecturally correct first choice. It’s traceable (the model can cite its sources), updatable without retraining (swap the vector database, not the model), and doesn\u2019t silently degrade when your data changes. In a separate piece I\u2019ve written on RAG-first architectures, I argue that roughly 70% of production AI use cases are better served by RAG and smart prompting than by fine-tuning. Retrieval is the clearest example of why.<\/p>\n\n\n\n

3. Natural language to SQL<\/h3>\n\n\n\n

What surprises people is that LLMs are actually quite good at translating natural language questions into SQL – as long as the context is right<\/em>. Pass in your schema, a few example queries, and relevant table descriptions, and you\u2019ve turned a hallucination-prone<\/a> model into a reliable query generator.<\/p>\n\n\n\n

4. Explaining what your models found<\/h3>\n\n\n\n

Statistical models are good at detecting anomalies<\/a> but terrible at explaining them to whoever needs to take action. An LLM can take the outputs of your monitoring system<\/a> – such as churn spike, support ticket surge, or regional drop-off – and turn them into a paragraph a product manager can actually use.<\/p>\n\n\n\n

This is also a relatively low-volume use case. You\u2019re not narrating a billion events; you\u2019re narrating a handful of alerts. The cost profile is fine.<\/p>\n\n\n\n

\n
\n
\n
\n

Future-proof database monitoring with Redgate Monitor<\/h3>\n
\n Multi-platform database observability for your entire estate. Optimize performance, ensure security, and mitigate potential risks with fast deep-dive analysis, intelligent alerting, and AI-powered insights. <\/div>\n <\/div>\n Learn more & try for free<\/a>\n <\/div>\n <\/div>\n<\/section>\n\n\n

Where LLMs do not<\/em> belong<\/h2>\n\n\n\n

Now the harder part. LLMs can technically be wedged into almost any pipeline task, but the question is whether you should<\/em>. Usually, the answer is no.<\/p>\n\n\n\n

1. Deterministic transformations<\/h3>\n\n\n\n

If your transformation has a single correct answer – convert this timestamp to UTC, calculate a 30-day rolling average, join two tables on customer ID – use SQL or<\/em> a dataframe operation. That\u2019s it. LLMs add probabilistic variability to tasks that require deterministic<\/a> correctness, and they do it at orders-of-magnitude higher cost and latency.<\/p>\n\n\n\n

I\u2019ve seen teams run LLM calls to \u201cclean\u201d structured fields that a two-line Python<\/a> function would have handled the same way, for a thousandth of the price. The justification is usually that the LLM handles edge cases better. Sometimes that\u2019s true. However, it more often means the edge cases haven\u2019t been properly defined. The fix here is to define them – not hand the problem to a model that charges per token.<\/p>\n\n\n\n

2. High-volume, low-latency enrichment<\/h3>\n\n\n\n

Petabyte-scale pipelines and LLM token costs do not coexist peacefully. If you\u2019re enriching a billion user events per day with API calls, the infrastructure bill will not survive a quarterly review. <\/p>\n\n\n\n

The lesson I\u2019ve learned working at that scale: LLMs belong at the edges, on the high-value, low-volume decisions. Not in the hot path where you need sub-millisecond responses at fractions of a cent per million operations. The table below emphasizes the magnitude of this gap:<\/p>\n\n\n\n

Approach<\/strong><\/td>Latency<\/strong><\/td>Cost \/ 1M rows<\/strong><\/td>Reliability<\/strong><\/td><\/tr>
LLM (GPT-4 class)<\/td>800ms\u20132s<\/td>$800\u2013$2,000<\/td>High (with guardrails)<\/td><\/tr>
LLM (smaller \/ local)<\/td>200\u2013500ms<\/td>$50\u2013$200<\/td>Medium<\/td><\/tr>
Classical ML<\/td>10\u201350ms<\/td>$5\u2013$20<\/td>High (deterministic)<\/td><\/tr>
SQL \/ Rule-based<\/td><10ms<\/td><$1<\/td>Very High<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Your numbers will vary by model, provider, and workload, but the order-of-magnitude differences are real. They don\u2019t just disappear with optimization.<\/p>\n\n\n\n

3. Regulated or compliance-sensitive outputs<\/h3>\n\n\n\n

If your pipeline produces outputs subject to regulatory audit<\/a> – such as fraud scores, healthcare decisions and financial calculations (to name a few) – LLMs are the wrong foundation. Regulators want explainability, reproducibility, and deterministic behavior under identical inputs. LLMs don\u2019t naturally offer any of those things.<\/p>\n\n\n\n

That doesn\u2019t mean LLMs are useless in regulated environments. They can draft, summarize, and flag items for human review. But the authoritative output – the number that goes in the record, the decision that gets logged – needs to come from something auditable.<\/p>\n\n\n\n

\n
\n
\n
\n

Write accurate SQL faster in SSMS with SQL Prompt AI<\/h3>\n
\n Write or modify queries using natural language, get clear explanations for unfamiliar code, and fix and optimize SQL with ease – all without leaving SSMS. <\/div>\n <\/div>\n Learn more and try for free<\/a>\n <\/div>\n <\/div>\n<\/section>\n\n\n

A practical decision framework<\/h2>\n\n\n\n

Before you add an LLM to any pipeline stage, ask these four questions. They’ve saved me from making several expensive decisions.<\/p>\n\n\n

\n