The five parts of a signal engineering setup

A few months ago I wrote an introduction to signal engineering. That post was about the why: what changed in the ad platforms, why late conversions became weak signals, and why this goes beyond another server-side tag manager setup.

This post is about what you actually have to build.

I am writing it for the person who gets handed the request. Usually that is a head of data or a CTO, and usually the request arrives from marketing sounding small. “Can we send earlier conversions to Meta?” It is not small. It is five connected assets, a feedback loop that has to keep running, and one timing constraint that quietly breaks the way the rest of your pipelines are designed.

The good news is that every one of the five parts has a lean version. You can start small. What you cannot do is skip one, and the part most teams skip first is the cheapest one to build.

We also ran a webinar on signal engineering that covers the wider picture at a higher level. If you want the full context before going into the architecture, watch the recording here. This post goes deeper on the build.

Signal engineering is a loop, not a pipeline

The most useful thing you can do before scoping anything is to get the shape of the project right. This is not a build you implement, test, hand over and then put on maintenance. It is a cycle that has to keep running. Four stages.

Predict. This is a constant stream of decisions, not a scheduled job that produces a score once. For every user you are asking: do we send a signal for this person at all, to which platforms, and with what value? Most of the time the answer is no. That is fine. The decision itself is the product.

Translate. Once you have a prediction you turn it into a signal. You take everything you know about that user at that moment, which might be almost nothing for someone who landed thirty seconds ago, or a full CRM record if they are logged in, and convert it into the shape the platform expects. The platforms behave differently, so this is also where the per-platform quirks get baked in.

Activate. You send the signals out through the conversion APIs. Quality checks happen before the send, because a payload the platform silently rejects is worse than no payload at all. And you log on your own side what you sent, which turns out to be the part that makes the next stage possible.

Learn. How are the campaigns performing, what is your signal to conversion ratio, which values worked, which platforms responded. You can wait for the final conversion and then look back at what you predicted. Often you cannot, because the conversion is months away, so you also need proxies that tell you earlier whether a prediction was any good. Whatever you learn goes back into the prediction stage.

The loop framing matters for a practical reason. A pipeline needs capture, transform and delivery. That is three things. A loop needs those three plus the two that let it close: a record of what you sent, and a way to check how it turned out. Those last two are the ones that get cut when the project runs late, and cutting them means you have built a delivery mechanism with no way to tell whether it works.

This is also why “we set up signal engineering last quarter” is not a sentence that makes sense. You do not finish it. All four stages need continuous work, and the setup gets better mainly because the loop has been running long enough to teach you something.

The 24 hour window is what breaks your normal pipeline design

Here is the constraint that catches data teams off guard, because it does not come from your systems. It comes from the platforms.

Your core pipelines almost certainly run once a day. For everything else you own that is the right call. Reporting, finance, product analytics, none of it gets meaningfully better by running hourly. Signal delivery does, and by a lot.

Platform	Practical freshness	Hard limit	What it means for your pipeline
Meta (Conversions API)	Real time to hourly. Noticeable decline for events sent more than two hours after they happen	Beyond 24 hours, attribution and delivery start breaking. Maximum 7 day click attribution window	Hourly is the floor. Streaming if you can get there
Google (offline conversion import)	Tolerant of lag. Conversions are backdated to the click date	90 day import window	Daily works technically, but your feedback loop inherits the full lag

Both of those limits are hard rather than advisory, and they are worth reading in the source rather than taking from a blog post. Meta rejects a server event outright if its event_time is more than seven days old. Google drops offline conversions uploaded more than 90 days after the associated click, and if you are using enhanced conversions for leads rather than GCLIDs, that window shortens to 63 days. The shorter number is the one that catches out B2B teams with long cycles.

Google looks generous until you follow the consequence. Yes, you can import a conversion 60 days after the click. You have also just tied your campaign feedback loop to 60 days. Change a creative or a value rule and you wait weeks before enough late data arrives to judge it, while spend continues. And late conversions are deep funnel events, which makes them scarce. If your closed won volume sits below the platform learning thresholds, you cannot optimise on them anyway and you fall back to an earlier proxy event. So even on the forgiving platform you usually end up wanting to signal earlier.

The practical translation is a pipeline that runs hourly for a narrow slice of your data. When I say this in scoping calls, data teams often hear “streaming project” and start estimating a quarter. It usually is not one. Most of the source data is already landing continuously, and what changes is the schedule and the dependency guarantees for one specific path. The real work is upstream: your capture layer has to actually be fresh enough to support an hourly job, and that is the first thing worth testing rather than assuming.

The other reason to name this early is organisational. Time criticality is not something a data team feels by default, because nobody on the team sits inside an ad account watching a learning phase reset. Marketing feels it constantly and usually cannot explain why in terms that survive a technical review. Putting the number in hours on the table in the first scoping meeting is the cheapest thing you can do for this project.

---

1. Capture: behavioural, zero party and first party data

The first asset is the one everyone assumes they already have. You track your website, so you have behavioural data, so capture is done. It usually is not, because signal engineering asks two things of this layer that normal analytics never asks: it has to carry identifiers that mean something to an ad platform, and it has to combine three different kinds of data about the same person.

Identifiers are the hard gate

Check this before you scope anything else. This is the one place where the project can be dead on arrival, and finding out in week six is expensive.

You need identifiers doing two separate jobs.

Identifier	Job	Where it breaks
Internal user or device id	Ties events together into one subject you can predict about	Sessions never get stitched, so behaviour never accumulates into a picture
Click id (GCLID for Google, fbclid or the fbp cookie for Meta)	Lets the signal be matched back to a click on the platform side	Not captured at landing, or captured but dropped before it reaches storage
Hashed email or phone	Matches a user without a click id and survives longer than one	Requires the person to hand it over, and a consent setup that covers this use

The click id is the one to worry about. If it is not captured the moment someone lands and then persisted somewhere durable, there is nothing to send back later. The platform has no way to connect your conversion to its click, and no amount of modelling downstream fixes that. Google’s own setup guide for offline conversions is blunt about this: the GCLID has to be stored at the point of capture, because it cannot be reconstructed afterwards. I would treat “can we prove we have a click id on a meaningful share of paid sessions” as the go or no go question for the whole project.

Hashed PII is worth having as a second path where consent allows it, because it keeps working for people who converted through a route that lost the click id. It is a complement, not a replacement.

One clarification on that gate, because it is about activation rather than about value. Traffic you cannot send back is not traffic that is useless to you. Aggregated, anonymised, single session behaviour needs no user id and no cookie, and it is legitimate input for working out which early behaviours predict value. You cannot activate on those people, but they can still teach the model what to look for. Keep the two questions apart: can we send a signal for this person, and can this person inform how we decide.

Zero party data is the underrated input

Behavioural data is what teams already have. Zero party data is what they under-use, and in this setup it is disproportionately valuable.

The reason is the timing constraint from the last section. At the moment you have to decide, a few minutes after someone landed, your behavioural evidence is tiny. Three or four page views. That is not much to predict a customer lifetime value from. Zero party data is dense by comparison. One answer in a form can carry more predictive weight than an entire session of clicks.

The explicit kind is the good kind: what people type into your forms. Company size, use case, budget range, what they are looking for. The user has told you directly, so there is nothing to infer.

Then there is the inferred kind, which is what content someone consumed, which videos they watched, which comparison page they sat on. This is genuinely useful but I would handle it more carefully than most teams do. You can find a pattern in anything if you look hard enough, and a behaviour that correlates with buying in a backtest is not the same as a behaviour that means anything. Treat these as candidate features that a model gets to accept or reject, not as rules you write yourself because they look sensible.

First party data is the third input and the least controversial. CRM records, subscription state, support history, whatever your backend knows about a known customer. If someone is logged in, you are not predicting from four page views. You are predicting from an account history.

Why this lands in the warehouse

All three inputs land in a warehouse. There is a leaner version where you try to do all of this inside a server side tag manager and never touch one, and I would not recommend it as your foundation. It works for one specific delivery case that I will come to in part three, but as the base of the setup it costs you the three things this layer exists to do.

Identity stitching. Anonymous behaviour, the click id and the known customer record have to resolve to one subject. This is the join that makes everything downstream possible, and it needs somewhere with actual history to happen.

Data evaluation. Zero party data in particular arrives messy. Free text fields, inconsistent dropdown values, job titles in five languages, people typing nonsense to get past a gate. Some of it needs reclassifying and some of it needs filtering out entirely, because an outlier that reaches a model does more damage here than in a dashboard.

History. The prediction stage runs on how users behaved months ago. A tag manager has no memory. This is the constraint people notice last and regret most.

In practice this is where most of the initial work in these projects goes, and it is rarely the interesting part. When we scope a signal engineering setup, the capture layer is usually the thing that determines the timeline, not the modelling.

The lean version: one event stream landing in your warehouse with a persisted click id and a stable user key, joined to whatever your CRM already holds. No streaming architecture, no CDP, no identity resolution vendor. If you can join a click to a customer record and a revenue number, you have enough to start.

2. Prediction: what your data science team needs from you

I am not going to tell you which model to use. That is a post of its own, and it is genuinely your data science team’s expertise rather than yours. Talk to them.

What is yours, and what mostly does not happen, is everything around the model: the brief, the dataset and the honest assessment of whether you have enough history for any of this to work yet.

How to brief your data science team

Most briefs for this arrive as a sentence. “Can you build us a model that predicts high value users.” A data scientist can do something with that, but they will make five assumptions on your behalf and at least two of them will be wrong in ways that only surface after the pipeline is live. Here is what the brief actually needs to say.

The target and its horizon. Not “will this user convert.” State what you want predicted and over what period. Predicted 12 month value is a different model from probability of purchase within 30 days, and both are different from probability of reaching second subscription renewal. Pick the one your business actually funds decisions with.

What exists at the moment of decision. This is the constraint that gets broken most often. A model trained on full user history will happily lean on features that only exist days after the point where you have to act. It scores beautifully in a backtest and is useless in production. Give the team a feature availability cut-off, in minutes, and be explicit that anything after it is off limits.

The cost of being wrong in each direction. A false positive spends budget teaching a platform to go find more people like someone who was never going to buy. A false negative costs you volume. Only the business can price that trade, and if you do not state it, the team will optimise for whatever metric is conventional in their world rather than the one that is expensive in yours.

A minimum firing rate. The model has to say yes often enough to keep the platforms out of the learning phase. As a working floor we usually recommend 50 or more signals per week per ad set on Meta, and 30 or more per week per campaign on Google. A precise model that qualifies twelve users a week is worse than useless here, because it starves the thing it is supposed to feed. Put the floor in the brief as a requirement, not a preference. If a thin funnel or consent loss means you cannot clear it, the answer is to signal on an earlier or higher funnel event rather than to keep pushing volume the platform has no way to learn from.

How the answer gets used. The output has to be computable at decision time and it has to map onto a value you can actually send. If it only runs as a batch job across a full history, it does not fit the architecture, no matter how good it is.

The feature store is your deliverable

Your job here is to hand over a dataset, not a problem.

In practice that means a feature store at user grain: one subject per row, attributes that resolve to true, false or a simple number. Campaign source, device, geography, what happened in the session, what the person typed into your forms, what your CRM already knows about them.

The part that is easy to skip and painful to retrofit is point in time correctness. You need to be able to reconstruct what was known about a user at a specific moment, not just what is known about them now. Without it you cannot enforce the feature cut-off from the brief, and leakage walks straight into your training set.

What comes back matters as much as what goes in. You want a set of criteria, not a mystery score. Someone arriving from a particular campaign, on a particular device, who answered a particular way in a form, has a high probability of becoming a valuable customer. Criteria are inspectable, they can be argued about with the marketing team, and they can be carried into the delivery layer. A probability that only exists inside a notebook cannot become a mapping table in a tag manager, which is a problem you will meet in the next section.

Historical depth is the real prerequisite

Your model is capped by your history, and this is where a lot of teams find out that they are not ready.

If you have been capturing behaviour, form answers and outcomes together for a couple of years, prediction gets much easier, and it matters most exactly where you need it most. Long conversion journeys need long history, because you cannot learn what predicts a purchase that happens in month six without having watched several cohorts get to month six.

If you started capturing three months ago and your sales cycle is six months, no model is going to rescue you. You have not observed a single complete journey.

That is not a reason to stop. It is a reason to invert the order. Start capturing properly now, ship signals based on criteria that humans agreed on, and let the tracking and validation layers generate the training data you do not have yet. The loop bootstraps itself, which is the main practical argument for building all five parts even when the middle one is barely there.

The lean version: no model at all. A rules list agreed between marketing and data, built on behaviours you already believe in, shipped through the delivery layer and tracked properly. It will not be optimal. It will be measurable, and measurable is the thing that makes the modelling worth funding later.

3. Delivery: the time window decides your architecture

Delivery is where a prediction becomes an actual API call. It is also the part of the setup with the least obvious answer, because there are two quite different ways to build it and the thing that decides which one you need is not your stack or your team’s preferences. It is how quickly you need to signal after the user does something.

Both are legitimate, and plenty of setups end up running both for different use cases, which I come back to at the end of this section. But they behave very differently, and starting with the one that does not fit your timing is expensive.

Path A: a mapping table in the server side tag manager

The fast path. You take the criteria that came out of the prediction stage and push them into your server side tag manager as a lookup table. Google’s tag manager is the dominant one, so assume that unless you have a reason not to. If you have never worked with a server container, Simo Ahava’s introduction to server side tagging is still the clearest walkthrough of how clients, tags and the event object fit together.

Each row is a combination of criteria. Against it sits whether that combination qualifies for a signal, and what value or value factor to attach. Behavioural data flows from your website into the tag manager, the incoming event gets checked against the table, and on a match the signal goes straight out to the platform conversion APIs.

Note what is not happening here. The model does not run in the tag manager. What gets deployed is the model’s output, frozen into a lookup. This is exactly why the previous section insisted that you want criteria back from your data science team rather than a score that only exists in a notebook. A probability you cannot express as rules cannot be deployed this way at all.

What you get is speed that no pipeline can match. The decision happens inside the request, seconds after the behaviour. If your case is someone with four page views who is about to leave and never come back, this is the only architecture that genuinely serves it.

What you pay for is that the mapping table is a disconnected copy of a decision made somewhere else. It drifts. Someone has to own updating it, reviewing it and keeping track of which version was live when, and that person is usually busy. There is limited room for quality checks before a send, no retry when a platform API fails, and no natural record of what went out unless you deliberately build one, which is the subject of the next section.

Path B: a warehouse pipeline on an hourly schedule

The controlled path. Scoring happens in the warehouse, qualified signals get written to a table, and a job picks up new rows and delivers them to each platform’s ingestion endpoint, the Conversions API on Meta and the Data Manager API on Google. The only unusual thing about it is the schedule.

This is the better environment for almost everything that is not raw speed. You can check payloads before they leave: are the identifiers actually present, is the value inside a sane range, have we already sent this one. You can retry and rerun when a platform API fails, and it will fail. You can backfill after an outage. You have the full feature set available at query time rather than whatever fits in a lookup table.

The signal log also falls out of this design almost for free, instead of being a separate thing you have to remember to build.

The constraint is a latency floor. Even a flawless hourly job means someone who landed at 10:05 gets signalled at 11:00. Against Meta’s degradation curve that is usually acceptable. Against a requirement to signal within minutes of first behaviour, it is not.

One dependency worth testing before you commit: an hourly signal job is only as fresh as the capture layer feeding it. If your event data lands in the warehouse once a night, an hourly delivery pipeline sends stale decisions on schedule. That is the thing to verify first, and it is usually a smaller problem than teams fear.

Which one you need

The question is not which tool is better. It is how early you need to signal.

	Server side tag manager	Warehouse pipeline
Time to signal	Seconds	Up to an hour
Decision input	Pre-computed criteria, pushed as a lookup	Full feature set, available at query time
Quality checks before sending	Limited	Full
Retries, reruns, backfills	Not really	Yes
Record of what was sent	Has to be built separately	Falls out of the pipeline
Maintenance	Mapping table drifts and needs an owner	Versioned with the rest of your code
Right when	You must signal within minutes of first behaviour	An hour of latency is acceptable

Most of the setups we build end up on the warehouse path, and the ones that do not are all the same situation: short consideration cycles, high bounce, a purchase decision that happens inside one session. If your users routinely decide in twenty minutes, an hourly pipeline is structurally too slow and no amount of engineering discipline fixes it.

The two paths are also not exclusive, and mature setups often run both. A thin early signal goes out through the tag manager while the user is still on the site, and a richer, corrected signal follows from the warehouse once more is known. If you do this, two things become mandatory rather than optional. You need a shared event id so the platforms deduplicate properly instead of counting the same user twice, and you need both paths writing into the same signal log, or your validation layer will only ever see half the picture.

The deduplication mechanics are worth knowing before you design this rather than after. Meta matches on event_id together with event_name, only considers events that arrive within 48 hours of each other, and when a browser event and a server event land within about five minutes it keeps the browser one. If your second signal arrives three days later carrying a corrected value, it is not a deduplicated update. It is a separate event.

There is a second constraint here, and it is asymmetric between the platforms. On Meta you can revise a value upward, but you cannot walk it back down. On Google you can correct in both directions, so an overstated prediction can be fixed once you know better. That difference decides how you set the opening value. On Meta, start conservative and let the later signal raise it, because an optimistic first value is permanent and you will be optimising against it for the life of that conversion. On Google you have room to open closer to your actual estimate, because you can restate it. This is the single most important reason to design the two signal setup deliberately rather than discovering it in production.

The lean version: one platform, one hourly job, one criteria set, delivered from the warehouse. Resist building a multi-platform delivery abstraction before you have sent a single signal. The abstraction you would design today is not the one you will need after the first round of validation.

4. Signal tracking: the part almost nobody builds

Signal tracking means that every time you send a signal to a platform, you also write down what you sent, what you predicted and what you knew at that moment.

It is the cheapest of the five assets to build and the one that gets cut first when a project runs late. It is also the one that makes the other four worth having, because without it you have no way to find out whether any of this works. This is the part that is missing in almost every setup we are asked to review, and it is usually the reason the team cannot answer why performance changed.

You track what you sent, not what the platform got

The instinct is to skip this because the platform already reports on conversions. It does, but its reporting is not a record of your decisions. It is aggregated, deduplicated, modelled, delayed, and shaped by attribution logic you do not control. It will not tell you what you tried to send and it silently dropped. It will certainly not tell you what you predicted, because it never knew.

You are not logging conversions here. You are logging decisions. Every time your system decided to send a signal, that decision had inputs, a model behind it, a value attached and an outcome at the API. All four of those need to survive, because in three months someone will ask why the campaigns changed and the only honest answer will come from this table.

There is a second thing worth logging that goes beyond the obvious, and it is optional in phase one. Consider keeping a sample of the users you decided not to signal. If you only ever record the yeses, you can measure precision, meaning how good the signals you sent turned out to be. You can never measure recall, meaning how much good traffic you left on the table because your criteria did not catch it. The second question becomes important faster than most teams expect.

A pseudo signal table

What the table looks like depends on what metadata you have, so treat this as a starting shape rather than a spec. The principle underneath it is simple: you send a handful of fields to the platform, and you keep everything else for yourself.

Most of that is unsurprising. Five columns do the heavy lifting later and are the ones teams leave out.

Column	Why you need it later
model_version and criteria_set_version	You will change the model. Without a version stamp on every row you cannot compare the new one against the old one, and every analysis in the next section slices by this
predicted_value alongside signal_value	Business rules almost always sit between the two: caps, floors, currency conversion, a per platform factor. If you only keep what you sent, you cannot tell whether a bad outcome came from a bad model or a bad rule. This matters more once you are feeding value based bidding, where the number itself is what the platform optimises against
feature_snapshot	The record of what was actually known at decision time. It lets you ask counterfactual questions later without rebuilding history, and it is nearly impossible to reconstruct after the fact
delivery_status and platform_response	A rejected signal and an underperforming signal look identical in your data if you do not record this. It is the cheapest possible protection against a week spent debugging a model that was fine all along
consent_state	Explains gaps in coverage, and you will be asked about it eventually

The identifiers are what make the whole thing joinable. A GCLID is not a strong identifier on its own, but it is the thread that later connects a signal to the campaign it came from and, if you also write click ids against your customer records, to whether that person eventually converted. That join is the entire basis of the next section.

One thing deserves more than a row in that table. The platforms send quality feedback back through the same APIs you deliver on, and match rate is the number to watch: of the signals you sent, how many the platform could actually tie to a person. Log it, trend it by platform and by identifier type, and treat lifting it as continuous work rather than a fix you do once. It is the metric that tells you whether your capture layer is quietly degrading.

It will also never reach 100%, and it is worth setting that expectation before someone reads the first report and concludes the project failed. Because you are acting early, on thinner evidence than you would have if you waited, the gaps here are wider than the ones you tolerate in attribution. That is the trade you accepted when you decided to signal inside the window instead of after it. The goal was never a perfect signal. It was a signal good enough to steer the platform, and one you can keep improving.

Snapshot or append

Two ways to maintain this. You can snapshot the current signal table on a schedule, or you can append a row for every send.

Append wins, for a reason specific to what this table is for. Validation is longitudinal by nature. You are comparing a decision made in January against an outcome that lands in June, so you need the decision preserved exactly as it was made, not as it looks after three model updates. An append only table gives you that automatically, and what you end up with is an event log of your own decision making.

Two rules that follow from it. Never update a row in place, because corrections belong as new rows, and rewriting history destroys the thing you built this for. And keep the log longer than your conversion horizon. If you care about 12 month value, a signal log with 90 day retention cannot answer a single question you have.

The lean version: one append only table, written by the same job that does the delivery. No separate service, no streaming, no new infrastructure. If you took the tag manager path in the previous section, this is the piece you have to build deliberately, because it will not appear on its own.

5. Validation: closing the loop

Validation is where you find out whether your predictions were any good, and it is the stage that turns four disconnected components into a system. Everything before this point is machinery for sending signals. This is the part that tells you whether the signals deserved to be sent.

The mechanics are less complicated than the thinking. You already have the pieces: a log of every decision, identifiers that connect those decisions to campaigns and customers, and eventually an outcome. The work is deciding what counts as a good outcome and how long you are willing to wait to find out.

Your proxy metrics come from your business model

The obvious way to validate a prediction is to wait for the truth. Did this user become valuable, yes or no. The problem is that the truth arrives long after you can do anything with it. If your meaningful outcome is 12 month value, a model shipped in January gets its first honest review the following January, and by then it has been quietly wrong for a year.

So you need checkpoints in between. Not one proxy, a ladder of them, each arriving earlier and meaning less than the one after it.

What sits on that ladder is entirely determined by how your business makes money. This is the part nobody can hand you as a list, because a good proxy in one revenue model is noise in another. If you sell software on subscription, the ladder runs through product usage and the states a user moves between: activated, using the thing that predicts retention, still there in month two. If you run a content platform, it runs through consumption depth and return behaviour. If you sell to businesses with a long cycle, it runs through the qualification stages your sales team already tracks.

Revenue model	Early proxy	Middle proxy	Final value
Subscription software	Activation, first use of the feature that predicts retention	Retained into second billing period	12 month retained revenue
Content or media	Consumption depth, return visit inside a week	Subscription or registration started	Lifetime value
Repeat purchase commerce	First purchase	Second purchase within 90 days	12 month contribution margin
B2B with a sales cycle	Lead accepted by sales	Opportunity created	Closed won contract value

Four things make a proxy worth using, and it is worth checking all four before you commit to one.

It has to arrive meaningfully earlier than the final outcome, otherwise it buys you nothing. It has to actually correlate with final value in your own historical data, which you can test rather than assume. It has to be measurable at signal grain, meaning you can join it back to a single decision. And it should not be something your own signals can inflate.

That last one is easy to miss. Suppose your early proxy is whether the user came back to the site. You start sending signals, the platform responds by retargeting those users harder, return visits go up, and your proxy improves without a single person becoming more valuable. You have built a loop that congratulates itself. Any proxy that sits close to ad exposure needs this check.

From signal to outcome

The join is the straightforward part. Your signal log gives you one row per decision, carrying the prediction, the model version and the identifiers. Those identifiers connect outward in two directions: to the platform, for campaign context on where the user came from, and to your own customer records, for what the person eventually did.

It is worth being clear about what those two directions are, because they run at completely different speeds. Your attribution reporting is slow, batched and backward looking, moving at the pace of settled reporting windows. Your prediction side is fast, per user and forward looking, with every signal you send nudging the next bid long before any report exists. They do not need to run on the same clock. They only need to meet in one place, and that place is here.

The output is a table at signal grain where each row holds what you predicted, what you sent and how far up the proxy ladder that user eventually climbed. That table answers the questions you actually care about. Which criteria produced valuable customers and which produced traffic. Whether the users you signalled outperform the ones you did not. What share of signals ever reach first revenue.

Run this on two cadences rather than one. A fast loop weekly against the early proxies, which is what you use to catch a model that has started drifting or a criteria set that stopped working. A slow loop quarterly against final value, which is the only thing that tells you whether the fast loop is measuring the right stuff. Teams that run only the fast loop optimise confidently toward a proxy that turned out not to matter.

Version everything or you cannot compare anything

You will change the model, the criteria, the values and the platform configuration, usually more often than you expect. Every one of those changes invalidates a comparison unless you recorded which version was live for which signal, which is why the previous section put version stamps on every row.

There is a subtler problem underneath. The obvious way to evaluate a change is before and after: run version one for six weeks, switch to version two, compare. That comparison is confounded, because the platform is learning and adjusting underneath you the entire time. Some of the difference you measure is your model and some is the algorithm responding to a changed signal mix, and after the fact you cannot separate them.

The way out is concurrent pools rather than sequential periods. Run more than one criteria set at the same time, keep them separated in the signal log, and compare them against each other rather than against last month. This also gives you somewhere to test criteria you are unsure about without betting the whole account on them, which matters more than it sounds once you realise how much your signals shape what comes back.

The signals also change your traffic

Everything above treats the campaigns as a fixed backdrop that your signals get measured against. They are not. When you tell a platform you want more users like these, it changes its targeting, and the traffic arriving tomorrow is different from the traffic that arrived yesterday.

That has a direct consequence for validation. Campaign level metrics move for reasons that have nothing to do with whether your model was right. Volume can drop while quality rises. Cost per acquisition can worsen while contribution margin improves. If you read those numbers without knowing when your criteria changed, you will draw confident conclusions from noise.

So the validation layer needs a campaign level view alongside the signal level one, with your model and criteria versions marked on the timeline. When performance shifts, the first question is whether it lines up with something you changed. This is harder than the signal grain analysis and needs more data work, which is why I would build it in a second phase rather than upfront.

The deeper version of this problem is that your signals actively shape your audience, and an audience can narrow to the point of saturation. I wrote about that dynamic in the introduction post, and it deserves its own piece rather than a paragraph here.

The lean version: one weekly query joining your signal log to whatever outcome data you already have, grouped by criteria set. No dashboard, no framework, no new tooling. The first time you run it you will find something that changes what you send, and that is the loop closing for the first time.

Start lean, but build all five

Everything above describes the mature version. Almost nobody starts there, and you should not try to. Each of the five parts has a small version that works, and the small versions still form a loop.

Part	The lean version	What breaks if you skip it entirely
1. Capture	One event stream in the warehouse with persisted click ids and a stable user key, joined to your CRM	Nothing can leave the building. No identifiers, no signals
2. Prediction	A rules list agreed between marketing and data	Nothing immediately. You just stop getting better
3. Delivery	One platform, one hourly job, one criteria set	You have judgement and no way to act on it
4. Signal tracking	One append only table, written by the delivery job	You can never learn anything. The loop cannot close
5. Validation	One weekly query joining the signal log to whatever outcomes you have	You optimise blind and find out how it went in a year

The asymmetry in that table is the argument. Prediction is the part every team wants to start with, and it is the only one you can genuinely defer, because criteria written by two sensible humans will get you moving. Signal tracking is the part every team defers, and it is the one you cannot, because it is what turns the next twelve months into evidence instead of anecdotes.

Build capture, rules based delivery and a signal log, and you have a working loop with no model in it at all. That loop then generates exactly the dataset a model needs, which means your phase two is funded by your phase one. Build it in the other order and you get a model nobody can evaluate, sending signals nobody recorded, into campaigns that changed for reasons nobody tracked. The setups we work on now start from capture and the signal log and leave the modelling for later, because that sequence compounds and the reverse one does not.

One thing does not have a lean version, and that is ownership. Someone has to own this, and the awkward part is that it sits across marketing and data with no obvious home, which I went into in the introduction post. The version of that problem specific to your side of the table is worth naming though. This is not a delivery request that arrives from marketing, gets built, and goes into maintenance. The loop only closes if the people who own campaign performance and the people who own the pipeline look at the same numbers on a regular schedule. If those two groups never meet, you will have built four of the five parts and validation will quietly never happen.

Wrapping up

I hope this gives you a concrete picture of what you are actually being asked to build when signal engineering lands on your desk.

The main things to take away:

• It is a loop, not a project. The four stages keep running, and the setup gets good mainly because it has been running for a while, not because it was designed well on day one

• The timing constraint drives the architecture. Decide how quickly you need to signal before you decide anything else, because that one answer determines your delivery path and most of what sits behind it

• Prediction is the part you can defer. Signal tracking is the part you cannot, and it is also the cheapest of the five

• Your proxy metrics come from your revenue model, not from a best practice list. Nobody can hand you that ladder

• Build all five parts small rather than two of them properly. A complete lean loop beats an excellent pipeline with no feedback

If you want the wider context on signal engineering before going into the build, the webinar recording covers the whole picture at a higher level, and the introduction post covers why any of this became necessary.