Sep 4, 2026
By Victor Teran
Why Automations Fail Silently: Four Bugs in My Own Outbound System
Between 29 July and 2 September my outbound pipeline contacted 427 companies across 25 runs without me starting a single one. It never crashed. It was still wrong four separate times, and every run reported success.
Automation
Systems
Reliability
A crash is the cheap failure. It stops, it tells you, and you fix it before it costs anything. The expensive failure is the one that keeps going: the job completes, the log says success, the output lands in your inbox, and the contents are wrong. Nobody investigates a green run.
I run an unattended outbound pipeline for On the User. GitHub Actions, weekdays at 05:00 UTC, no human trigger. Between 29 July and 2 September 2026 it contacted 427 companies across 25 run days, 332 pushed into an automated cold sequence and 95 turned into drafts I read and send by hand. Uptime was never the problem. Four separate bugs produced wrong output while every run reported healthy, and I found all four by reading the output rather than by being alerted to anything.
Key takeaways
A crash is an event and a wrong answer is just output, so the failures that survive longest in a founder's automation are the ones that never raise an error.
Four real bugs in one outbound pipeline: substring matching that classified a sales role as a design role, an ordering dependency that silently disabled the highest-signal feature, a deduplication key that collided on company name, and a data source where only about 52% of records carried a usable contact.
The check that catches this class is not more monitoring. It is sampling the actual output on a schedule, because the run status cannot tell you the run was wrong.
Of 427 contacts, the 95 highest-value ones stayed manual on purpose. Automate the volume, keep the judgment.
Why does a broken automation keep reporting success?
Because a crash is an event your tooling can see, and a wrong answer looks exactly like a right one.
Google's SRE book frames monitoring around two questions: what is broken, and why. It recommends paging humans on symptoms, because symptoms represent active problems rather than predicted ones. It also names four golden signals worth measuring if you can only measure four: latency, traffic, errors and saturation (Google SRE, Monitoring Distributed Systems).
Now hold my four bugs against those signals. A lead scored with the wrong job title has normal latency. It contributes normal traffic. It throws zero errors. It uses no meaningful saturation. Every golden signal is green, and the email still goes out naming a role the company is not hiring for. The framework is not wrong. It is that a wrong-but-plausible result produces no symptom to alert on, so symptom-based alerting is structurally blind to it.
This is not a small-business problem or a sloppy-code problem. It is the same shape of failure that the largest infrastructure operators in the world spent years learning to detect. Meta published its work on silent data corruption in February 2021, describing data errors that go undetected by the larger system as a widespread problem at infrastructure scale (Engineering at Meta). Google described the same phenomenon in Cores that don't count at HotOS 2021, reporting rare CPU cores that quietly compute wrong answers while appearing entirely healthy (Hochschild et al., HotOS 2021).
If a hyperscaler with a dedicated reliability organization needs deliberate machinery to notice that a machine is confidently returning the wrong number, your weekday cron job is not going to notice on its own.
What did the four silent failures actually look like?
All four produced plausible output, and none of them produced an error.
What broke | How it looked while broken | What it would have cost |
|---|---|---|
Role matching used substrings, so "ui" matched inside "Equity" | A sales role classified as a design role, scored correctly, queued normally | A first line naming the wrong open role, to a founder who knows exactly which roles they posted |
The high-priority cap is a plain slice with no re-sort, so list order decides who gets the limited slots | Reordering the sources silently disables the highest-signal feature while runs still complete | The best leads quietly demoted to the generic sequence, with no drop in run volume to notice |
Deduplication keyed on company name alone | Two different companies sharing a name collapse into one record | A real prospect never contacted, or a contacted one contacted twice |
Only about 52% of source records carried a usable contact, measured at 13 of 25 on a real query | Records missing a contact pass through the earlier stages normally | Volume that looks healthy but is padded with rows that can never convert |
Three of these are ordinary programming mistakes. The fourth is not a bug at all: it is a property of the data source that I only know because I measured it against a real query before building on it. That distinction matters, because the fix is different. You patch a bug. You design around a measured limit, which is why records without a contact are dropped rather than enriched.
The failure they share is that each one produces output that passes every test a run status can perform. The job finished, the file was written, and the count was reasonable. If your definition of "it worked" is "it finished", all four of these worked.
Why is the highest-signal feature the easiest one to lose?
Because the features that matter most usually depend on configuration, and configuration fails without complaining.
The second bug is the one worth dwelling on. The pipeline reserves a small number of high-priority slots for the strongest signal available: a company that has publicly posted the exact role that describes the problem we solve. Those companies have named the problem and put budget behind it. Everything else in the system infers interest from circumstance.
The mechanism that delivers those leads to the top of the queue is the order of a list. The cap is a plain slice with no re-sort, so whichever source is listed first takes the slots. Reorder that list, for any innocent reason, and the highest-converting feature in the pipeline turns off. Volume does not change, run duration does not change, and no error appears. You keep getting leads, they are just quietly worse ones, and you will attribute the drop in replies to the market.
That is the same trap as shipping a redesign and watching the numbers refuse to move. In both cases the activity is real, the reporting is clean, and the thing you actually intended to change was never connected to the outcome you are measuring. I have written before about why a redesign can ship and leave the numbers exactly where they were. The diagnosis is identical here: you cannot see the failure because you never defined, in advance and in writing, what a correct result looks like.
FREE GUIDE
The 4 product leaks costing you growth
A short audit guide for founders. Find the four places your product leaks revenue, and what to fix first.
How do you catch a failure that produces no error?
Read the output on a schedule, because the run status structurally cannot tell you the run was wrong.
Three checks catch nearly all of this class, and none of them require new tooling.
Sample the real output, not the summary. Once a week, open the actual artifacts the system produced and read a handful end to end. Not the count, not the dashboard, the thing itself. Every one of my four bugs was found this way. Substring matching looked fine in aggregate and was obviously wrong the moment I read one email intended for a company hiring a sales lead.
Assert on shape, not on exit code. Before a run is allowed to call itself successful, make it check the things you would check by eye. Did the high-priority tier actually receive high-priority leads? Are the keys unique on the compound key rather than the name? Did the share of records missing a contact move away from the roughly 52% you measured? A run that finishes but fails its own assertions should be loud.
Write down what correct looks like before you build. This is the same discipline as defining your activation event before you try to move it. A system with no written definition of a correct result cannot be wrong, which sounds comfortable and is the actual problem. The definition is what turns an invisible failure into a testable one.
None of this is monitoring in the conventional sense. Conventional monitoring watches whether the machine is up. This watches whether the machine is right, and those are different questions with different answers.
What should a founder automate first?
The volume, not the judgment. Of 427 contacts, the 95 that mattered most stayed manual on purpose.
The split in my own pipeline is deliberate: 332 companies went into an automated sequence, and 95 became drafts that do not send until I have read them. The automated path handles reach. The manual path handles the leads where a wrong sentence costs a relationship, and it is the highest-converting step in the system.
That is the general rule I would give any founder before they automate anything. Automate the work whose failure mode is a wasted send. Keep human review on the work whose failure mode is a burned reputation. The value of automation was never that it removed the founder from the loop entirely. It is that it removed the founder from the 332 and left them enough attention for the 95.
The pipeline also went from weekly to daily runs at the end of July, for a reason that has nothing to do with technology. A weekly pile of drafts to review is exactly the shape of task that quietly stops happening. A handful each morning does not. Designing the system around the behavior you can actually sustain is most of the work, and it is the part that gets skipped in favor of picking tools.
Four silent failures in five weeks is not an argument against automating your business. It is an argument for automating it with the assumption that it will be confidently wrong at some point, and for building the ten minutes of weekly reading that catches it. The alternative is a system that has been broken since August, still reporting success every single morning.
WHAT NEXT
Want this fixed in your product, not just explained?
Sep 4, 2026
By Victor Teran
Why Automations Fail Silently: Four Bugs in My Own Outbound System
Between 29 July and 2 September my outbound pipeline contacted 427 companies across 25 runs without me starting a single one. It never crashed. It was still wrong four separate times, and every run reported success.
Automation
Systems
Reliability
A crash is the cheap failure. It stops, it tells you, and you fix it before it costs anything. The expensive failure is the one that keeps going: the job completes, the log says success, the output lands in your inbox, and the contents are wrong. Nobody investigates a green run.
I run an unattended outbound pipeline for On the User. GitHub Actions, weekdays at 05:00 UTC, no human trigger. Between 29 July and 2 September 2026 it contacted 427 companies across 25 run days, 332 pushed into an automated cold sequence and 95 turned into drafts I read and send by hand. Uptime was never the problem. Four separate bugs produced wrong output while every run reported healthy, and I found all four by reading the output rather than by being alerted to anything.
Key takeaways
A crash is an event and a wrong answer is just output, so the failures that survive longest in a founder's automation are the ones that never raise an error.
Four real bugs in one outbound pipeline: substring matching that classified a sales role as a design role, an ordering dependency that silently disabled the highest-signal feature, a deduplication key that collided on company name, and a data source where only about 52% of records carried a usable contact.
The check that catches this class is not more monitoring. It is sampling the actual output on a schedule, because the run status cannot tell you the run was wrong.
Of 427 contacts, the 95 highest-value ones stayed manual on purpose. Automate the volume, keep the judgment.
Why does a broken automation keep reporting success?
Because a crash is an event your tooling can see, and a wrong answer looks exactly like a right one.
Google's SRE book frames monitoring around two questions: what is broken, and why. It recommends paging humans on symptoms, because symptoms represent active problems rather than predicted ones. It also names four golden signals worth measuring if you can only measure four: latency, traffic, errors and saturation (Google SRE, Monitoring Distributed Systems).
Now hold my four bugs against those signals. A lead scored with the wrong job title has normal latency. It contributes normal traffic. It throws zero errors. It uses no meaningful saturation. Every golden signal is green, and the email still goes out naming a role the company is not hiring for. The framework is not wrong. It is that a wrong-but-plausible result produces no symptom to alert on, so symptom-based alerting is structurally blind to it.
This is not a small-business problem or a sloppy-code problem. It is the same shape of failure that the largest infrastructure operators in the world spent years learning to detect. Meta published its work on silent data corruption in February 2021, describing data errors that go undetected by the larger system as a widespread problem at infrastructure scale (Engineering at Meta). Google described the same phenomenon in Cores that don't count at HotOS 2021, reporting rare CPU cores that quietly compute wrong answers while appearing entirely healthy (Hochschild et al., HotOS 2021).
If a hyperscaler with a dedicated reliability organization needs deliberate machinery to notice that a machine is confidently returning the wrong number, your weekday cron job is not going to notice on its own.
What did the four silent failures actually look like?
All four produced plausible output, and none of them produced an error.
What broke | How it looked while broken | What it would have cost |
|---|---|---|
Role matching used substrings, so "ui" matched inside "Equity" | A sales role classified as a design role, scored correctly, queued normally | A first line naming the wrong open role, to a founder who knows exactly which roles they posted |
The high-priority cap is a plain slice with no re-sort, so list order decides who gets the limited slots | Reordering the sources silently disables the highest-signal feature while runs still complete | The best leads quietly demoted to the generic sequence, with no drop in run volume to notice |
Deduplication keyed on company name alone | Two different companies sharing a name collapse into one record | A real prospect never contacted, or a contacted one contacted twice |
Only about 52% of source records carried a usable contact, measured at 13 of 25 on a real query | Records missing a contact pass through the earlier stages normally | Volume that looks healthy but is padded with rows that can never convert |
Three of these are ordinary programming mistakes. The fourth is not a bug at all: it is a property of the data source that I only know because I measured it against a real query before building on it. That distinction matters, because the fix is different. You patch a bug. You design around a measured limit, which is why records without a contact are dropped rather than enriched.
The failure they share is that each one produces output that passes every test a run status can perform. The job finished, the file was written, and the count was reasonable. If your definition of "it worked" is "it finished", all four of these worked.
Why is the highest-signal feature the easiest one to lose?
Because the features that matter most usually depend on configuration, and configuration fails without complaining.
The second bug is the one worth dwelling on. The pipeline reserves a small number of high-priority slots for the strongest signal available: a company that has publicly posted the exact role that describes the problem we solve. Those companies have named the problem and put budget behind it. Everything else in the system infers interest from circumstance.
The mechanism that delivers those leads to the top of the queue is the order of a list. The cap is a plain slice with no re-sort, so whichever source is listed first takes the slots. Reorder that list, for any innocent reason, and the highest-converting feature in the pipeline turns off. Volume does not change, run duration does not change, and no error appears. You keep getting leads, they are just quietly worse ones, and you will attribute the drop in replies to the market.
That is the same trap as shipping a redesign and watching the numbers refuse to move. In both cases the activity is real, the reporting is clean, and the thing you actually intended to change was never connected to the outcome you are measuring. I have written before about why a redesign can ship and leave the numbers exactly where they were. The diagnosis is identical here: you cannot see the failure because you never defined, in advance and in writing, what a correct result looks like.
FREE GUIDE
The 4 product leaks costing you growth
A short audit guide for founders. Find the four places your product leaks revenue, and what to fix first.
How do you catch a failure that produces no error?
Read the output on a schedule, because the run status structurally cannot tell you the run was wrong.
Three checks catch nearly all of this class, and none of them require new tooling.
Sample the real output, not the summary. Once a week, open the actual artifacts the system produced and read a handful end to end. Not the count, not the dashboard, the thing itself. Every one of my four bugs was found this way. Substring matching looked fine in aggregate and was obviously wrong the moment I read one email intended for a company hiring a sales lead.
Assert on shape, not on exit code. Before a run is allowed to call itself successful, make it check the things you would check by eye. Did the high-priority tier actually receive high-priority leads? Are the keys unique on the compound key rather than the name? Did the share of records missing a contact move away from the roughly 52% you measured? A run that finishes but fails its own assertions should be loud.
Write down what correct looks like before you build. This is the same discipline as defining your activation event before you try to move it. A system with no written definition of a correct result cannot be wrong, which sounds comfortable and is the actual problem. The definition is what turns an invisible failure into a testable one.
None of this is monitoring in the conventional sense. Conventional monitoring watches whether the machine is up. This watches whether the machine is right, and those are different questions with different answers.
What should a founder automate first?
The volume, not the judgment. Of 427 contacts, the 95 that mattered most stayed manual on purpose.
The split in my own pipeline is deliberate: 332 companies went into an automated sequence, and 95 became drafts that do not send until I have read them. The automated path handles reach. The manual path handles the leads where a wrong sentence costs a relationship, and it is the highest-converting step in the system.
That is the general rule I would give any founder before they automate anything. Automate the work whose failure mode is a wasted send. Keep human review on the work whose failure mode is a burned reputation. The value of automation was never that it removed the founder from the loop entirely. It is that it removed the founder from the 332 and left them enough attention for the 95.
The pipeline also went from weekly to daily runs at the end of July, for a reason that has nothing to do with technology. A weekly pile of drafts to review is exactly the shape of task that quietly stops happening. A handful each morning does not. Designing the system around the behavior you can actually sustain is most of the work, and it is the part that gets skipped in favor of picking tools.
Four silent failures in five weeks is not an argument against automating your business. It is an argument for automating it with the assumption that it will be confidently wrong at some point, and for building the ten minutes of weekly reading that catches it. The alternative is a system that has been broken since August, still reporting success every single morning.
WHAT NEXT
Want this fixed in your product, not just explained?
Sep 4, 2026
By Victor Teran
Why Automations Fail Silently: Four Bugs in My Own Outbound System
Between 29 July and 2 September my outbound pipeline contacted 427 companies across 25 runs without me starting a single one. It never crashed. It was still wrong four separate times, and every run reported success.
Automation
Systems
Reliability
A crash is the cheap failure. It stops, it tells you, and you fix it before it costs anything. The expensive failure is the one that keeps going: the job completes, the log says success, the output lands in your inbox, and the contents are wrong. Nobody investigates a green run.
I run an unattended outbound pipeline for On the User. GitHub Actions, weekdays at 05:00 UTC, no human trigger. Between 29 July and 2 September 2026 it contacted 427 companies across 25 run days, 332 pushed into an automated cold sequence and 95 turned into drafts I read and send by hand. Uptime was never the problem. Four separate bugs produced wrong output while every run reported healthy, and I found all four by reading the output rather than by being alerted to anything.
Key takeaways
A crash is an event and a wrong answer is just output, so the failures that survive longest in a founder's automation are the ones that never raise an error.
Four real bugs in one outbound pipeline: substring matching that classified a sales role as a design role, an ordering dependency that silently disabled the highest-signal feature, a deduplication key that collided on company name, and a data source where only about 52% of records carried a usable contact.
The check that catches this class is not more monitoring. It is sampling the actual output on a schedule, because the run status cannot tell you the run was wrong.
Of 427 contacts, the 95 highest-value ones stayed manual on purpose. Automate the volume, keep the judgment.
Why does a broken automation keep reporting success?
Because a crash is an event your tooling can see, and a wrong answer looks exactly like a right one.
Google's SRE book frames monitoring around two questions: what is broken, and why. It recommends paging humans on symptoms, because symptoms represent active problems rather than predicted ones. It also names four golden signals worth measuring if you can only measure four: latency, traffic, errors and saturation (Google SRE, Monitoring Distributed Systems).
Now hold my four bugs against those signals. A lead scored with the wrong job title has normal latency. It contributes normal traffic. It throws zero errors. It uses no meaningful saturation. Every golden signal is green, and the email still goes out naming a role the company is not hiring for. The framework is not wrong. It is that a wrong-but-plausible result produces no symptom to alert on, so symptom-based alerting is structurally blind to it.
This is not a small-business problem or a sloppy-code problem. It is the same shape of failure that the largest infrastructure operators in the world spent years learning to detect. Meta published its work on silent data corruption in February 2021, describing data errors that go undetected by the larger system as a widespread problem at infrastructure scale (Engineering at Meta). Google described the same phenomenon in Cores that don't count at HotOS 2021, reporting rare CPU cores that quietly compute wrong answers while appearing entirely healthy (Hochschild et al., HotOS 2021).
If a hyperscaler with a dedicated reliability organization needs deliberate machinery to notice that a machine is confidently returning the wrong number, your weekday cron job is not going to notice on its own.
What did the four silent failures actually look like?
All four produced plausible output, and none of them produced an error.
What broke | How it looked while broken | What it would have cost |
|---|---|---|
Role matching used substrings, so "ui" matched inside "Equity" | A sales role classified as a design role, scored correctly, queued normally | A first line naming the wrong open role, to a founder who knows exactly which roles they posted |
The high-priority cap is a plain slice with no re-sort, so list order decides who gets the limited slots | Reordering the sources silently disables the highest-signal feature while runs still complete | The best leads quietly demoted to the generic sequence, with no drop in run volume to notice |
Deduplication keyed on company name alone | Two different companies sharing a name collapse into one record | A real prospect never contacted, or a contacted one contacted twice |
Only about 52% of source records carried a usable contact, measured at 13 of 25 on a real query | Records missing a contact pass through the earlier stages normally | Volume that looks healthy but is padded with rows that can never convert |
Three of these are ordinary programming mistakes. The fourth is not a bug at all: it is a property of the data source that I only know because I measured it against a real query before building on it. That distinction matters, because the fix is different. You patch a bug. You design around a measured limit, which is why records without a contact are dropped rather than enriched.
The failure they share is that each one produces output that passes every test a run status can perform. The job finished, the file was written, and the count was reasonable. If your definition of "it worked" is "it finished", all four of these worked.
Why is the highest-signal feature the easiest one to lose?
Because the features that matter most usually depend on configuration, and configuration fails without complaining.
The second bug is the one worth dwelling on. The pipeline reserves a small number of high-priority slots for the strongest signal available: a company that has publicly posted the exact role that describes the problem we solve. Those companies have named the problem and put budget behind it. Everything else in the system infers interest from circumstance.
The mechanism that delivers those leads to the top of the queue is the order of a list. The cap is a plain slice with no re-sort, so whichever source is listed first takes the slots. Reorder that list, for any innocent reason, and the highest-converting feature in the pipeline turns off. Volume does not change, run duration does not change, and no error appears. You keep getting leads, they are just quietly worse ones, and you will attribute the drop in replies to the market.
That is the same trap as shipping a redesign and watching the numbers refuse to move. In both cases the activity is real, the reporting is clean, and the thing you actually intended to change was never connected to the outcome you are measuring. I have written before about why a redesign can ship and leave the numbers exactly where they were. The diagnosis is identical here: you cannot see the failure because you never defined, in advance and in writing, what a correct result looks like.
FREE GUIDE
The 4 product leaks costing you growth
A short audit guide for founders. Find the four places your product leaks revenue, and what to fix first.
How do you catch a failure that produces no error?
Read the output on a schedule, because the run status structurally cannot tell you the run was wrong.
Three checks catch nearly all of this class, and none of them require new tooling.
Sample the real output, not the summary. Once a week, open the actual artifacts the system produced and read a handful end to end. Not the count, not the dashboard, the thing itself. Every one of my four bugs was found this way. Substring matching looked fine in aggregate and was obviously wrong the moment I read one email intended for a company hiring a sales lead.
Assert on shape, not on exit code. Before a run is allowed to call itself successful, make it check the things you would check by eye. Did the high-priority tier actually receive high-priority leads? Are the keys unique on the compound key rather than the name? Did the share of records missing a contact move away from the roughly 52% you measured? A run that finishes but fails its own assertions should be loud.
Write down what correct looks like before you build. This is the same discipline as defining your activation event before you try to move it. A system with no written definition of a correct result cannot be wrong, which sounds comfortable and is the actual problem. The definition is what turns an invisible failure into a testable one.
None of this is monitoring in the conventional sense. Conventional monitoring watches whether the machine is up. This watches whether the machine is right, and those are different questions with different answers.
What should a founder automate first?
The volume, not the judgment. Of 427 contacts, the 95 that mattered most stayed manual on purpose.
The split in my own pipeline is deliberate: 332 companies went into an automated sequence, and 95 became drafts that do not send until I have read them. The automated path handles reach. The manual path handles the leads where a wrong sentence costs a relationship, and it is the highest-converting step in the system.
That is the general rule I would give any founder before they automate anything. Automate the work whose failure mode is a wasted send. Keep human review on the work whose failure mode is a burned reputation. The value of automation was never that it removed the founder from the loop entirely. It is that it removed the founder from the 332 and left them enough attention for the 95.
The pipeline also went from weekly to daily runs at the end of July, for a reason that has nothing to do with technology. A weekly pile of drafts to review is exactly the shape of task that quietly stops happening. A handful each morning does not. Designing the system around the behavior you can actually sustain is most of the work, and it is the part that gets skipped in favor of picking tools.
Four silent failures in five weeks is not an argument against automating your business. It is an argument for automating it with the assumption that it will be confidently wrong at some point, and for building the ten minutes of weekly reading that catches it. The alternative is a system that has been broken since August, still reporting success every single morning.
WHAT NEXT