The Loop: The Automation That Broke When the Site Changed

Welcome to The Loop. Every Wednesday we take one real, repetitive workflow that someone does by hand and unwind it into something you can run. This week it's the one nobody plans for: the automation you already built, and the Tuesday morning you lose putting it back together after somebody else's redesign.

Here's the idea worth carrying out of this one. The boring task you repeat is a specification in disguise, and the task being repeated here is the repair. Every time you go fix it, you're rewriting the same sentence: the price is the dollar figure next to the Add to Cart button. You just wrote that sentence the first time as div:nth-child(4) > span.p_2kQ, which is directions to a house rather than a description of the house. Directions expire the moment somebody renumbers the street. Descriptions don't.

So let's sit next to somebody doing the repair by hand, price what the repair really costs, and then build the version that tells you it's in trouble instead of quietly handing you blanks.

The Loop, in Full

Before you fix anything, you have to see the loop clearly. Picture Marcus, who runs operations at a three-person industrial distributor. Eleven months ago he wrote a script that opens four supplier portals every weekday at 7am and pulls the current unit price, stock status, minimum order quantity and lead time for about forty parts into one sheet, so the two people quoting customers work from real numbers instead of a PDF from March. It's the single best thing he's built. It ran without a word from him until the Tuesday one supplier shipped a new design.

Here's what happened next, and it's the fourth time in eleven months he's done it:

1. Somebody in sales asks why supplier C's prices are blank. He opens the sheet and finds they've been blank since Friday, quietly, in the middle of a column full of good numbers.
2. He opens the portal by hand. The page loads, the login works, the price is sitting right there in front of him. Nothing is down. Something moved.
3. Devtools open, right-click the price, Copy selector. Out comes #root > div.lay_3fXa > section > div:nth-child(4) > span.p_2kQ, a string that means nothing and will mean nothing different next quarter.
4. Paste it into the script, rerun for one part number. The price comes back. Stock status is now empty, because that moved too, so he goes back for it. Then minimum order quantity. Then lead time.
5. Backfill Friday, Monday and Tuesday by hand so the sheet isn't lying about last week. Commit the fix. Wait for supplier D to redesign.

Two hours, and not one minute of it needed his judgment. He already knew what the four fields were on day one, and they're the same four fields they've always been. What expired was the address, and the address is the only part he wrote down. That's the tell for an automatable loop: the thinking happened once, and everything since has been re-typing the answer in a language that keeps changing under him. It's the same reason people give up on browser work entirely instead of learning to put a browser agent on the tab work properly.

The Manual Tax

The real cost of a loop is never just the minutes on the clock. It's the minutes, plus the mistakes, plus the time the work sits waiting. What makes this loop expensive is that only the first of those three is visible, and it's the smallest one.

The minutes alone are worse than they feel. In its 2026 write-up on managing change in scraping systems, PromptCloud puts unmanaged structural change at roughly 40% to 60% of all recurring maintenance effort on production extraction pipelines. Not the anti-bot arms race, not the infrastructure. Just pages moving their furniture around. Half of the total upkeep bill for reading the web goes to re-finding things that never moved in the first place.

Then add the two costs hiding behind the minutes:

• Errors. A missing field is the friendly failure. The dangerous one is the selector that still matches something. When a badge gets inserted above the price, div:nth-child(4) now points at the shipping surcharge, which is a number, in dollars, that validates fine and is completely wrong. Nothing errors. The sheet fills in. Somebody quotes from it.
• Latency. Marcus lost four days between the redesign and the question from sales, and that gap is the whole problem. Every decision made in those four days used data that was either absent or false, and no alert existed to say so, because from the script's point of view the run finished successfully.

The automation didn't fail on the day the site changed. It failed on the day somebody noticed, which is a different day, and a much more expensive one.

So the goal isn't to write selectors that never break, because you can't and neither can anybody else. The goal is to stop paying the tax: make the build describe what it wants, try several ways of getting it, and shout the moment it can't. Done once, that turns a workflow you babysit into a workflow you own, which is the difference between a script and something you can hand a whole recurring job to.

Unwinding the Loop

Automating starts with description, not code. Describe the loop precisely enough that a machine could follow it without you in the room, and most of the work is done before a line is written. For this loop the description does something extra: writing down what each field is gives you the thing your original script never had, which is a way of recognizing the field after it moves.

Here's the resilient read captured as a spec. The interesting parts are the decisions, not the steps:

That last row is the one people leave out, and it's the one that buys you the most. Everything after this table is choosing how hands-off you want to be. If you'd rather answer questions than fill in a spec, that's exactly how BYOBot turns a task into a spec, one question at a time, and the two middle rows are where the argument is worth having.

The Build: One Loop, Three Ways to Run It

The same described loop becomes three builds depending on how much you want running on its own. Start at the top and move down only when you're ready. For this loop the first build is the one that matters most, because it produces the artifact the other two run on.

The Prompt

Save the page you care about (right-click, Save Page As, or just copy the section of source that holds your fields) and paste it into any AI chat tool with this. You're not asking it to extract anything yet. You're asking it to write the field contract: several ways to find each value, ranked by how long each one will survive, plus the rule that says whether the value is any good.

Here is the saved HTML of a page I need to read on a schedule.
[paste the page source, or the section containing the fields]

The fields I need: unit price, stock status, minimum order quantity, lead time.

For each field, give me:

1. Three ways to locate it, ranked most durable first. Prefer, in order:
   a) structured data already in the page (JSON-LD, microdata, meta tags)
   b) a stable attribute (data-testid, id, itemprop, aria-label, role + name)
   c) the words of a nearby visible label, matched by text, not by position
   d) a CSS path, only as a last resort, and label it as one

2. A validation rule: the type, the format, and a plausible range or value set.

3. The exact message to raise when nothing matches, naming the field
   and what was expected.

Then list every locator you proposed that depends on class names, element
order, or nesting depth. Those are the ones that break first, and I want
to know how much of this contract is resting on them.

Return it as a table I can save as the field contract for this page.

Any of the big chat tools will handle it, so use whichever one you already have open. The final instruction is the one that earns its keep: it tells you honestly how fragile the page is before you've built anything on top of it. A page where three of four fields can only be reached by a CSS path is a page that will break, and knowing that on day one changes what you build. Writing this down before you start is the same discipline as writing the workflow spec before you build, and it saves the same morning.

The Script

This loop has to open a page, log in, and read rendered output, so the honest build is a browser agent rather than a plain API call. The thing to get right is where the intelligence sits. Matching should be cheap, deterministic and boring on the ordinary day, with a model call reserved for the moment everything boring has failed. Put the model in the hot path and you've bought yourself slow runs, a bill per page, and a system that hands you a confident wrong answer instead of an error.

The locator ranking isn't a BYOBot invention, and it's worth stealing from the people who fought this out first. Playwright builds its whole locator API around role, label and text rather than CSS paths, for exactly this reason. Testing Library publishes a priority order that puts accessible role and label at the top and test IDs near the bottom. And before you write any locator at all, check whether the page is already handing you the answer as data: many commerce and listing pages embed Schema.org markup with price, availability and identifier as labeled fields, because search engines asked for it and the site maintains it deliberately.

// pseudo-shape of the resilient reader BYOBot generates for you
const CONTRACT = readFile('supplier-c.contract.json');   // from the prompt build

for (const sku of SKUS) {
  const page = await browser.open(urlFor(sku));
  const row = {}, matchedBy = {};

  for (const field of CONTRACT.fields) {

    // 1. TRY each strategy in order, most durable first, stop at the first hit
    for (const loc of field.locators) {              // jsonld > attr > label > css
      const value = await page.tryLocate(loc);
      if (value != null) { row[field.name] = value; matchedBy[field.name] = loc.kind; break; }
    }

    // 2. VALIDATE the value, not just its presence
    if (!field.isPlausible(row[field.name], field.lastKnown)) {
      snapshot(page, sku, field.name);               // keep the HTML you failed on
      hold(sku, `${field.name}: got ${show(row[field.name])}`);
      delete row[field.name];
    }
  }

  // 3. REFUSE to write a partial row. A blank cell is a lie.
  if (CONTRACT.required.every(f => f in row)) write(sheet, row);

  // 4. REPORT how each field was found. A demotion is a warning, not a detail.
  log(sku, matchedBy);     // 'jsonld' yesterday, 'css' today = this page moved
}

Steps two and four are where the resilience lives. Validation compares against the last known value rather than just checking the field exists, which is the only thing that catches the surcharge-instead-of-price failure. And step four turns the build into its own monitoring: you find out a page is drifting while the fallbacks are still holding, not when the last one gives out. You don't write this from a blank page. BYOBot generates the full version for your sites, including the login handling, the contract, the validators and the hold queue, and the same skeleton covers every other page you watch on a timer, which is where it starts making sense to automate the market and pricing checks you run by hand as a set.

The Schedule

Putting the script on a timer is the small part: a cron job, a GitHub Actions workflow on the free tier, or a scheduled scenario in a no-code tool like Make all do it. The part this particular loop needs is the canary. Run a cut-down version nightly whose only job is to open one known page and confirm every field in the contract still resolves by its primary locator, whether or not there's anything to collect that day. Marcus lost four days because his automation only looked at the site when it wanted something. A canary that runs on Saturday finds the Friday redesign on Saturday. Pair it with the demotion log and the monitoring is effectively free, which is the difference between an agent you check on and one that's genuinely running on its own while you do something else.

Here's how the three builds stack up, so you can pick your stopping point:

Steal This Build

Five lines, and they're the whole build. Copy them, swap the field names for your own, hand them to BYOBot or write them yourself this afternoon.

• Trigger: the scheduled run, plus a canary that checks the contract on the days there's nothing to collect.
• Locate: each field by the most durable strategy that works, structured data first and CSS path last.
• Validate: type, format, range, and distance from the last known value, not just whether something came back.
• Refuse: never write a partial row. Save the page you failed on and hold the record with the field named.
• Report: which strategy matched each field, so a demotion becomes the warning that the page is moving.

The loop you just watched is a supplier price check, but the shape is universal. Anything that reads something you don't own, whether that's a partner's dashboard, a government filing page, a job board, or an internal tool another team ships on Fridays, wants the same three moves: describe the field instead of its address, check the value is believable before you trust it, and make failure loud enough to catch on the day it happens. Learn that once and the entire category of "it worked until they changed the site" turns into a notification rather than a lost morning, which is what it looks like to automate a recurring workflow properly the first time.

Frequently Asked Questions

• Why does my automation break every time the website updates? +
  Because it was told where to look instead of what to find. A CSS path or an XPath is a set of directions through the page: third div, second span, the element with class p_2kQ. Those class names are generated by a build tool and the nesting is a side effect of how the layout was assembled, so both change whenever a front-end developer touches the page, even when nothing a visitor can see has changed. A locator built on a stable attribute, a visible label, or the structured data already embedded in the page survives that, because those describe the field rather than its address. The same reframe fixes the other classic browser-automation failure, the one where a popup you never planned for eats the run, and it's worth reading how to handle the interstitials that stop a browser agent at the same time.
• Can AI just re-find the element when the page changes? +
  It can, and that's the right fallback, but it's the wrong primary. A model call on every field of every run is slow, costs money per page, and gives you a different answer on a bad day. Worse, it almost never fails outright: asked for the price on a page where the price has moved, it'll confidently hand you the shipping surcharge. Use it the way a good build uses it, as the last resort that proposes a new locator when the durable ones all miss. Then validate that proposal, pin it into the field contract, and go back to cheap deterministic matching on the next run. Self-healing is a repair crew, not a foundation.
• Should I use the site's API instead of reading the page? +
  Yes, whenever one exists, and check harder than you think you need to before deciding it doesn't. Many pages that look like plain HTML are rendering from a JSON endpoint the browser is already calling, which you can see in the network tab in about thirty seconds. Many more carry JSON-LD or microdata in the markup for search engines, which hands you the price, the availability and the identifier as labeled fields rather than as text you have to parse. Check both before you write a single selector. An endpoint or a structured data block is a contract the site is deliberately maintaining. A CSS path is a detail it never promised you.
• How do I find out my automation broke before somebody downstream does? +
  Three habits cover almost all of it. Validate on the way out, so a value that's empty, the wrong type, or wildly outside its usual range stops the write instead of filling a cell. Run a structure check on a schedule even when there's nothing to collect, so a quiet weekend redesign gets caught on Sunday rather than on the Thursday somebody quotes from it. And log which locator strategy matched each field, because a field that used to resolve from structured data and now resolves from a CSS path is telling you the page moved, days before it breaks completely. Build those three in from the start and the maintenance stops being a surprise, which is the whole point of choosing to hand the repetitive work off rather than grinding through it.