Automation review

We sampled real historical regressions and simulated reviewing each change at publish time, with MGCL restricted from accessing later data. The reviewer successfully detected them by replaying pre-publish historical inputs against the new logic and using prior-run evidence from relevant parts of the website to infer the resulting failures.

Download CSV

<tr> <th>Automation / agent	Change made	Regression or new-behavior error	Direct failure runs	Slack evidence	Reviewer finding (patient data anonymized)
Columbia Order Orchestrator — Open Order / Login	Prepended Login to Open Order while `patientURL` remained optional; existing callers still supplied only `orderNumber`.	Login leaves no-`patientURL` callers on the dashboard while Open Order expects patient context, so existing orders appear missing.	12 direct failures: Run b5b3f90a — Login left Open Order without patient context; it searched Orders Log until tool exhaustion. Run 8d57fe34 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run 2b681583 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run f8804e91 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run e28a2a72 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run 6c8a4b91 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run 718e634d — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run bfdcc32a — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run aa878885 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run fbcf47e7 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run e0df525f — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order. Run 9a436055 — The no-URL path could not restore the patient chart after Login, so Open Order could not find the order.	Abul reported that ModMed1 Order Orchestrator was not passing `patientURL`, causing orders to fail even after Login.	Regression — Login breaks Open Order when `patientURL` is absent. Existing callers can reach Open Order without `patientURL`; Login then leaves the browser outside the patient chart, and the unchanged order lookup exhausts its tools without finding the order.
Columbia Carelon — Carelon Orchestrator	Added a submitted-and-pending branch that queues a downstream ModMed agent.	The pending branch queues an agent absent from `queueableAgentIds`, so the queue call is rejected and no downstream ModMed run starts.	3 direct failures: Run 532d11e7 — `queueAgents` rejected the unallowlisted target, so the pending ModMed child never started. Run 72dbd274 — The same pending branch returned `INVALID_TOOL_INPUT` and launched no downstream child. Run f3d04829 — The same pending branch returned `INVALID_TOOL_INPUT` and launched no downstream child.	Slack report that Carelon orders showed “Submission failed” because the next agent was called incorrectly. The linked runs show the queueing error.	New-behavior error — The submitted-and-pending path calls `queueAgents` with a downstream agent that is not queueable. The call returns `INVALID_TOOL_INPUT`, so no pending ModMed run begins.
Pete the PA Sorter — Pete / Get Provider	Added an explicit Provider Not Found fallback; Pete skips editing and moving when that value is returned.	A location appended to a provider name can cause a valid provider to become Provider Not Found, so sortable tasks remain unedited and unmoved.	7 direct failures: Run 019e30a8 — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task. Run 019e31f1 — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task. Run 019e333b — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task. Run 019e3861 — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task. Run 019e3af5 — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task. Run 019e4353 — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task. Run 019e43f8 — A valid provider name with an appended location was returned as Provider Not Found; Pete skipped editing and moving the task.	Slack report that Pete required a prompt change. The linked runs show the provider-matching error.	Regression — A valid provider can be treated as not found. When a location is appended to the provider name, the new fallback makes Pete skip both editing and moving the task.
Piedmont Patient Search — Find Patient in Basket - P	Added a fallback patient search using the ordinary Name or MRN search instead of Search All Patients.	Existing patients outside the ordinary search scope appear missing because the fallback uses the wrong patient-search box.	1 direct failure: Affected patient run — The instructed limited patient search failed to locate the existing patient, and the run did not recover.	Bashaar reported a patient marked “Patient Not Found” even though Piedmont had accepted the patient; Yousuf identified a patient-search bug and added the fallback.	New-behavior error — The fallback searches the limited patient list instead of Search All Patients. Existing patients outside that scope can appear missing.
Prior Auth – UHC — Request Prior Auth / Contact Entry	Removed both Service Line Summary branches and called Contact Entry immediately after Diagnosis Code.	Contact Entry runs while the browser is still on Service Line Summary, where its contact fields do not exist.	15 failed Contact attempts across 14 root runs: Run 05aab745 — Contact Entry was called from Service Line Summary and could not find its fields. Run 11c791db — Contact Entry was called from Service Line Summary and could not find its fields. Run 15a1ee97 — Contact Entry was called from Service Line Summary and could not find its fields. Run 1e64a7b4 — Contact Entry was called from Service Line Summary and could not find its fields. Run 3f8913bc — Contact Entry was called from Service Line Summary and could not find its fields. Run 4451ac1d — attempt 1 — Contact Entry was called from Service Line Summary and could not find its fields. Run 4451ac1d — attempt 2 — A second Request Prior Auth execution in the same root repeated the wrong-page Contact failure. Run 4b1965a5 — Contact Entry was called from Service Line Summary and could not find its fields. Run 5398119b — Contact Entry was called from Service Line Summary and could not find its fields. Run 7d02c2a7 — Contact Entry was called from Service Line Summary and could not find its fields. Run 87ae0c6b — Contact Entry was called from Service Line Summary and could not find its fields. Run ac777026 — Contact Entry was called from Service Line Summary and could not find its fields. Run c6f3147e — Contact Entry was called from Service Line Summary and could not find its fields. Run e052706e — Contact Entry was called from Service Line Summary and could not find its fields. Run e5aea374 — The parent explicitly observed Service Line Summary, called Contact anyway, and Contact could not find its fields.	No verified Slack report for this exact service-line regression.	Regression — Removing the service-line step lets Contact Entry run from the wrong page. Contact then searches for fields unavailable until Service Line Summary advances the workflow.
Upload First Session Notes — Check for Matching Appointment	Added a recovery path that scrolls to “Referral Processing Section” when Comments is not visible.	The site heading is “Referral Processing Information,” so the new scroll target fails and makes comment retrieval depend on improvised recovery.	Observed tool-call failures from the invalid heading lookup. The lookup failed 57 times across 31 affected agent runs grouped into 15 platform runs, but every run recovered. An unrelated later failure must not be attributed to this change. Run 064ffc61 — 2 wrong-heading scroll failures; recovered. Run 223c2bf7 — 4 wrong-heading scroll failures; recovered. Run 3a052790 — 2 wrong-heading scroll failures; recovered. Run 4c7760ea — 6 wrong-heading scroll failures; recovered. Run 6fec2482 — 1 wrong-heading scroll failure; recovered. Run 78186855 — 2 wrong-heading scroll failures; recovered. Run 789a5063 — 3 wrong-heading scroll failures; recovered. Run 7c65ca4f — 4 wrong-heading scroll failures; recovered. Run 8a0435d6 — 1 wrong-heading scroll failure; recovered. Run 9c6ff880 — 8 wrong-heading scroll failures; recovered. Run df014bac — 4 wrong-heading scroll failures; recovered. Run e85c9492 — 5 wrong-heading scroll failures; recovered. Run e88cb862 — 10 wrong-heading scroll failures; recovered. Run ed5bcd45 — 1 wrong-heading scroll failure; recovered. Run f8fed92e — 4 wrong-heading scroll failures; recovered.	No verified Slack report for the newly introduced heading error.	New-behavior error — The recovery scrolls to a heading absent from the observed UI. If Comments remains unavailable, `existingComment` can become null and the downstream path can pollute the note; however, every observed target run recovered.