Our first customer is us
The request is concrete: guard every QA.army pull request and every other connected project, then show the result in one place. The team should be able to start with an application URL, authenticate, connect GitHub, and turn its most important customer flow into a plain-language Test.
This closes a delivery loop: customer need → feature → Test → Run → evidence → pull-request decision. If the signup flow breaks, QA should identify the failed expectation and show the browser evidence. If the testing infrastructure cannot finish, it should say so and keep the merge blocked.
Saved regression Tests preserve expectations that must continue to hold. Tests derived from a PR diff would investigate the behavior that changed. Both need to execute against the right application revision before either can justify a merge.
One QA view across many Workspaces
The requested “one lens” is a view across authorized work, with a clear hierarchy:
A user can belong to multiple Workspaces. Each Workspace contains Projects; each Project organizes Test Groups and Tests; every execution creates a Run with its own outcome and evidence. Workspace remains the security and billing boundary.
A suite is a Test Group in this model. A unified view should let a customer find the affected repository, journey, and Run without combining permissions or hiding missing coverage. Seeing a Project does not grant access to another Workspace.
The inspiration came from the Neuronpedia Jacobian Lens demonstration, which presents model readouts alongside steering experiments. For QA.army, the useful analogy is making state inspectable and checking the effect of a change. This article uses that analogy for a product view; it does not claim that QA.army integrates Jacobian Lens or can inspect a testing model’s private reasoning.
What the dogfood actually showed
Our September 15 inspection found a working GitHub connection and deployment-triggered Test execution, alongside gaps that still prevent an “every PR is guarded” claim. These are observations from our own connected Project, not a measure of every customer’s configuration.
| Boundary | Observed state |
|---|---|
| Repository connection | The connected Project exposes the QA.army repository. Other repositories still require their own authorized connection and Project setup. |
| Saved coverage | Two Homepage Tests are selected for deployment QA. Signup, Dashboards, and Billing groups have no Tests yet. |
| Latest inspected check | Two Runs were inconclusive. A GitHub check exists, but this is not passing journey evidence. |
| Merge protection | At inspection, the default branch had no required status checks. Posting a result alone did not guard a merge. |
| PR-diff Tests | The current trigger executes selected saved Tests after a successful deployment event. Automatic diff-derived Test generation is still a delivery target. |
| GitHub conversation | Result comments are implemented. Authorized comment commands and a complete conversational setup flow remain to be delivered. |
| Signup and overview | Email and Google authentication exist. GitHub-only signup remains release work. The website field and unified view have since shipped; see the delivery update below. |
One specific defect illustrates why this matters: an inconclusive aggregate was published as a neutral GitHub check. GitHub accepts neutral conclusions for required status checks. Our correction makes every non-passing aggregate block the check while keeping the underlying Run outcome truthful. Fixing that mapping is one part of delivery; the configured guard still needs production proof.
Delivery update · September 15
The non-passing check correction is now deployed and production-verified: two inconclusive Runs produce a failed GitHub check while retaining their underlying Run outcomes. The QA overview now expands authorized Workspaces into Projects, Test Groups, Tests, and recent Run links. Signup carries an optional website into Project setup and offers the updated agent prompt.
The execution blocker is concrete. The inspected deployment target redirected from a protected Vercel preview to Vercel SSO; navigation failed before the saved steps ran. That marketing preview also cannot prove an AWS product change. The remaining work is a reachable preview of the correct product revision, configured repository coverage, enforced merge checks, and authorized GitHub comment commands. This is not an all-repositories-protected release claim.
Make the pull request the primary conversation
The target interface is the PR comment thread: explain what will be tested, link the live Runs as they start, publish the outcome, and keep the evidence close to the code review. A person or coding agent should be able to request an authorized follow-up without reconstructing context in a second chat.
Comments must operate through the same server-authorized Projects, Tests, and Runs as the product UI. Repository text is untrusted input. A comment must never grant Workspace access, expose credentials, run arbitrary code, or silently retry an action that may already have changed the target.
Authentication and repository connection are separate steps. Today’s email and Google sign-in should remain usable while GitHub onboarding is completed. The agent setup contract remains one short-lived claim link and one user code, with reusable credentials held in the native operating-system credential store.
What must be true before we call the loop closed
- The connected repository is covered. Its intended Project, deployed application URL, and enabled Test Groups are explicit.
- The exact PR revision is tested. A result from production or an unrelated marketing preview cannot prove a changed product backend.
- The guard is enforced. The required check comes from the QA.army GitHub App. Missing, failed, cancelled, or inconclusive execution cannot clear it.
- The conversation has evidence. The PR links to live Runs, then to truthful terminal outcomes and accessible recordings or screenshots.
- The customer can repeat the setup. A fresh customer or agent can connect the application, establish the authorized scope, and complete the same path.
This is why QA is important: it turns “we shipped the feature” into a claim someone can inspect. The same standard applies to QA.army itself.
Start with one critical customer journey.
Use the current authentication contract and public API. Treat unsupported PR automation as a visible setup gap, never as an assumed success.