Offsite 2026
Exercise: Things we'd want to know about a project
- Lines of Code
- Architecture Diagrams
- Test Coverage
- Big bloated files
- Linters
- Code Coverage
- How to run the project
- Does it compile?
- Tech stack
- Build pipeline structure
- Persistence structure (databases)
- External dependencies
- What AI infrastructure (skills, etc.) are in place?
Jeffrey leverages ask mode for agents to figure out what the best tools are for each of the above, picks the tools, and has the agent leverage all of them to create a report after it runs all the tools.
Main Presentation
Computational limits of LLMs
What the models cannot do on their own
No semantic awareness: LLMs have no true understanding of what a given instruction means
No completion detection: LLMs cannot determine when a task is actually done
No quality assessment: LLMs do not have intrinsic values of quality and possess no means to evaluate its own output
Because models cannot self-assess, quality must be judged from the outside.
Not all code carries the same weight
Some elements of a program are structural, others are cosmetic. Today's LLMs cannot tell them apart and every change prompted by a developer is treated with equal consideration.
We can split changes into two categories: structural and superficial.
Structural
Structural changes impact core architectural patterns, domain models, and integration interfaces.
Superficial
Superficial changes impact UI components, formatting logic, and utility functions.
Review in proportion to risk
Automated tests alone are no longer sufficient for ensuring code stability. Since AI cannot judge significance, humans apply risk-proportional review; light reading where mistakes are "cheap" in terms of time and effort to fix, in-depth scrutiny when structural changes are up for review.
The bottleneck is now going to move to validation: to keep pace, build-and-test pipelines must be reegineered for speed. 10 minute builds are no longer sufficient for evaluating the state of a program following changes. Jeffrey is suggesting a drastic decrease in build and test speed to around 2 minutes.
Quality gates must become stronger and faster. For things like PR validation build and test runs, they should be fast.
Fred Brooks gets his runback: The return of surgical teams
The notion posited by Fred Brooks of a "surgical software team" is relevant again: smaller highly-skilled teams can now match the output that demanded much larger teams.
2-3 senior engineers with AI proficiency can match or exceed a larger traditional team. They'll be able to deal with:
- API documentation research
- StackOverflow consultation
- Library usage discovery
- Boilerplate generation
Function points
Function points measure what software does. Unlike story points, which track change velocity, function points track actual system complexity. It is language independent.
Five requirements for effective AI integration
- Recognize LLM limits
- Risk-stratified review
- Optimize the pipeline
- Develop engineering skill
- Adapt the organization
Conclusion
Engineering discipline still wins
Second Presentation: "First, Do No Harm"
The illusion of "just one more commit"
Adding more and more fixes and "going fast" keeps you going around in circles. You keep making band-aids and fixes and keep the program "moving" but never stable or truly fixed.
Three things for every single commit:
- Tests pass
- Build is "green"
- Program deploys
Small changes, verified completely. This has to be our natural pattern for all projects.
The AI Kool-aid
As he reviewed AI code more and more, Jeffrey realized that the AI was making the same mistakes he himself used to make. "A string of commits, every build broken. Assertions removed to make the tests & build pass."
The difference is inspection
Jeffrey notes that when he breaks the build, he notices it immediately. "I'm sitting there. I see the error. I fix it right away."
When the AI breaks it, he had to review the code all at once and go backwards, trying to figure out where it all went wrong.
Powerful tools are still tools
AI has the same power that we do to harm code as it does to improve or add to it. The difference is intention: humans have intention when they make changes, AI are simply following instructions without context or broader intention.
Feedback systems are our guardrails
Broken builds, failing tests, and systems that won't deploy tell you immediately that something has gone wrong: you made a mistake and destabilized things.
Source control allows us to step back the moment things go wrong: revert, understand, and try again.
Before we hand work over to AI, you should ask:
- How much freedom am I giving this tool?
- How large is the change?
- How quickly will I review?
- Can I inspect each step?
We break it down, one atomic change at a time. Review the atomic change, verify its correctness, verify stability, merge. Loop.
This is slower than doing it all at once but staying on the path is where stability lies and being stable makes you faster over time.
There is "fake" speed and "real" speed - you can move code fast but it can easily jeopardize system health and stability. Your process should be fast.
Every change should move the system toward health
An unstable build is harm. A broken test is harm. Skipping quality checks is harm. A doctor inherits a patient in an unknown state, and no step they take to cure the patient should result in harm.