Skip to main content

Offsite 2026

Exercise: Things we'd want to know about a project

  • Lines of Code
  • Architecture Diagrams
  • Test Coverage
  • Big bloated files
  • Linters
  • Code Coverage
  • How to run the project
  • Does it compile?
  • Tech stack
  • Build pipeline structure
  • Persistence structure (databases)
  • External dependencies
  • What AI infrastructure (skills, etc.) are in place?

Jeffrey leverages ask mode for agents to figure out what the best tools are for each of the above, picks the tools, and has the agent leverage all of them to create a report after it runs all the tools.

Presentation

Computational limits of LLMs

What the models cannot do on their own

No semantic awareness: LLMs have no true understanding of what a given instruction means

No completion detection: LLMs cannot determine when a task is actually done

No quality assessment: LLMs do not have intrinsic values of quality and possess no means to evaluate its own output

Because models cannot self-assess, quality must be judged from the outside.

Not all code carries the same weight

Some elements of a program are structural, others are cosmetic. Today's LLMs cannot tell them apart and every change prompted by a developer is treated with equal consideration.

We can split changes into two categories: structural and superficial.

Structural

Structural changes impact core architectural patterns, domain models, and integration interfaces.

Superficial

Superficial changes impact UI components, formatting logic, and utility functions.

Review in proportion to risk

Automated tests alone are no longer sufficient for ensuring code stability. Since AI cannot judge significance, humans apply risk-proportional review; light reading where mistakes are "cheap" in terms of time and effort to fix, in-depth scrutiny when structural changes are up for review.

The bottleneck is now going to move to validation: to keep pace, build-and-test pipelines must be reegineered for speed. 10 minute builds are no longer sufficient for evaluating the state of a program following changes. Jeffrey is suggesting a drastic decrease in build and test speed to around 2 minutes.

Quality gates must become stronger and faster. For things like PR validation build and test runs, they should be fast.

Fred Brooks gets his runback: The return of surgical teams

The notion posited by Fred Brooks of a "surgical software team" is relevant again: smaller highly-skilled teams can now match the output that demanded much larger teams.

2-3 senior engineers with AI proficiency can match or exceed a larger traditional team. They'll be able to deal with:

  • API documentation research
  • StackOverflow consultation
  • Library usage discovery
  • Boilerplate generation

Function points

Function points measure what software does. Unlike story points, which track change velocity, function points track actual system complexity. It is language independent.

Five requirements for effective AI integration

  • Recognize LLM limits
  • Risk-stratified review
  • Optimize the pipeline
  • Develop engineering skill
  • Adapt the organization

Conclusion

Engineering discipline still wins

Back to top