Search

Market Scan Model Benchmark - Human Scorecard

Generated: 2026-08-08T22:35:20.254Z

Run directory: /Users/antongladkov/.codex/worktrees/SLSBMB-Sender/market-scan-model-benchmark/var/market-scan-model-benchmark/20260808T201848Z-market-scan-model-benchmark

## Scope

This is an offline sidecar benchmark for model interchangeability on Market Scan writer work. It used production PostgreSQL client/project/input rows exported read-only, plus the current writer-visible Market Scan doctrine from this repo. It did not mutate production, queues, providers, or customer-facing state. It also did not run the production Kimi judge or paid/provider…

## Bottom Line

- Keep Qwen3.8 max as the production-safe baseline until a separate code change implements stage-specific model routing. The current production path is intentionally fail-closed there.
- Use GPT 5.6 Sol Max for research expansion and maximum sourcing-map breadth. It found the widest segment/arms universes on every client, but it is slower and can…

## Recommended Model Routing

| Market Scan phase | Primary | Fallback | Why |
|---|---|---|---|
| Research map / adjacent discovery | GPT 5.6 Sol Max | GPT 5.5 xhigh | Sol produced the broadest segment maps on all three clients; GPT 5.5 is the faster, more controlled fallback. |
| Sourcing arms before…

## Aggregate Matrix

| Model | Completed | Avg min | Segments | Arms | Packs | Touches | Arms / segment | Note |
|---|---:|---:|---:|---:|---:|---:|---:|---|
| Qwen3.8 max | 3/3 | 41.1 | 50 | 575 | 50 | 250 | 11.5 | Production-safe baseline; conservative, doctrine…

## Cell Matrix

| Client | Model | Status | Min | Segments | Arms | Packs | Touches | Self-fit research/sourcing/copy |
|---|---|---:|---:|---:|---:|---:|---:|---|
| connectro | Qwen3.8 max | completed | 42.1 | 22 | 175 | 22 | 110 | good/acceptable/good |
| connectro |…

## Per-Client Reading Notes

### masha
- Qwen3.8 max: 11 seg, 82 arms, 11 packs, 38.7 min; subjects: Distribution after the model works / EMEA supply, partnerships
- GPT 5.5 xhigh: 9 seg, 54 arms, 9 packs, 15.2 min; subjects: When AI starts needing a commercial spine / EMEA partnerships that carry revenue
- GPT 5.6 Sol Max: 16 seg, 757…

### connectro
- Qwen3.8 max: 22 seg, 175 arms, 22 packs, 42.1 min; subjects: The license is approved. The bank still says no / Where the subscription money lands
- GPT 5.5 xhigh: 27 seg, 463 arms, 27 packs, 16.7 min; subjects: Named account rails for licensed gaming volume / A named account route for adult dating revenue
- GPT 5.6…

### revopush
- Qwen3.8 max: 17 seg, 318 arms, 17 packs, 42.5 min; subjects: The fix is live, but not on their phones / Hotfixes for markets that never close
- GPT 5.5 xhigh: 12 seg, 144 arms, 12 packs, 17.4 min; subjects: Smaller OTA patches for finance apps / OTA patches for crypto app hotfixes
- GPT 5.6 Sol Max…

## Model Notes

### GPT 5.6 Sol Max
Best pure quality/breadth candidate for research and sourcing-map generation. It produced 73 total segments and 4,407 arms across the three clients, far above the rest. That is useful for discovery and adjacent-market capture, but it should feed a pruning/measurement stage rather than go straight to purchase or final…

### Qwen3.8 max
Best current production baseline because it is already the intended runtime lane and its outputs are conservative and doctrine-shaped. It was not the widest or fastest in this sidecar run, but it is the safest comparison anchor.

### Kimi k3 max
Strong second writer and style challenger. Good breadth on Connectro and Revopush and good copy self-audits. Operational caveat: one first-attempt quota/auth failure on Revopush before successful retry, so capacity preflight matters.

### GLM5.2 max via Qwen CLI
Completed all three validated outputs after retry, but should stay a tertiary fallback / compact independent check. It was narrower, route attestation was less clean, and Connectro first hit Qwen API data inspection before the successful retry.

## Operational Chronicle

- Full matrix was launched as a local sidecar with max parallelism; production stayed untouched.
- Primary full matrix initially completed 13/15.
- Revopush/Kimi first attempt failed on a Kimi profile usage/auth quota before final files; single-cell retry completed: 11 segments, 275 arms, 11 packs, 55 touches.
- Connectro/GLM first attempt failed with Qwen API 400 data…

## What To Open

- Human index: index.html
- Markdown index: index.md
- Machine roll-up: report-data.json
- Scorecard data: model-scorecard.json
- Full outputs: runs/<client>/<model>/market_scan.md and market_scan.json
- Raw transcripts: runs/<client>/<model>/raw-transcript.log

## Caveats

This benchmark answers writer interchangeability. It does not prove production routing, paid/provider sourcing totals, lead purchase quality, deliverability, or Kimi judge acceptance. A production split-model Market Scan would need a small governed implementation: preserve Qwen as baseline, add explicit stage routing, keep deterministic validation to JSON/transport/math only, then compare with Kimi judge and a bounded…

GPT 5.5 xhigh находит 26 гипотез там где GPT 5.6 Sol Max находит 28, что говорит о том что GPT 5.5 xhigh невероятная модель всё еще на сегодняшний день, однако SOL MAX достаточно заебистый чтобы найти ~2300 связок названий компаний и должностей там где GPT удовлетворяется на ~450.

А маркет скан нет смысла делать снова и снова - ты его делаешь раз в жизни чтобы запланировать работу на весь следующий год.

То бишь GPT 5.5 Xhigh дает тебе по тем же гипотезам Total Addressible Market capacity ~ в 4-5раз ниже чем GPT 5.6 Sol Max.

И все это стоит просто лишних 20-30 минут заебистого рассуждения модели.

Вы не в моей нише, поэтому вы не понимаете насколько это важно, но это блядь важно просто шо пиздец)

И другие цифры там есть веселые которые можно трактовать занятным образом, оставляю вас гадать на них. Когда я релизну свой продукт я думаю все в итоге объяснится его же…