GPT-4

## GPT-4: What the Benchmarks Reveal — and What They Obscure

### 1. The Actual Qualitative Leap

GPT-4 marks the transition from a text-only model to a natively multimodal one: it now accepts image inputs alongside text, producing text outputs. This is not a cosmetic addition. The ability to reason over visual content — charts, screenshots, medical diagrams, handwritten code — unlocks use cases that GPT-3.5 was structurally unable to address.

On standardized professional and academic benchmarks, the numbers are unambiguous: GPT-4 scores around the 90th percentile on the Uniform Bar Exam, versus roughly the 10th percentile for GPT-3.5. On the USMLE (US medical licensing exam), it clears the ~60% passing threshold required for certification. On GRE Verbal, it lands in the 99th percentile. These scores are not statistical curiosities — they signal that a language model can now operate at a formally certifiable competence level in high-stakes, regulated domains.

### 2. What the Benchmarks Don't Capture

OpenAI is explicit about the core limitation: GPT-4 remains "less capable than humans in many real-world scenarios." That phrasing deserves unpacking. Academic benchmarks measure pattern recognition over massive training corpora — they do not measure causal reasoning, deep contextual ambiguity handling, or reliability on long, multi-step tasks without human supervision.

The model also inherits the structural limitations of its predecessors: factual hallucinations, sensitivity to prompt engineering, no native persistent memory, and a training data cutoff that renders it blind to recent events. The image→text multimodality is one-directional: GPT-4 does not generate images (unlike DALL-E), which precisely delimits its operational scope.

### 3. Competitive Repositioning and Likely Losers

The announcement compresses the competitive landscape on several fronts simultaneously:

**Specialized legal and medical NLP tools**: startups like Harvey (legal) or clinical NLP solutions built on fine-tuned GPT-3.5 or open-source LLMs see their differentiation erode. If GPT-4 passes the bar exam at the 90th percentile in zero-shot, the cost of building a specialized layer on top of a generalist model drops dramatically.

**Traditional computer vision vendors**: the integration of image+text in a single model reduces the need for hybrid pipelines (OCR → NLP, or vision model → LLM). Integrators who monetized that assembly complexity are directly threatened.

**Google and the search market**: GPT-4's integration into Bing (already announced via the Microsoft partnership) positions a search engine with multimodal reasoning capability against a Google that has not yet deployed Gemini. The timing is strategically unfavorable for Mountain View.

**Anthropic and alternative LLMs**: Claude (Anthropic) and open-source models like LLaMA (Meta, released days earlier) are immediately repositioned as second-tier alternatives on formal benchmarks, even if their value propositions around safety or cost remain distinct.

### 4. What Practitioners Need to Watch Now

GPT-4 access is gated through the OpenAI API (waitlist at launch) and ChatGPT Plus ($20/month). Per-token cost is significantly higher than GPT-3.5-turbo — a critical factor for high-volume applications. Teams that have optimized prompts and architectures for GPT-3.5 will need to reassess the cost/performance ratio before any systematic migration.

The extended context window reaches 32,000 tokens (versus 4,096 for standard GPT-3.5-turbo), which fundamentally changes retrieval-augmented architectures: less need for aggressive chunking, ability to pass entire documents in context.

Finally, benchmark reproducibility remains an open question. OpenAI has not released model weights or full evaluation methodology details — a decision consistent with their closed-model pivot, but one that prevents independent verification of the announced performance figures.