Llama is a family of open-weight large language models developed by Meta AI, designed for research and commercial use. For example, Llama 3 can be run locally or fine-tuned on custom datasets using libraries like Hugging Face Transformers.
ARIADNE is a training-free framework for dynamic adapter selection at inference time. It represents each adapter through centroids computed from embeddings of its training set. Tested on Llama 3.2 1B across 23 NLP tasks, it recovers 97.44% of upper-bound performance and achieves 89.7% average selection accuracy on 44 tasks.
ProfiLLM is an agentic LLM pipeline deployed at DiDi to extract semantic user profiles from massive behavioral logs. The system uses 27 analytical tools to mine platform-scale data and generates utility-aligned profiles, achieving +6.14% AUC improvement and +0.47% GMV gain in A/B testing.
llama.cpp merges PR #23976 adding model management via API. On-demand downloading, loading, and unloading from directory. UI coming soon. Full lifecycle deployment and management through API alone.
llama.cpp optimizes GPU memory management. Key parameters: --no-mmproj-offload frees 1GB for vision models, --cache-type-k/v reduces KV cache by 50-75%, --spec-draft-n-max=2 optimizes speculative decoding. Flash attention enabled by default. Tested on Qwen 3.6-27B with 150k context on RTX 3090.
Local models shifted from marginal tools to viable solutions in one year. Gemma, Qwen, GLM, Kimi now replace some API calls for coding, private documents, and local workflows, though gaps remain on complex tasks requiring planning and error correction.
Analysis of low narrative diversity in LLM-generated stories. The author examines why models produce repetitive tales with similar characters and structures despite varied prompts.
GPT-NL is a sovereign language model trained for Dutch, developed in the Netherlands. The project aims to reduce dependence on American models and preserve linguistic technological independence.
Glint Research introduces Glimmer 1, a foundational 10k parameter language model trained on 500K tokens of FineWeb-Edu. Standard Llama architecture with 16 hidden dims, 2 layers, 4 attention heads, 512 token context window. Benchmarks: arc_easy 25.46%, wikitext-2 byte perplexity 14.73.
Article arguing for open-weight models against frontier labs. Criticizes power concentration among few companies and advocates for accessibility and transparency of AI model weights.
N2 Pro (rebranded as Rio-3.5) shows strong performance on coding benchmarks on 128GB macOS. User reports 100% consistency without hallucinations on private llama.cpp tests, outperforming previously tested models except GPT-5.x.
AeroLLM, open-source app optimized for Apple Silicon, runs local LLMs, TTS, and STT through a GUI. Uses MLX backend for native inference, downloads models from Hugging Face with RAM-based recommendations, exposes optional API endpoint. v0.1.0 released.
Mechanistic interpretability audit of LLaMA 3.1-8B-Instruct on 54 moral prompts using Transluce platform. Reveals Situational Anchor Effect: domain-specific representations dominate activation rankings regardless of ethical content. Ethics capacity remains constant but salience is highly sensitive to prompt's interpretive frame. Identifies candidate ethics neuron (L16/N3837) stable across temperatures.
Nex2 mini Phase Twin: 30B model optimized for 16GB VRAM. Designed for Intel A770 cards, runs on single GPU and scales with two. Achieves 89 tok/s on A770 16GB. Auto-calibrates to hardware.
Pull request #24080 on llama.cpp adds UI/SVG block rendering. Video demonstration shows SVG rendering capabilities integrated into the project.
llada.cpp is the first NPU-aware inference framework accelerating diffusion LLMs on mobile devices. Three techniques optimize execution: Multi-Block Speculative Decoding, Dual-Path Progressive Revision, and Swap-Optimized Memory Runtime. LLaDA-8B achieves 17x-42x latency reduction vs CPU baseline.
Comparative study of pruning vs. training from scratch on Llama-3.1-8B (ratios 0.5–0.8, 6 methods). Pruning outperforms random initialization with equal token budget, but advantage narrows with more tokens. Fine-grained pruning retains benefit even with unlimited budget; coarse structured pruning can be matched by training from scratch.
EAGLE support has been merged into llama.cpp. EAGLE is an inference acceleration technique for language models that reduces latency by predicting multiple tokens in parallel.
The cloud-based LLM gold rush is ending. Inference costs are falling, competition intensifies, and margins compress. Providers must innovate beyond API access to survive.
Heretic announces a decentralized backup system for uncensored local models. Models compressed to 9 KB enable phone storage. The project builds takedown-resilient infrastructure with official website and redundant documentation.
PR #24260 merged into llama.cpp adds Cohere-MoE architecture support. Users testing North Mini Code models, which are ~3GB smaller in Q8 quantization compared to Qwen 3.6 27B for local inference on homelabs.
A frontier cloud model was globally suspended within 48 hours after a U.S. Commerce Department order restricted access to foreign nationals. The incident highlights cloud dependency risks: a local 70B model remains operational without interruption, slower but autonomous.
llama-launcher v1.3 adds Bayesian optimization via Optuna to automatically tune llama-server parameters. The tool reports up to 15% speed improvement on Gemma 12B MTP with no manual intervention.
Discussion on decentralized distribution of open-source LLM models. Author proposes prioritizing sharing of unquantized base models (fp16/bf16) over derived variants, arguing base models are essential primary data to preserve against growing restrictions from closed model providers.
PWA support merged into llama.cpp (PR #23871). The llama-server web UI can now install as a native app on desktop/home screen, with standalone window mode and proper icons for faster reopening and better caching.
Dual-GPU benchmark (2× RTX 3080 20GB) comparing llama.cpp (row/tensor split) vs ik_llama (graph split) on Qwen3.6-27B-Q8_0. Row split: 1732 t/s prompt, 23 t/s generation, VRAM 18.2/18.5 GB. Tensor and graph split results incomplete in excerpt.
Hands-on experience with MTP (Multi-Token Prediction) speculative decoding in llama.cpp. MTP assistants are not interchangeable: identical names and architectures don't guarantee same performance. Gemma 4 26B Q4: ~30 t/s → 55-62 t/s with correct assistant. Unquantized assistant models outperform Q4 versions (~10 t/s faster).
EAGLE3 merged into llama.cpp after 6 months of development. The helper model receives guidance from the main model, unlike MTP where it operates independently.
InfiniteKV compresses KV cache into 104-byte searchable records stored in RAM or disk instead of deleting old tokens. Mistral-7B correctly answers at token 76,747 (2.3× its 32,768 training window). One million tokens requires ~3 GB instead of 122 GB.
LLM context compression technique achieves 16x compression ratio, outperforming traditional KV cache approaches. Method significantly reduces memory usage while maintaining response quality.
HCPD, a hallucination detection method without access to model internals or external references. An LLM agent adaptively decomposes judgment into weighted, interpretable criteria aligned via weak supervision on semantic consistency. Code released.
Study of internal anchoring mechanisms in language models. Researchers localize circuits responsible for anchoring bias (where irrelevant numbers influence answers) in Qwen and Llama 7B-8B models. Edge-level attribution methods recover this signal more faithfully than node-level methods.
llama.cpp benchmark on Intel 250K Plus CPU: optimizing --threads argument yields +80% performance gain (49 → 88 tok/s). 16 threads optimal vs 6 threads (P-cores only). Using all 18 cores drops performance without throttling detected.
User with RTX 3080 Ti 12GB + RTX 3080 20GB optimizing asymmetric dual-GPU inference. Gemma 4 31B Q4_K_XL reaches 20t/s with standard cache, 70t/s when compressing K/V cache to q4_0. Seeks clarification on GGUF memory expansion and dual-GPU configuration advice.
Discussion on speed optimizations for llama.cpp with MoE models on multi-GPU setups. Author explores -ngl, -ncmoe, -fitt, -ub flags and their impact on throughput (50→120 tps in prompt processing). Questions practical relevance of these optimizations for AI career prospects.
Community discussion on NVFP4 in llama.cpp. Users compare NVFP4 against Q4-Q8 quantizations for 8GB GPUs (RTX 4060, AMD, Intel). Questions: NVFP4 quality vs Q6/Q8, benchmarks (speed, perplexity), recommended models (Qwen 3.5-9B, Gemma-4-12B). Resources: HuggingFace NVFP4 and GGUF lists.
Minimax plans to release M3 model weights openly on Friday. The open-weights release will enable developers to run the model locally without proprietary restrictions.
AutoMine, a scenario mining method using LLMs and VLMs, extracts safety-critical scenarios from autonomous driving logs. Combines semantics-preserving prompt augmentation with robust trajectory atomic functions and VLM-based perception. Achieves HOTA-Temporal score of 36.38 in Argoverse 2 Scenario Mining Competition at CVPR 2026.
AutoGen multi-agent framework for automated reinforced concrete barrier design. Achieves 98% design accuracy in closed-loop generation-evaluation-optimization cycle. An 8B model outperforms unconstrained 631B flagship models, reducing computational costs while meeting AASHTO-LRFD compliance standards.
LLMs become overconfident when reasoning budget exceeds a task-specific threshold. This phenomenon, termed Calibration Drift Under Reasoning (CDUR), is studied on Llama-3.1-8B and Llama-3.3-70B. Authors propose CABStop, a calibration-aware stopping rule that halts reasoning when confidence diverges from actual accuracy.
Activation steering to reduce sycophancy in Llama-3-8B-Instruct also suppresses agreement with factually correct statements. Dual-stance evaluation reveals sycophantic and factual agreement occupy distinct geometric subspaces, yet the steering direction projects equally onto both, illustrating a gap between readable and writable representations.