Hugging Face raises $100 million to accelerate open-source and collaborative machine learning development. The funding supports platform expansion, model development, and infrastructure tools.
OpenAI introduces a hierarchical method for text-conditional image generation using CLIP latents. The approach improves quality and consistency between text descriptions and generated images.
OpenAI releases new versions of GPT-3 and Codex that can edit or insert content into existing text, beyond simple text completion.
Hugging Face integrates constrained beam search into Transformers to guide text generation according to specified constraints. This technique improves control over model outputs by applying rules during decoding.
OpenAI built a neural theorem prover for Lean that solves challenging high-school olympiad math problems, including AMC12, AIME competitions, and two problems adapted from the IMO.
OpenAI introduces a new API endpoint for text and code embeddings. This tool enables semantic search, clustering, topic modeling, and classification tasks.
OpenAI releases a contrastive pre-training method for generating text and code embeddings. The technique improves semantic representation of data for search and similarity tasks.
Gradio, open-source platform for building web interfaces for ML models, joins Hugging Face. The acquisition strengthens HF's ecosystem for model deployment and demonstration.
OpenAI fine-tuned GPT-3 to answer open-ended questions with improved factual accuracy using a text-based web browser. This approach enhances the reliability of language models.
Perceiver IO is a scalable, fully-attentional model that processes any modality (text, image, audio, video) through unified attention mechanisms. The architecture eliminates convolutions and recurrence, enabling efficient cross-domain generalization with a single model design.
Hugging Face details training CodeParrot, an open-source code model. Full process: 20GB Python code dataset, custom tokenizer, GPT-2 architecture, fine-tuning on 100 GPUs. Model and source code released.
Hugging Face releases Optimum, an optimization toolkit for scaling Transformer models. It includes quantization, distillation, and compilation to reduce latency and memory consumption in production.
OpenAI releases TruthfulQA, a benchmark measuring language models' ability to provide factual answers rather than mimicking common human misconceptions. The dataset contains adversarial questions designed to test whether models reproduce popular false beliefs.
OpenAI releases an improved version of Codex, its AI system that translates natural language to code, in private beta via its API.
Hugging Face releases Accelerate, a library for training and inference on multiple GPUs/TPUs without code changes. PyTorch-compatible, it simplifies distributed computing and resource optimization.
BigBird introduces block-sparse attention mechanism reducing transformer quadratic complexity to linear. The approach combines local, global, and random attention to process sequences up to 4096 tokens, improving efficiency without sacrificing performance.
OpenAI discovers multimodal neurons in CLIP that respond to the same concept regardless of presentation format (literal, symbolic, or conceptual). This finding explains CLIP's accuracy in classifying unusual visual renditions and advances understanding of associations and biases learned by such models.
OpenAI scaled Kubernetes clusters to 7,500 nodes to support training of large models (GPT-3, CLIP, DALL·E) and iterative research. Critical infrastructure for language model scalability.
Hugging Face integrates ZeRO (Zero Redundancy Optimizer) from DeepSpeed and FairScale to reduce GPU memory and accelerate model training. ZeRO partitions optimizer states, gradients, and parameters across GPUs, enabling training of larger models with fewer resources.
Hugging Face achieved 100x speedup in transformer inference for API customers through quantization, dynamic batching, and KV cache optimization. Models like Llama 2 and Mistral show measurable latency and throughput gains.
Hugging Face documents porting the WMT19 translation system from fairseq to the transformers library. Technical migration of a neural machine translation architecture to the transformers ecosystem, with reproduction of WMT19 benchmark results.
OpenAI licenses GPT-3 technology to Microsoft for integration into Microsoft's products and services.
OpenAI applies generative language modeling to automated theorem proving. The approach combines language models with search techniques to solve formal mathematical problems, improving AI systems' ability to reason over complex logical statements.
OpenAI trains language models for summarization using reinforcement learning from human feedback (RLHF). The approach improves the quality of generated summaries.
OpenAI demonstrates that a transformer trained on pixel sequences generates coherent image completions, mirroring text generation capabilities. The best generative model achieves competitive performance in unsupervised image classification versus top convolutional networks.
OpenAI releases analysis showing that since 2012, compute required to train a neural network to equivalent ImageNet classification performance halves every 16 months. Training to AlexNet-level performance now requires 44x less compute than 2012 — far exceeding Moore's Law's predicted 11x improvement. Algorithmic progress outpaces classical hardware efficiency for heavily-invested AI tasks.
OpenAI introduces Jukebox, a neural network generating music including rudimentary singing as raw audio across various genres and artist styles. Model weights, code, and an exploration tool are released.
OpenAI releases Microscope, a collection of visualizations of every significant layer and neuron across eight vision model organisms. The tool enables analysis of internal features in neural networks to advance interpretability research.
Technical guide on decoding methods for text generation with Transformers: greedy search, beam search, sampling and variants. Explains trade-offs between quality, diversity and computational cost.
Hugging Face publishes a comprehensive guide for training a new language model from scratch using Transformers and Tokenizers libraries. The tutorial covers data preparation, custom tokenizer creation, and model training on a custom corpus.
OpenAI trains an AI agent using large-scale deep reinforcement learning to master Dota 2, a complex strategic game. The agent competes with top professional players using neural networks and advanced optimization techniques.
OpenAI demonstrates that double descent phenomenon occurs in CNNs, ResNets, and transformers: performance improves, then worsens, then improves again with increasing model size, data size, or training time. The effect is universal but not yet fully understood.
OpenAI releases Procgen Benchmark, 16 procedurally-generated environments to measure how quickly reinforcement learning agents learn generalizable skills.
OpenAI releases Safety Gym, a suite of environments and tools for measuring progress towards reinforcement learning agents that respect safety constraints during training.
OpenAI releases a benchmark for evaluating safe exploration in deep reinforcement learning. The study measures agents' ability to explore efficiently while respecting safety constraints, a key criterion for real-world applications.
OpenAI releases the 1.5B parameter version of GPT-2 with code and model weights, completing its staged release plan. Goal: test a responsible publication process and provide detection tools for GPT-2 outputs.
OpenAI trained two neural networks to solve a Rubik's Cube with a humanoid robot hand. Reinforcement learning in simulation uses OpenAI Five code and a new Automatic Domain Randomization (ADR) technique. The system handles unseen situations during training, such as being prodded by a stuffed giraffe.
OpenAI observes emergence of complex strategies in a simulated hide-and-seek environment. Agents discover six distinct strategies and counterstrategies through multi-agent co-adaptation, some unanticipated. This suggests multi-agent interaction may produce extremely complex and intelligent behavior.
OpenAI releases the full GPT-2 model (774M parameters) following staged releases since February (124M, 355M). Includes open-source legal agreement for model-sharing partnerships and technical report on publication norms coordination with the AI research community.
Microsoft invests $1 billion in OpenAI to support AGI development. The companies partner to build a hardware and software platform on Azure, with Microsoft becoming OpenAI's exclusive cloud provider.