Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

In three linesNew pre-training approach combining MLM (Masked Language Modeling) and JEPA (Joint Embedding Predictive Architecture) for text encoders. Hybrid model trained on English Wikipedia with identical compute budget. Results: more uniform embeddings (-0.16 vs -0.05), richer spectral geometry, better semantic-to-lexical balance on GLUE benchmarks.

Summary generated by Claude — human-verified