Training a Reproducible Long Context Text Embedder

This technical report details the training of the nomic-embed-text-v1 model, which is a fully reproducible, open-source, long-context English text embedding model. The model is capable of handling a context length of 8192 tokens and surpasses the performance of OpenAI's Ada-002 and text-embedding-3-small models in both short and long-context tasks. The report includes the release of the training code and model weights under an Apache 2 license, along with a dataset containing 235 million curated text pairs that facilitate complete replication of the model. The document also discusses the importance of text embeddings in NLP applications and outlines the limitations of existing models with shorter context lengths. It presents a comparison of various models and highlights the unique attributes of nomic-embed-text-v1, including its performance metrics against other models in benchmark evaluations. The technical details regarding the training data and architectural modifications implemented to accommodate long sequence lengths are also included.