Deepanshu Is your goal to build a classifier and generate labels for the documents? Or is it being used in a more general LLM RAG Retrieval system?
1)
A lot of NLP teams are moving toward using LLMs as classifiers given the flexibility and strength but this can highly depend on your use case. If you have low volume, I probably would start with an LLM "Ask GPT-4 to classify your documents into x classes [X, Y ,Z] based on description" . If you have a high volume of classifications you need to generate daily, 100k-millions+, the approach you are thinking of create an embedding then clustering (or add a classifier) is an option.
In terms of how much to label, in the world of using LLMs as classifiers that are already pre-trained, the number of labels you need are orders of magnitude less than if you are trying to build a training set. The labeling is just to benchmark your system, can be 1000s of samples depending on the number of categories. So the question is, are you training something or is the output of LLM/Embedding good enough to classify/label without training. That will depend on system volume.
2) Given the latest context windows and strengths of LLMs I would try the text-embedding-3-small and text-embedding-3-large models that replace Ada.
If you are using an LLM as the final classifier & embedder (that is what goes on internally), embeddings won't matter.
If you are building a classifier at the end of the embedding stage, you can get more fancy in embeddings but start simple. Colbert would be a search and retrieval example of how complex you can get.
