We were manually testing in the chat interface, we noticed the second time we tried we got the right answer.
We thought the only difference in the context window was the previous chat question, so we thought let’s just double it up on every question.
TEMPLATE + “ \n\n “ + TEMPLATE
Now my next question, probably rhetorical, unless you know, is WHY. I feel like there will be a research paper explaining why that helps performance / reasoning
For the life of me, nothing strikes me as obvious
I think we are all trying to figure out the WHY. A somewhat of a handwavy guess is the repetition helps improve the attention that is attending to the "relevance" question answer. I think without more clear evidence and without understanding the internals, its our best guess.
Claude is still is not nearly as good as GPT-4 even with the improved prompt. We are testing different prompts to see if there is any issue with the prompt template
