Arize AI Community Icon

Home
Events

Arize AX Releases
Arize AX Support
Arize News
Arize Observe
Discussions
Introductions
Phoenix Support

Powered by Tightknit

Advice Needed for Building Medical Notes from Patient Transcripts | Arize AI Community

Advice Needed for Building Medical Notes from Patient Transcripts

·Mar 06, 2024 09:36 PM

Hi all! I’m building a project that generates medical notes given a transcript of a patient-doctor interaction. i figured this community would be a good place to ask for advice. im having trouble processing LARGE transcripts (1+ hour long). gpt-4 (api) context window is not large enough, and i don’t currently have access to the 32k model. but even if i did, i imagine that is not the most effective/efficient way of doing this (in terms of cost). are there open source LLMs that traditionally perform well on these sort of tasks with large context windows? another thing: customization. i’ve been hyper-customizing the gpt prompt to my needs and i mean it works well maybe 70% of the time. But is there a better way to do this? More specifically, I want to customize the style of the LLM’s output. I tell it to not give me bullet points, but 10% of the time it will give it anyways (that’s just one example). Is this where fine-tuning should come in? I have a good amount of data (input transcript, properly-styled output note) A lot of this is new to me. So really I’d be truly grateful for any advice you guys may have :)

9 comments

· Sorted by Oldest

Xander S.
·
im having trouble processing LARGE transcripts (1+ hour long). gpt-4 (api) context window is not large enough, and i don’t currently have access to the 32k model. but even if i did, i imagine that is not the most effective/efficient way of doing this (in terms of cost). are there open source LLMs that traditionally perform well on these sort of tasks with large context windows?
You might try a map-reduce or some other strategy. It sounds like a summarization task?
Xander S.
·
another thing: customization. i’ve been hyper-customizing the gpt prompt to my needs and i mean it works well maybe 70% of the time. But is there a better way to do this? More specifically, I want to customize the style of the LLM’s output. I tell it to not give me bullet points, but 10% of the time it will give it anyways (that’s just one example). Is this where fine-tuning should come in? I have a good amount of data (input transcript, properly-styled output note)
If you've squeezed the most you can out of prompt engineering, fine-tuning could be a good option. It's easy to do with the OpenAI API in particular.
Xander S.
·
Jason was just telling us that repeating parts of the prompt multiple times and in multiple places (e.g., at the start and at the end) is needed sometimes to get the desired output, even for GPT-4-turbo.
🙌1
Xander S.
·
Providing examples of input/ output pairs is also helpful,
Chaninder R.
·
thank you for the response Xander S.! i don’t believe i should summarize it. there are specific details throughout the transcript that are needed for the final note. summarization would likely cause it to lose too much info. what do you think?
Chaninder R.
·
gothca ok. and is this an appropriate tasks for fine-tuning? i’ve heard that fine-tuning should be done only when there’s a task that the base LLM can’t do. Does styling count?
Xander S.
·
thank you for the response Xander S.! i don’t believe i should summarize it. there are specific details throughout the transcript that are needed for the final note. summarization would likely cause it to lose too much info. what do you think?
I'm not sure, it probably depends on the task.
👍1
Xander S.
·
gothca ok. and is this an appropriate tasks for fine-tuning? i’ve heard that fine-tuning should be done only when there’s a task that the base LLM can’t do. Does styling count?
The general recommendation I've heard is to go as far as you can with prompt engineering before you resort to fine-tuning. That being said, fine-tuning is straightforward with the OpenAI API (if you're able to send your data to a third-party API). It is something you can try with relatively little effort.
Chaninder R.
·
got it, ok ill give it a short. thanks!