PARTNERSHIP
From last place to the frontier in a week
Logan Grasby, Mason Pierce, Sean Geiger, Jackson Stokes · October 15, 2026
On October 5, Nolla Health became the first organization in the United States approved to use AI to write a patient’s first prescription without direct doctor oversight. The announcement was viewed 2.2 million times, and the team was featured in Bloomberg, Fox News, and ABC News.
Nolla is starting with acne. It is common, well understood, relatively low risk, and visible in a photo, which makes it solvable with good data. However, as AI systems take on more medical tasks, they need to work more like doctors. Typically, a doctor will see the visual symptoms, but then talk to the patient to methodically rule out confounding conditions in a process called differential diagnosis.
In April, we trained NollaMD on Nolla’s data to see whether a small model, MedGemma 27B, could become state of the art at differential diagnosis. We tested this on a holdout set, where the model outperformed the frontier at a fraction of the cost of their models at the time. While the model was accurate at identifying conditions from images, the process of differential diagnosis is nuanced and as such the model still had a lot of room to improve. This time, we vastly improved the simulated patients and spent more of the training time on full conversations. The question we had was: how quickly can an open model pass the frontier, and by how much?
In a week we were able to nearly double the score of Nemotron 3.5 Super VL from 35% to 67%, beating GPT-6 Astra, Gemini 3.8 Flash, and Fable 5.1 by about 10% while decreasing the amount of output tokens to get to the correct answer by 22%. Here’s how we did it.
Nemotron 3.5 VL Super
The new Nemotron is an open-weight, multimodal model with 1M context and only 12B active parameters. Beyond its benchmark scores (which are great,) it has a degree of deployment ease and trainability that we haven't seen in multimodal models of its class before.
We got early access to put the model through its paces and assess it's trainability. We put the model in a training loop, and watched it learn how to accurately pull out features and reason through additional case information to identify the right diagnosis for a dermatological condition, out of over 3000 potential diagnoses.
Prepping the data
We started with about 2.5 million images of skin conditions, and after filtering and balancing, we ended up with 115,396 multiple-choice questions across 2,864 conditions, and held out 500 of them as our eval.
The harder data to prepare was the conversational environment. Models are bad at simulating real human behavior, and so it was critical that the model representing the patient matched real humans as closely as possible. In real conversations, we saw that patients are concise, impatient, and resistant to answering probing questions in depth. We carefully calibrated the patient model to match this behavior.
From this environment we generated 10,000 synthetic patients and their backgrounds, using the images from Nolla’s data.
In order to build the rubric for the conversational environment, we asked real clinicians to review generated conversations and point out things that they look for across many different conditions. We took that feedback and synthesized it into the rubric below.
The wrong model on paper
The things that matter most when selecting a model are how amenable it is to learning, how efficiently it can train, and how fast it runs in production. In our own experiments, Nemotron models are stable and quick learners, run efficiently on NVIDIA GPUs, and have native support in NeMo RL. When NVIDIA gave us early access to Nemotron 3.5 Super VL, we expected it to be a practical choice for this problem.
Before training, we ran Nemotron and three frontier models on our 500-question eval.
Nemotron came in last against the frontier models. On a larger held-out set of 7,026 questions, it averaged 2,213 output tokens per question and hit the 4,096-token limit on almost a quarter of them.
Training for visual diagnosis
To see how much Nemotron could learn, we trained it in two stages with GRPO in NeMo RL, on 32 NVIDIA B200s over about a week. The first was 400 steps on the multiple-choice questions.
Training reward and held-out validation both climbed steadily, with validation going from 40% to 65%. Because of time constraints, we had to cut this stage off at 400 steps to move on to the synthetic patient samples, but it was still improving when we stopped.
Generating rollouts took about 80% of each step, and because of that we used NeMo RL’s colocation feature so we could utilize all GPUs for inference. Partway through, to improve utilization even more, we halved the output limit to 4K tokens and doubled the batch size to 512. Dynamic sampling dropped groups where every attempt scored the same, so each step generated more than 512 rollouts to fill the batch. After making these changes, rollout throughput went from 1,450 to nearly 5,000 tokens per second, and time per step dropped from 13.7 minutes to 9.
On the 500-question eval from earlier, trained Nemotron scored 67%, ahead of all three frontier models. Its answers also got shorter, because training capped each response’s length and the model learned to fit inside it. On 7,026 held-out questions, it ran out of room under 4% of the time, down from almost a quarter. Average length fell 22% while accuracy doubled, giving 2.59 times as many correct answers per million tokens.
The training of this model was remarkably stable. We think this is because the model is not overfit to benchmarks out of the box; it's designed to be trained.
Training for differential diagnosis
After the visual training, we trained for another 69 steps on full conversations with the simulated patients. Each rollout here is a whole consultation rather than a single answer, so generation took even longer, and we again used NeMo RL’s colocation to run inference on every GPU. Even so, the slower rollouts meant we got through far fewer steps than in the first stage.
The reward came from a rubric we wrote in collaboration with physicians at Nolla. Gemini 3.8 Flash used it to grade each conversation and the clinical note the model produced at the end. Over the run, mean reward rose from 0.42 to 0.54.
Before training, the model abandoned a third of the 75 held-out consultations and got the diagnosis right 48% of the time. We evaluated the checkpoint from step 50, which spent 47% fewer tokens per finished note, completed every consultation, and got the diagnosis right 64% of the time.
We also scored the trained model and GPT-6 Astra against each part of the rubric. Nemotron scored higher on the quality of its differential diagnosis (95.6% vs 83.9%) and its SOAP notes (86.3% vs 75.1%), and somewhat higher on image observation and photo handling. The differential diagnosis row grades the quality of the reasoning, not whether the top answer was right. On picking the correct top diagnosis, the two were close, with Astra slightly ahead at 65.3% to 64.0%. Astra also scored slightly higher on question strategy, treatment plan, and tone. The clearest gap in Astra’s favor was triage urgency, 99.2% to 89.5%, which is where we’d focus next.
Serving cost
In a serving test on four NVIDIA RTX PRO 6000 Blackwell GPUs, the trained model generated 1,294 tokens per second, about 0.86 GPU-hours per million tokens. At $3 per GPU-hour, that is about $2.58 per million output tokens, before any caching or quantization.
Although Nemotron started last on Nolla’s evals, after only a week of training, it was able to match the frontier with plenty of room to go on training. Most teams in healthcare already have the data and the clinical expertise to do this. What they usually don’t have is time to build the patient simulator and the training setup around it, which is the part we do at TrainLoop. If you want a state-of-the-art model trained for your eval, email us at sales@trainloop.ai.
Thanks to NVIDIA for early access to Nemotron and for NeMo RL, to Nebius for the B200s, and to the Nolla Health team for their data and clinical expertise.