Color has been working to improve access to healthcare for a decade, serving more than 7 million patients since it was founded. In 2023, they partnered with the American Cancer Society to help employers and health plans take control of cancer—the second most common cause of death in the United States and the leading driver of American healthcare costs.

Color’s copilot is helping clinicians create customized, comprehensive plans to start cancer treatment

Color Health uses OpenAI’s APIs to integrate patient medical data with clinical knowledge. The outcome is a copilot application that creates customized, comprehensive treatment plans for providers to review and use in their patient care. 

“Color’s vision is to make cancer expertise accessible at the point and time when it can have the greatest impact on a patient’s healthcare decisions,” says Othman Laraki, CEO of Color Health. 

"As a healthcare company, technology that improves access and equity has to go hand-in-hand with technology that supports patient safety and privacy. OpenAI's HIPAA-compliant data protection standards are key."

The copilot application’s output is analyzed by a clinician at every step and, if need be, modified before being presented to the patient. It works as follows:

1. It extracts, processes, and normalizes patient information, such as family history and individual risk factors, along with clinical guidelines and data from trusted sources. The Color team was particularly impressed with GPT-4o’s ability to extract and normalize information that was buried within pages of inconsistently structured and phrased information, often in different formats, such as with PDFs or clinical notes. 

2. Using this data, it answers key questions like, “What screenings should the patient be doing?” to identify missing diagnostics and generate a personalized screening plan. It also generates documentation required to complete any diagnostic workups, such as medical necessity documents and insurance pre-authorizations.

3. The clinician-in-the-loop evaluates the output, which includes source information. The clinician can edit the copilot’s output, which also helps refine future iterations.

4. Once the clinician-in-the-loop is satisfied with the result, they can add the information to the patient’s existing treatment plan.

Missed screenings and delayed cancer treatment impact patient outcomes

Screening, diagnosis, and treatment for cancer is notoriously complex and time-consuming. And every delay makes a difference: patients whose treatments are delayed by just four weeks face a 6–13% higher risk of mortality(opens in a new window).

Screening needs are also often highly individualized. More than a third of Color’s patients, for example, require earlier, different screening approaches based on individual risk factors not addressed by standard guidelines. “I've witnessed the complexities of developing personalized cancer screening plans for my high-risk patients,” says Dr. Keegan Duchicela, a primary care physician at Color. “The guidelines are constantly evolving, and individual risk factors aren't always immediately clear.” 

Beyond screening, diagnostic workups create more challenges. Documenting and performing a single patient’s diagnostic workup can take weeks, with the majority of patients arriving at their first oncology appointment without a complete workup. “Today, there are real gaps in oncology care based on where a patient receives initial diagnosis,” says Dr. Allison Kurian, a professor at Stanford University School of Medicine and clinically active oncologist. “Many of my patients require weeks to complete all of the tests and evaluations necessary to provide appropriate treatment, during which precious time is lost and additional administrative burden is placed on clinicians.”

Building a fast, safe, and secure proof of concept with OpenAI

Color began working with OpenAI in 2023, with the goal of using AI to improve cancer patient care and health equity. With the challenges of cancer screening, diagnosis, and treatment in mind, Color was looking for a solution that could:

• Interpret inconsistently-formatted patient data

• Analyze dense healthcare guidelines

• Protect patient data privacy

• Support clinician-in-the-loop workflow design to ensure patient safety

• Integrate with electronic health records (EHRs) and core hospital systems

During initial exploration, Color set up their approach for rapid experimentation, including testing the performance of GPT-4 and GPT-4o in complex tasks such as extracting information from PDFs of clinical guidelines for cancer diagnosis. These PDFs are often hundreds of pages of complicated diagrams that outline care paths based on diagnostic workup. Together, OpenAI and Color developed a method of asking GPT-4 Vision to describe screenshots of these diagrams that was most effective in maintaining output accuracy.

OpenAI also helped guide the Color team to prototype clinical workflows using the standard ChatGPT interface and generate sample cases using a custom GPT–gaining effective proofs of concept before committing extensive engineering resources.

With OpenAI’s expert guidance, powerful models, and HIPAA-compliant data protection standards, Color was able to focus on deconstructing complex medical decision-making, refining prompts, and designing clinician-in-the-loop workflows to create the initial version of the copilot.

For example, OpenAI engineers guided Color to use retrieval-augmented generation (RAG) instead of model fine-tuning to increase output quality and rewrite clinical documentation for easier processing by ChatGPT. Ultimately, after experimenting, Color selected OpenAI as its AI solutions provider, with GPT-4o at the core of its cutting-edge copilot application.

Reducing time to treatment for cancer patients

To measure the impact of this tool, Color is partnering with the University of California, San Francisco Helen Diller Family Comprehensive Cancer Center (UCSF HDFCCC). For the initial implementation, Color and UCSF will conduct a retrospective evaluation, followed by a targeted rollout. Based on the evaluation, there is potential to integrate the copilot into clinical workflows for all new cancer cases at UCSF. 

“UCSF is a leader in implementing cutting-edge technology to improve patient care,” says Dr. Alan Ashworth, PhD, FRS, President of the UCSF HDFCCC. “Patients frequently come to primary oncologists with incomplete diagnostic workups, and the time it takes to collate and accurately identify the completion of those workups prevents providers from working at the top of their license. We are interested in tools that can improve the efficiency and accuracy of pre-visit charting and avoid costly delays in treatment initiation for cancer patients at UCSF.” 

Dr. Karen Knudsen, CEO of the American Cancer Society, agrees. “The idea of combining AI technologies with digitally-enabled clinical workflows to expedite that process would be a positive advancement for all parties involved - the patient and their clinicians, as well as the payer covering the cost of treatment.”

Color is taking a measured approach in rolling out the copilot, and has started an initial phase-in for its own clinicians, applying the tool to a limited number of cases. These cases receive several layers of quality assurance:

• Healthcare providers using the copilot are able to identify 4x more missing labs, imaging, or biopsy and pathology results than those without the copilot.

• Using the copilot, it takes on average 5 minutes for clinicians to analyze patient records and identify gaps. Without the copilot, data is fragmented and can lead to weeks of delay.

Through the second half of 2024, Color intends to use the copilot application to provide AI-generated personalized care plans, with physician oversight, for over 200,000 patients.