ayyuce/medgemma-dermatology-dermnet-adapters

MedGemma Fine-tuned on Dermnet (LoRA Adapters)

This repository contains the LoRA adapters for google/medgemma-4b-it fine-tuned on the Dermnet dataset.

Model Details

• Dataset: Dermnet (~15k images)
• Classes: 23 Dermatology Conditions
• Method: QLoRA (4-bit quantization)

Supported Diagnoses

1. Acne and Rosacea Photos
2. Actinic Keratosis Basal Cell Carcinoma and other Malignant Lesions
3. Atopic Dermatitis Photos
4. Bullous Disease Photos
5. Cellulitis Impetigo and other Bacterial Infections
6. Eczema Photos
7. Exanthems and Drug Eruptions
8. Hair Loss Photos Alopecia and other Hair Diseases
9. Herpes HPV and other STDs Photos
10. Light Diseases and Disorders of Pigmentation
11. Lupus and other Connective Tissue diseases
12. Melanoma Skin Cancer Nevi and Moles
13. Nail Fungus and other Nail Disease
14. Poison Ivy Photos and other Contact Dermatitis
15. Psoriasis pictures Lichen Planus and related diseases
16. Scabies Lyme Disease and other Infestations and Bites
17. Seborrheic Keratoses and other Benign Tumors
18. Systemic Disease
19. Tinea Ringworm Candidiasis and other Fungal Infections
20. Urticaria Hives
21. Vascular Tumors
22. Vasculitis Photos
23. Warts Molluscum and other Viral Infections

Usage

To use this model, load the base model and then attach these adapters:

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
import torch
from PIL import Image

# 1. Load Base Model
base_model_id = "google/medgemma-4b-it"
base_model = AutoModelForImageTextToText.from_pretrained(
    base_model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

# 2. Load Adapters
repo_id = "ayyuce/medgemma-dermatology-dermnet-adapters"
model = PeftModel.from_pretrained(base_model, repo_id)
processor = AutoProcessor.from_pretrained(repo_id)

# 3. Inference
image = Image.open("your_image.jpg")
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this skin condition clinically."}
        ]
    }
]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=500)
print(processor.decode(outputs[0], skip_special_tokens=True))