modernbert-CGEdit-AAE

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0152

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 0.06
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
0.8456	1.0	26	0.2637
0.2834	2.0	52	0.1345
0.1724	3.0	78	0.0872
0.0933	4.0	104	0.0557
0.0516	5.0	130	0.0387
0.0330	6.0	156	0.0418
0.0182	7.0	182	0.0240
0.0112	8.0	208	0.0218
0.0110	9.0	234	0.0164
0.0059	10.0	260	0.0145
0.0039	11.0	286	0.0170
0.0043	12.0	312	0.0131
0.0030	13.0	338	0.0168
0.0034	14.0	364	0.0123
0.0028	15.0	390	0.0133
0.0023	16.0	416	0.0162
0.0020	17.0	442	0.0155
0.0018	18.0	468	0.0147
0.0018	19.0	494	0.0143
0.0018	20.0	520	0.0146
0.0015	21.0	546	0.0146
0.0014	22.0	572	0.0147
0.0014	23.0	598	0.0146
0.0013	24.0	624	0.0146
0.0012	25.0	650	0.0148
0.0013	26.0	676	0.0147
0.0011	27.0	702	0.0148
0.0011	28.0	728	0.0149
0.0012	29.0	754	0.0148
0.0011	30.0	780	0.0149
0.0010	31.0	806	0.0149
0.0011	32.0	832	0.0149
0.0010	33.0	858	0.0150
0.0010	34.0	884	0.0150
0.0010	35.0	910	0.0151
0.0009	36.0	936	0.0150
0.0009	37.0	962	0.0151
0.0009	38.0	988	0.0151
0.0009	39.0	1014	0.0151
0.0008	40.0	1040	0.0152
0.0008	41.0	1066	0.0151
0.0008	42.0	1092	0.0152
0.0008	43.0	1118	0.0152
0.0009	44.0	1144	0.0152
0.0008	45.0	1170	0.0152
0.0008	46.0	1196	0.0152
0.0009	47.0	1222	0.0152
0.0007	48.0	1248	0.0152
0.0008	49.0	1274	0.0152
0.0008	50.0	1300	0.0152

Framework versions

• Transformers 5.0.0
• Pytorch 2.5.1+cu121
• Tokenizers 0.22.1