pixel_100m

This repository contains a PIXEL checkpoint exported for use with the PIXEL codebase. It includes the model checkpoint, tokenizer files, exported config metadata, and this model card.

What This Model Is

pixel_100m is a decoder-only transformer checkpoint from the PIXEL project. This bundle is intended to be used with the PIXEL runtime rather than the Transformers AutoModel API.

Architecture

• Approximate parameter class: ~76,466,688
• Vocab size: 1262
• Context length: 1024
• Layers: 12
• Hidden size: 768
• Attention heads: 12
• Key/value heads: 4
• Intermediate size: 2048
• RoPE base: 500000
• Uses MoE: False

Included Files

• latest.pt: PIXEL checkpoint
• manifest.json: exported checkpoint pointer
• pixel_tokenizer.model: SentencePiece tokenizer model
• pixel_tokenizer.vocab: SentencePiece tokenizer vocab
• pixel_model_config.json: exported typed model config
• pixel_training_config.json: exported training config when available

Training Snapshot

• Training size preset: 100m
• Total steps saved in checkpoint: 10
• Sequence length: 32
• Batch size: 1
• Gradient accumulation: 2

Runtime Snapshot

• Device: cpu
• GPU count: 0
• Dtype: torch.float32

Usage With PIXEL

Clone the PIXEL codebase, place or download this bundle, then run:

python infer.py --model checkpoints/pixel_100m/latest.pt --prompt "Hello from PIXEL"

Make sure the checkpoint and tokenizer come from the same export bundle.

Limitations

• This checkpoint is not guaranteed to be instruction-tuned.
• Output quality depends on the training corpus and training duration used for this run.
• This bundle is PIXEL-specific and is not advertised as a drop-in Transformers checkpoint.

Export Provenance

• Source checkpoint: latest.pt
• Source tokenizer model: pixel_tokenizer.model
• Source tokenizer vocab: pixel_tokenizer.vocab