What's new

Fixed ✅

• Fixed default value of --tokenizer argument to scripts/prepare_tulu_data.py to be an absolute path, not relative path, the script can be run from other directories.
• Added the option to directly pass input embeddings to OLMo and OLMoForCausalLM.
• Added support for Python 3.8.
• Added code to throw an error if output_attentions is set to True in forward call to OLMoForCausalLM. This functionality hasn't been implemented yet.
• Fixed running with data loading workers on LUMI

Added 🎉

• Added output_hidden_states argument and associated functionality to OLMo and OLMoForCausalLM to return model intermediate hidden states.
• Added MMLU downstream evaluation tasks, with prompt variations.
• Added support for PyTorch v2.2.
• Added ability to show logs from all ranks
• Added option for QKV clipping.

Changed ⚠️

• Refactor torch.load monkey patching for legacy checkpoint unsharding in anticipation of unsharding implementation change.

Commits

c4996325 Add option for QKV clipping (#489) 31d85287 Pull checkpoint patch from mitchish-gqa-2 03d7643c Merge pull request #486 from allenai/shanea/monkey-patch-ctx-manager fd3a57ba Merge pull request #483 from allenai/shanea/storage-cleaner-unshard-improvements 1d264e44 Merge pull request #481 from allenai/WorkersOnLumi 70ad30c6 Merge pull request #480 from allenai/Firehose 493c0b83 Add MMLU prompt variants (#484) cb711e24 Add support for PyTorch v2.2 (#476) 67d24f5e Merge pull request #468 from allenai/mmlu-downstream 0c58beed Fix bug when clipping is disabled 922db6aa Only run the profiler through a single cycle (#463) 37ca7893 Merge pull request #462 from allenai/epwalsh/fsdp-wrap-patch cc367095 Add attn bias arg to HF wrapper (#458) 7f7abbb6 Merge pull request #451 from sarahwie/main 9fd9130d Add support for Python 3.8 (#448) d9c09937 Require Python>=3.9 for now 97296e61 Merge pull request #442 from allenai/shanea/add-input-embedding-arg 3be4c1ec add link to W&B logs for 1B run d7d4de4c Add link to OLMo-7B-Twin-2T W&B logs cf121084 Update README.md (#429) 15af6688 freeze official configs for reproductions (#421) 7739fe17 Add link to W&B logs for OLMo-7B 80db5e3d Fix default value of --tokenizer 6765317e Add link to paper in README badge