What's new

Added 🎉

• Added support for Grouped Query Attention.
• Added commonsense_qa and social_iqa downstream evaluation tasks
• Makes it possible to read from http/https the same way we read from s3/r2.
• Added MMLU multiple choice (A/B/C/D) 5-shot variant downstream tasks
• Tokenizer patch
• Added option to specify number of model replicas when using hybrid sharding.

Changed ⚠️

• Rename Olmo to OLMo everywhere in the codebase
• Disabled automatic garbage collection during training, instead we run manually at regular intervals to avoid ranks getting out-of-sync with their own gc.

Removed 👋

• Removed AMDLayerNorm, since the original layer norm bug has been fixed and we don't need this workaround anymore.
• Removed OLMoParallelBlock.

Fixed ✅

• Don't log garbage on nodes that aren't rank 0
• Don't crash in the HF code when we are referring to a tokenizer in a local file
• Point official training scripts to publicly available URLs
• Corrected the resize_token_embeddings method in the OLMoForCausalLM class to properly update the token embeddings when resizing the vocabulary.
• Changed tie_weights method to a no-op as weight tying is handled in olmo/model.py
• Fixed the size calculation for qk layer norm
• Fixed pipeline test failure that occurs due to a bug in transformers version 4.39.1
• Make hf_olmo compatible with transformers versions >=4.40.0

Commits

3b16e218 Merge pull request #556 from allenai/shanea/make-hf-olmo-support-new-transformers ccf7bf0a Merge pull request #555 from allenai/shanea/wandb-cancel-failure-bypass 7be71cd7 use correct PG when collecting metrics with HYBRID shard (#551) 06786a7b Merge pull request #548 from allenai/shanea/fix-olmo-name-hf 4ed135e2 Merge pull request #540 from allenai/shanea/hybrid-sharding-num-groups-2 2eae9888 Merge pull request #546 from allenai/shanea/add-olmo-1.7-7b-checkpoints d2afcaaf Add cfg option --scheduler.warmup_min_lr (#542) 9d408986 Merge pull request #537 from allenai/AkshitaB-tokenizer-patch 62c7954e Merge pull request #536 from allenai/shanea/storage-cleaner-wandb-path-from-checkpoint 657a55e8 Merge pull request #494 from allenai/shanea/storage-cleaner-move-entry 9a0a84a1 Merge pull request #527 from allenai/PublicTrainingData 0de5fdc8 Merge pull request #501 from djliden/dl/fix-embedding-resize 4792f94c Adds a new experimental sharded checkpointer from OLMo-core (#532) 1c129802 make garbage collection interval configurable (#533) db2dee2e Merge pull request #503 from djliden/dl/hf-weight-tying 8fad6498 Merge pull request #534 from allenai/shanea/fix-transformer-cache-position-regression 71f7014e Merge pull request #528 from allenai/add-mmlu-mc-5shot 8472d0b4 Merge pull request #521 from allenai/davidbrandfonbrener-patch-1 194012a0 Merge pull request #523 from allenai/davidbrandfonbrener-patch-2 8949bd85 Added deprecation for memmap (#517) 83cc8b10 Merge pull request #464 from allenai/olmo7-ablations f8aef844 Merge pull request #509 from allenai/epwalsh/manual-gc 0ac82a93 Merge pull request #508 from allenai/RunDataloader 74de51d3 Merge pull request #414 from allenai/mitchish65-2 417af0ed Merge pull request #504 from allenai/add-csqa-siqa 666da70f Patch other S3 methods with 404 detection fix 0b6e28c0 Fix checking HTTP status code for boto3 responses 0b835a8d Merge pull request #500 from allenai/shanea/expose-official-checkpoints 50da7a49 Add work-arounds for new-style checkpointing issues 6d42d7ab Fix hang when training is canceled 7eb7f3d6 Merge pull request #455 from gahdritz/main ed47c298 Merge pull request #453 from hxdtest/only_rank0_log_metrics ad8198e4 Merge pull request #495 from allenai/add-basic-math 1511fed2 Merge pull request #487 from allenai/fix-mmlu-prompt-bug c2840e4f Merge pull request #493 from allenai/shanea/storage-cleaner-move-improvements 658f7cc1 Merge pull request #466 from allenai/rename eb5b2dad Merge pull request #490 from allenai/RemoveAMDLN 752353bf Merge pull request #488 from allenai/shanea/optimize-unsharding-2