[CL] Spike No More: Stabilizing the Pre-training of Large Language Models
https://t.cn/A6jAvpLM
研究了大型语言模型在预训练过程中经常出现的损失峰值问题。通过理论分析，确定了导致损失峰值的两个主要因素，并提出防止梯度爆炸的要求。介绍了一种初始化方法和对嵌入层的简单修改的组合，以满足这些要求。通过各种实验证明，这种组合对于预训练过程中的损失峰值问题具有有效性。

前天simulation口试。昨天去sweco做演讲，下车的地方旁边是俄罗斯大使馆，公交站名是解放乌克兰。
口试和pre都结束了。难得清闲。昨晚看完了拾荒者统治Scavengers Reign。今天一起做叉烧肉。好好吃啊。下楼一出电梯偶遇半年多没见的Fran。一分钟十来句英文密集地抛过来。

- Did 2023 treat you well？
- Yeah I think so. [太开心]

来自【mp.weixin.qq.com】的分享 https://mp.weixin.qq.com/s?fontRatio=1&pre_clickid=14378037379349361472-1705094241-138211&search_click_id=14378037379349361472-1705094241-138211&__biz=MzA5MjM5ODg4NQ==&mid=2650475782&idx=1&sn=03003a16060278022fc7645f56d1d8fb&passparam=searchid%3D16517422732006010210&scene=26&subscene=343&sessionid=1705068466&ascene=64&fasttmpl_type=0&fasttmpl_fullversion=7029148-zh_CN-zip&fasttmpl_flag=0&realreporttime=1705094241125&clicktime=1705094241&enterid=1705094241&devicetype=android-33&version=28002d34&nettype=WIFI&abtest_cookie=AAACAA%3D%3D&lang=zh_CN&countrycode=CN&exportkey=n_ChQIAhIQIhH%2FGV5huX3mccnH%2BusSWRLcAQIE97dBBAEAAAAAAN28As%2FPsbYAAAAOpnltbLcz9gKNyK89dVj0HXBdQqdbilzHa51MTOSQCHEgYam4Ndo8PSPWvSoKhH3E5ffy5G1R%2FLMBMgPQrVknLnZicQj59GNkOYgziPhiWydETOeLgE%2B%2Bv5VSYo%2BZ%2BTXdOhBMziVbmE%2Bk%2B9rVI8bid%2Fljg2A6wTWYyc2kJ9LWyNL5eLCfUDep9s8uCmo%2FgtPoSjoGR%2FDbTWygw%2FMyr4GLA3aE2iuGNpYJTnVkG%2BOMgmyYIEe%2BjHnYpeY7DKGp%2BueouXgsXZw%3D&pass_ticket=cXtH%2BTF5rgv1Mo0UmFyMCH3YiRtzHoowiQN1AmR9hv%2FjmkaWkqXXL%2FnVGhILvv%2BWw5wsz6wdHNqw6LV6IhMmQg%3D%3D&wx_header=3&t=2