Matches in Nanopublications for { <https://w3id.org/np/RAszCcnWVTYomkTM0J8K9p0PSp-gwfAIP8oDv_sfGKsdQ#assertion> ?p ?o ?g. }
Showing items 1 to 23 of
23
with 100 items per page.
- assertion comment " You only learn a few parameters, with your parameter "efficient" finetuning. The rest is💩 A whole line of works🧵 shows that by throwing redundancy we can get better LoRas, keep less memory and of course model merge https://twitter.com/LChoshen/status/1833879920348422216/photo/1 ComPeft shows you can improve LoRAs by pruning aggressively and making the remaining weights binary (+/-) It also means parameter efficiency still relies on overparametrization(but only during training) https://x.com/prateeky2806/status/1727589818618523783 Laser shows it on full models https://x.com/pratyusha_PS/status/1739025292805468212 https://twitter.com/LChoshen/status/1833879922500080084/photo/1 In merging, many find that with only those few weights one can make a "multitask" model, keeping the important ones for each model and switching. those e.g. 1% of the weights also represent tasks well Many.. https://www.alphaxiv.org/abs/2408.13656 https://www.alphaxiv.org/pdf/2405.07813 https://www.alphaxiv.org/pdf/2310.01886 Those works are focused on efficient multitask learning that compresses the models, can keep many models and switch between them as necessary. Another option to compress is to SVD the LORA, separately or to a shared space, saving the tiny differences https://x.com/RickardGabriels/status/1810368375455207470 And just because we discussed compression, of course this is all just "model compression" if you want to compress to just save space, there are smarter ways: https://github.com/zipnn/zipnn " assertion.
- assertion summarizes 2407.00066 assertion.
- assertion wasAttributedTo 0000-0002-0085-6496 provenance.
- assertion wasAttributedTo RAoSadUw99CeqDlR2400018nqTzR_38fT86OrTzk16Vts provenance.
- assertion creator RAoSadUw99CeqDlR2400018nqTzR_38fT86OrTzk16Vts assertion.
- assertion agreesWith 1810368375455207470 assertion.
- assertion agreesWith 1739025292805468212 assertion.
- assertion wasGeneratedBy activity provenance.
- assertion wasAssociatedWith LChoshen provenance.
- assertion endorses 2407.00066 assertion.
- assertion recommends zipnn assertion.
- assertion keywords "finetuning" assertion.
- assertion keywords "low-rank-adapters" assertion.
- assertion keywords "model-compression" assertion.
- assertion keywords "multitask-learning" assertion.
- assertion keywords "serving-systems" assertion.
- assertion discusses 1727589818618523783 assertion.
- assertion discusses 2408.13656 assertion.
- assertion discusses 2310.01886 assertion.
- assertion discusses 2405.07813 assertion.
- assertion includesQuotationFrom 1810368375455207470 assertion.
- assertion includesQuotationFrom 1739025292805468212 assertion.
- assertion linksTo 1833879920348422216 provenance.