This reminds me of bias tuning, a LoRA competitor. One can get decent adapters by only finetuning a vector added to each linear layer activations. I think I saw it first while reading [1] but there are other instances.

[1] https://arxiv.org/pdf/2304.15010.pdf