Hey man, I think I came across your blog at some point before while trying to figure out my own power plan for this beast (check my comment to OP for more context), so kudos to you for that.

I would say that power limiting is a potential work around, and it should work perfectly fine for inference, but when it comes to trainning you will want to squeeze every ounce of power. So, depends on your goal.

What CPU/Mobo/Storage are you running with those two 3090s for a 700w to work? I am gonna say, if at any point you're pushing more than 500w out of that PSU, you might be risking the 80% safety rule. I would have at least used a 850w just to be safe with two 3090s + rest of hardware.

Thank you for this post. I’d read it in ~June and it helped quite a bit with manual ‘nvidia-smi’ runs. I just recently created the systemd service description and am still delving related power and performance possibilities.