I have a factory amplifier in my car under the seat, and it doesn't seem to give much power to the speakers (it sounds like a car stereo w/o an amp). I was wondering if there's a way to test it using a regular multi tester like you'd use for batteries or electrical outlets.

Not really. Accurately measuring the output of a car stereo can't really be done with that sort of meter. It's not a constant voltage (DC) and it's not a regular AC power supply, either, so a regular meter won't tell you anything useful. Maybe if you had a single tone CD and a oscilloscope, but that's probably not in your toolbox.