How does a computer turns on?

In an AT style computer, the power switch is on the AC line to the power supply.

Pretty much everything gets power as the supply comes up.

For an ATX system, things are a bit different.

There is a 5 volt standby power line that provides 5 volts DC to the power control section of the motherboard at all times.
Depending on the motherboard design, other sections may be kept powered as well.

When you press the power button, circuitry on the motherboard signals the power supply to turn on the other voltages.
These come up almost instantaneously.
If the voltages come up properly, the power control circuit then generates a power good signal to the power supply, and the power supply stays on.

Other than the 5 volt standby line, power is applied to nearly everything in the PC at the same time.

Then, there is the battery on the motherboard that keeps the CMOS memory and real-time clock running.

Chas