Where the money goes
Earnest money is not paid to the seller. It is deposited with a neutral third party, usually the title company or a broker escrow account, and it sits there until settlement.
At closing it is credited to you. It reduces the cash you wire, dollar for dollar.
When it is at risk
Your contract contingencies are the protection. If you withdraw for a reason the contract allows, such as an inspection issue or a financing denial within the stated window, the deposit is generally returned.
Risk appears when you walk away outside those windows, or when a deadline passes without written notice. Deadlines in a purchase contract are real dates, not suggestions. Calendar them the day you go under contract.