It’s no secret that the housing market is competitive, especially in big metropolitan areas. Winning a bidding war for the home you want can mean making a bid above the asking price, too.

Where do young, millennial homeowners find the money for a down payment? According to a recent Bank of the West survey of more than 600 adults ages 21-34, one-third of homeowners either withdrew from or took a loan against their retirement savings. The study also found that one in five millennials plan to do the same when they buy their first home.

Borrowing or withdrawing money from your IRA or 401(k) is allowed, but it’s not the wisest choice because the reduction in retirement savings isn’t the easiest thing to make up.

Here are the rules. According to federal law, a first-time homebuyer can take up to $10,000 out of a Roth IRA without penalty, if he or she is under 59 and a half years old and has had the account for at least five years. There are no additional fees because the owner of the account has already paid taxes on the funds.

A first-time buyer who has had a Roth IRA for less than five years can still take up to $10,000 out of the account but has to pay income taxes on the withdrawal.

With a 401(k), it’s advisable to borrow from the account, instead of withdrawing from it. You can take out up to 50 percent of the total amount in your account or $50,000, whichever is less. A loan means you don’t have to pay income taxes on the money and there is no withdrawal penalty unless you don’t pay back the loan on time.

The issue here is that you likely have to keep your job to maintain the terms of the loan. If you leave your company, you generally have to pay the loan back within 60 days.

It’s important to think carefully before you dip into your retirement savings to buy a house and end up in over your head.