If you owe enough in delinquent American federal taxes, the U.S. government can take your passport. The IRS certifies “seriously delinquent” tax debt to the U.S. State Department. This is an individual's unpaid, legally enforceable federal tax debt, including interest and penalties, that totals more than $55,000. (That amount is adjusted yearly for inflation.)