What Happens When The IRS Calls Me?

Benjamin Brandt:         Welcome back to the Retirement Tax Podcast, I am your humble co-host as always Benjamin Brandt, and joining me as always, Mr. Steven Jarvis.

Steven Jarvis:     Ben, how are you today?

Benjamin Brandt:         I’m doing fantastic. It’s August, kids are getting ready to go back to school. The triplets are in all day preschool, Monday, Wednesday, Friday, so my wife and I are extremely excited about that. The triplets are very excited as well, they love school. How about you?

Steven Jarvis:     I’m just a little bit sore today. As we are recording this, I’m the day after my most recent half Ironman triathlon. And since I host some podcasts and put out a lot of content, as I was getting ready for the race, I was thinking about all these analogies that I was going to think of and record right after the race and all this stuff.

And then I got into the race and it was hot and exhausting and really, really hard. And I ignored all of those things and just did my race. But I finished it. I hit my goal and so, it was a lot of fun.

Benjamin Brandt:         So, remind me what a half Ironman is?

Steven Jarvis:     Half Ironman is a 1.2-mile swim, a 56-mile bike ride and a 13.1-mile run. So, something that 99% of the world never wants to do ever.

Benjamin Brandt:         That sounds terrible. So, how long does it take you to swim that far? One and a half miles seems easy.

Steven Jarvis:     Well, 1.2 miles, but it took me 38 and a half minutes.

Benjamin Brandt:         Oh my gosh. That part doesn’t sound easy.

Steven Jarvis:     No, no, it was good times though. But as much as everyone’s interested in my racing habits, we should probably talk about some tax stuff today as well.

Benjamin Brandt:         Yeah, so what we’re talking about today is what should I do if I get a letter or a phone call from the IRS? Maybe that’s the first thing we … are we going to get a letter or a phone call if the IRS is looking to get a hold of us?

Steven Jarvis:     Yeah, let’s start with that because I get that question sometimes of “What do I do if the IRS calls?” And for the most part, someone calling and telling you they’re from the IRS should put up a huge red flag for “Is this fraud, is this a scam?” Because the IRS never starts with a phone call.

It is possible the IRS could call you, but only after they’ve sent you letters and they’re going to the specific letter they sent you. They aren’t going to call demanding that you make payment over the phone.

So, in general, it is very, very much the exception that the IRS would ever call you. So, letters is really what we’re going to talk about today, because that can be somewhat common that you could receive a letter from the IRS.

Benjamin Brandt:         Okay, so lesson number one, the IRS is probably going to send you a physical piece of mail first — not an email, not a phone call. And if you get a phone call and the other person is irate, demanding you sent money, that’s definitely a scam. So, they’re going to want to a paper trail. They’re going to want to make it be official. So, the IRS has sent you a physical letter, what should we do?

Steven Jarvis:     Yeah, so you’re going to get this letter in the mail. It’s going to come from the Department of Treasury and you’re going to open it and you’re going to think, hey, it looks like somebody pecked this out on a typewriter about four decades ago. I don’t know why they don’t update the font, but I never like it when I see it. Yeah, that’s kind of an aside.

Typically, pretty close to the top of the letter, kind of your focus is going to go right to a box that has numbers in it. Most of these letters will have some sort of box with numbers that usually, is here’s how much more we’re asking you to pay.

And a lot of times our attention is going to get immediately drawn to that, and I completely get it, but we need to make sure we’re reading this letter. Especially if you’re a DIY tax preparer, you need to make sure you’re reading the whole thing.

If you work with a tax professional or financial professional, definitely something you want to share with them because it’s going to include a lot of details that are going to be very important, because these letters can come for a lot of different reasons.

And like so many other things with taxes, just because it’s on an official looking form, doesn’t make it automatically incontrovertible law. Sometimes, there are mistakes, sometimes there’s clarifications that we need to make, sometimes you are going to have to pay that amount. There’s kind of a range of possible outcome.

Benjamin Brandt:         Okay, so the step number one, don’t panic. Step number two, familiarize yourself with what they’re actually asking for. Don’t just get lost in the numbers, in the panic. If I got to write a check, try to get some context as to what they’re asking for.

So, at what point do we sort of pick up the phone and call … hopefully, we’re working with like a CPA or something like that?

Steven Jarvis:     Yeah, if we’re getting letters from the IRS, if you’re currently a DIY tax preparer, this might be kind of indication of maybe just for this year or just for this situation, I’m going to talk to somebody because we want to make sure we really know what’s going on.

So, maybe let’s talk through a couple of examples of some of the letters that I’m commonly seeing right now that we’re recording this near the end of June, so we’re about six months from the end of 2021. And so, some of the more automated letters that the IRS can send out are starting to hit people’s mailboxes and so, just in the last couple of weeks, I’ve seen a few.

One that I’ve seen a couple of times is a letter that basically says, “Hey, we looked at your tax return and we think you owe us more money.” And again, that’s why it’s so important to read those details. They are going to tell you why they think that. One of the things that I’m seeing is related to estimated payments.

Unfortunately, what can happen is that you think you made four quarterly estimated payments for 2021, but you decided that because you went back and looked at your bank account and you saw, oh, in April of 2021, I made a payment to the IRS, that must be an estimated payment. In reality, it was for the year before.

So, you claimed that you made this payment of a thousand dollars, but it was for a different year. And so, now, the IRS is saying, “Oh, it turns out we double check and so you actually have to pay us that thousand dollars now.”

So, that’s one of the examples that I’ve been seeing a couple of times recently of getting really clear on, okay, what was actually paid to the IRS? What year did it apply to? Commonly, those are situations where the IRS’s automated systems do tend to get it right. But I still always go back with clients and double check to say, “Hey, do we have evidence that maybe this was applied to the right year?”

Because like I said, just because it comes on a really official looking form, doesn’t necessarily make it truth. We can take a look at those and make sure that really is what should have happened.

Benjamin Brandt:         And you’ve got kind of a clever trick for making those estimated payments, and how you can separate what I send in Q1, Q2 — because I think about when I log in on the IRS website, I’m seeing a few years’ worth of payments, but they’re all very similar looking numbers.

So, how do I know if your Q1’s payment got attributed to last year’s final payment versus Q2, versus Q1? If memory serves, you got a kind of a fun trick to deal with that.

Steven Jarvis:     Yeah, quite often, tax preparation software is just going to tell us to pay the same amount every quarter. So, they’re going to say, “Hey, we think you’re going to make payments of $4,000 in total, so pay a thousand dollars each quarter.”

So, what can help for tracking purposes is for Q1 instead of paying a thousand dollars, pay $1,001 to correspond Q1. So, Q2 will be 1,002, Q3 would be 1,003 and hopefully, everyone can figure out what Q4 would be.

So, yeah, it might seem like a simple little thing. It is a simple little thing, but it will make a difference, especially if something like this comes up, that we can go back and look; did those payments get applied to the right quarter?

Benjamin Brandt:         Yeah, especially 18 months from now when we’ve sort of forgotten exactly what frame of mind, what we’re thinking about at the time, either as an advisor or as a taxpayer, it’s helpful to have some of those tricks so that even if we forget about it, that the corresponding numbers will still match up and we’ll be good.

Steven Jarvis:     Yeah, another one that I’m seeing that again, the letter will kind of start with “We think you owe us more for the following reasons …” — another one I’m seeing a lot this year is related to economic impact payments or COVID stimulus money as most of us tend to call it.

There’s a lot of confusion for taxpayers for 2021, especially for married couples because each spouse got a separate letter from the IRS showing their individual amount. And what I found was that a lot of people read those and thought, “Oh, this is the amount we received in total and so that’s what I’m going to report on my tax return.”

And so, a lot of people underreported how much they received in COVID money. And so, then when the IRS goes back through and does all their double checks, they say, “Wait a second, you told us (the IRS is saying to the taxpayer) you didn’t receive all your COVID money, but you did receive it all, and so, now, we’ve paid it to you twice. And so, now, we’re going to ask for that back.”

So, hopefully, that’ll be a little bit unique to this year. Right now, they haven’t announced any plans to do that again for 2022, but just throwing that out there as a possible reason you’re getting that letter so that it doesn’t feel … like you said before, Ben, we want to start by not hitting the panic button. And so, kind of having an idea that these things are possible helps take away a little bit of anxiety.

Unfortunately, it can mean that you are going to pay a little bit more than you thought in taxes, but at least then we understand why, and there isn’t this fear or anxiety about why is the IRS coming after me.

Benjamin Brandt:         And that would’ve been tax year 2021, right? There’s nothing that’s 2022 that’s COVID stimulus payment-related, right?

Steven Jarvis:     That’s correct.

Benjamin Brandt:         And was there something similar with the child tax credit, the prefunded child tax credit? They would send out letters to the spouse A and spouse B that was similarly confusing. Is that right?

Steven Jarvis:     Yeah, there definitely was a lot of confusion around the child tax credit, but the child, I haven’t seen as many issues with that just because the child tax credit had already existed. It got a little bit more complicated, but it was pretty typical that at tax time, you had to reconcile how much did I receive and how much should I have?

And so I haven’t seen as many issues on that. As I was talking through that, one thing that came to mind that I also want to point out is that getting that letter from the IRS does not inherently mean that you’re being audited or that they’re taking legal action against you, or that you’ve done something criminal, because I’ll also get that question from taxpayers of, “Well, does this go on my permanent record?”

And I’m not entirely sure what permanent record they mean, but just the fact that you’re getting a letter from the IRS does not mean that you have inherently done something terribly wrong that’s going to forever haunt you.

It’s more of reconciliation. It’s really, especially for the 2021 tax year, a lot of the letters going out right now are software-driven, they have these double checks in place to say, “Do all these things line up and are there any discrepancies?”

The one discrepancy that the IRS checks for that sometimes can take a bit longer is whether all of your 1099s, whether that’s for your side hustle, your self-employment income, for your IRA distributions, whatever you’re getting 1099s for; you definitely can leave that off of your tax return, but the IRS is going to notice.

Sometimes, it takes them a little bit longer and so you can get what are called IRS mismatch letters. And they’ll say, “Hey, wait a second, you didn’t report this $10,000 of income (whatever the number might be) but we saw that the XYZ company told us that they paid you that $10,000. And so, unless you can tell us why you shouldn’t have to pay taxes on it, you need to go back and pay taxes on it.”

And so, the IRS does have systems in place to come back around and make sure those things all line up.

Benjamin Brandt:         And if you get a mismatch letter, is that something that happened within the last year or so, or is this something that might happen three, four years after the fact? Well, how far back do we have to worry about something like that?

Steven Jarvis:     Typically, it’s just a couple of years. I’m sure there is a hard cutoff, but it’s certainly, it’s not as quick of a timeline as like the economic impact payment letters that I’ve been seeing. Sometimes, these will be a couple years old.

Unfortunately, I think the last I saw was that the IRS’s backlog was up to something like 21 million returns, paper returns that they’re behind on. And so, what you have is kind of this dichotomy of the IRS uses software for a lot of things they do, but they’re also hugely backlogged on the paper filings.

And so, that actually a letter that a client sent me just in the last couple of weeks was this notice saying that they were being charged $1,300 in penalties because they had a partnership that had two partners. And so, the IRS made up the penalty per partner to say, “Hey, you never filed your tax return. So, here’s your penalty for not filing.” This isn’t even taxes due, this is saying, “We’re slapping you with a fine because you didn’t file.”

But what had happened was that they filed their extension before they started working with us and they filed their extension on paper. And so, even though the IRS knows they have 21 million returns they haven’t processed, they’re still sending out these letters. They’re still sending out these notices, they’re still charging people penalties even though they know they have this huge backlog.

So, this is why it’s so important to understand and read the details in there because … so, in this example, since we don’t agree with what the IRS has said in this case, we went back and we helped the client find the extension they filed. And then we went back and pulled the copy of their return that’s been filed now too.

And we’re going to draft a letter for them that very nicely says “These were filed on time on these dates, and we feel we’re in full compliance and so, we’re not going to pay the penalty, and we’re going to send that back in,” which will of course go back into the IRS’s backlog. But unfortunately, the IRS doesn’t really give us a way to resolve these letters electronically.

It gives you some instructions; if you disagree with what’s in the letter, that there’s a phone number you can call, or you can send a letter back, but there isn’t an electronic way to easily resolve these. And so, some of these are very much a matter of keeping track of the important dates and details, and then following up and following through on making sure this gets resolved.

Because with this client, I’m just kind of assuming at this point that we’ll probably get another letter from the IRS at some point that still says, “Hey, you’re still delinquent” and we’re going to have to just kind of wait it out and see when we get through this process, because unfortunately, calling the IRS usually, puts you on hold for the whole day and you’re lucky if you get through.

Benjamin Brandt:         Well, these are consistent, the same thing if you try to call Medicare or social security from the retirement planning side — hopefully, there’s a local office, hopefully you can set an appointment three months from now, because yeah, waiting on hold is not pleasant.

But is there a way that we can maybe skip the line or try to avoid the backlog? You mentioned filing it by paper. You’re encouraging people to then e-file, is that what the current recommendation is or …?

Steven Jarvis:     Yeah, so specifically in response to that letter, we are going to have the client send in a letter. But outside of that, anytime we can, whether we’re talking about estimated payments or extensions or returns — anything I can do electronically with the IRS, I’m going to do electronically. It really increases the chances, increases how quickly things will get processed by the IRS.

And so, like I said, 21 million returns, paper returns the IRS hasn’t filed or hasn’t processed. So, anything we can do to file electronically, I strongly, strongly encourage clients to go ahead and do.

Benjamin Brandt:         How big of a room is it, do you think, that has 21 million manila envelopes in it? It’s got to be like the staple center, right?

Steven Jarvis:     Yeah, I don’t know. I’m sure some … but we we’ve got to have some engineers or architects or somebody listening that can do the math on it for us. I do tax math, I don’t do spatial math.

Benjamin Brandt:         Right. Yeah, I love it. That’s like when you were a kid, you guess how many jelly beans are in the jar? That’s kind of the same. It’s the same. I understand there’s some state-specific things that we could chat about.

Steven Jarvis:     Yeah, the IRS isn’t the only one that can send you letters. Just again, so that if you happen to get a letter in the mail that … we’re taking at least a little bit of a surprise away that this is possible; I’ll just give two specific examples, because this is going to vary by state.

We have nine states that don’t have an income tax, but they have other kinds of taxes. Then we have 41 states that have an income tax and they can all send you a letter, if you like. For our friends from Minnesota, you might have recently gotten a letter from the state that basically says “Whoopsie, we published the wrong tax form for the last two years and so, everyone did their taxes wrong and now we want to charge you for it.”

That’s a little bit of an oversimplification, but that’s the gist of it for people in the state of Minnesota who had over a certain amount of income, the state put out the wrong forms. And even though the state made the mistake, they’re still going back and asking taxpayers to go back and pay the correct amount of taxes.

Also, recently, I’ve had a couple of experiences with the city of Portland in Oregon because for those of us in states without a state income tax, we’re really only thinking about taxes at the federal level. For a lot of people, you got to think about the state level and then there’s certain cities that also have income taxes and Portland happens to be one of them for certain types of activities.

And working with a client who has rental properties in Portland, but doesn’t live there, and so wasn’t familiar with the rules and got a letter from the city saying, “Hey, here’s all the taxes you owe us.” Thankfully, there’s one we were able to work through and we’re going to be providing some additional information, and the amount originally quoted to them is not the amount that they’re actually going to have to pay.

And so, when we’re getting those letters, it can be really helpful to have someone that we can use that is a professional that does this all the time because there’s a lot of nuance to it.

Benjamin Brandt:         Absolutely. Well, Steven, could we move on to a listener question?

Steven Jarvis:     Yeah, let’s do it. The listener question that we want to talk about for this episode has to do with tax loss harvesting, which I was really excited to see come through because we did an episode, not too long ago, talking a little bit about things we can do when the market’s down, but love getting the specific questions.

And this listener kind of went through and listed some specific stocks that he has or might consider selling at a loss, and then the replacement stocks. We’re not going to go so far as to make recommendations on specific stocks here, but he kind of sums it up with this question of; okay, if I sell at a loss and then ultimately, hold my stocks for a gain in the future, doesn’t kind of the whole thing just kind of offset and I end up in the same place?

Which I think is a great question. And Ben, I’ll let you go first on talking about how you explain this to clients.

Benjamin Brandt:         Yeah, boy. So, if we could help people visualize the numbers. So, what was the company from Wall Street? Was it Anacott Steel? So, let’s say we bought Anacott Steel at $10 and now it’s worth 5. What we could do is sell it 5, and then we have a $5 loss essentially that we could use to offset something.

But if we want to own Anacott Steel for a long time, we would wait 30 days, and then we’d buy it back again. And then hopefully, we write it up to 10, 20, 30, $40, right? That’s the idea of tax loss harvesting; is I have a loss, let’s exercise the loss, we can offset gains or potentially offset a little bit of income with it.

But what the reader is I think really eloquently asking is, what did I really accomplish the? Because now, I bought at 10, sold at 5, waited the time period, bought back at five. Now, my basis was 10 and now, it’s five. So, if it’s still going to go up to $20, at some point, I was on a path to buy it 10 and sell at 20. Now ,I’m on a path to buy at five and sell at 20.

What did I really accomplish? And while we did a whole bunch of things and we did some things on our taxes and it kind of felt good, what really did we accomplish? Really nothing, because we set our basis lower. So, sometimes when I think about that, my eyes kind of cross and think about, “Did we actually accomplish anything or not?” But I think to answer the question, what did we accomplish? We have to set reset and we have to think what’s really the purpose of this investment?

Is this something we’re going to hold literally forever? If we’re going to hold Anacott Steel forever, then as current tax sales, we will get a step up in cost basis after we die. So, it does make sense to reset lower because we’re not planning on exercising higher. We’re not planning on selling higher and paying capital gains.

So, that’s kind of where I’m thinking, it depends on if this is a trade or if this is something that is going to be something permanent. And of course, we recommend permanent investing, we recommend building a Swiss army knife portfolio that you’re never going to completely sell in or out of. You’re just going to be fully invested all the time and we’re going to focus our energies elsewhere.

But there are some instances right now where essentially all bond funds are down. So, there could be an opportunity to re diversify into bond funds that are dissimilar enough, that you could get some tax benefit from that. But overall, I think the idea of tax loss harvesting gets a lot of headlines, but depending on your goals, you might not actually be accomplishing a whole lot, but what say you Steven?

Steven Jarvis:     Yeah, I think you’re hitting it real well there, Ben, that a lot of times as tax loss harvesting is discussed, it leaves out this piece of it resets your basis lower. And there could still be advantages you can use because we all know that ordinary income has higher tax rates than capital gain, capital income.

And so, we can sell at a loss and then use $3,000 of that loss each year to offset ordinary income. So, I mean, we could really get into the math of which way it comes out ahead. Some of it for me, is really just making sure that we’re being intentional and that we were making choices when choices are available to us.

Some of it really might just be that we want to take control over the timing of our income. So, some of it is understanding what our tax situation is now versus what it might be in the future. But understanding that we are resetting our basis lower, it is really important so that we’re not surprised by that later on.

Benjamin Brandt:         The listener specifically mentioned what sector of stocks they were buying, and this is in the tech sector. So, one thing that I might think about is are we looking to tax loss harvest like a mutual fund or an ETF, or is it an individual stock?

Because while we log into our custodian, we see that big red number of a loss — in order to properly harvest the losses, we’ve got to wait it out and we can’t buy back for 30 days. If it’s a mutual fund, well, the likelihood if it’s down enough to tax loss harvest is probably not going to spring back within those 30 days. We’re market timing, we don’t exactly know.

But with some of these individual stocks, while we’re waiting out that 30 days to buy back in, if it’s an individual stock, we could miss out on the run up back. So, again, really to understand how to tax loss harvest, you are doing a bit of market timing. So, if it’s an individual stock in a highly volatile market, you just got to understand the risks that you’re taking on by selling it and then saying, “I’m going to buy it back at some point in the future.”

You could get a better entry point or you could get a much worse one. We don’t know in advance, but it’s just one more thing to consider when we’re looking at this as an option.

Steven Jarvis:     Yeah, Ben, that’s a great reminder that sometimes headlines make these things seem simple, but we want to make sure we’re going in with real clear expectations of the things we should consider and what we should expect as an outcome.

Benjamin Brandt:         Right. Sometimes there’s sizzle and sometimes there’s stake, Steven.

Steven Jarvis:     That’s a great way to put it.

Benjamin Brandt:         So, that’s about all we’ve got prepared for you today. Thanks for the listener question for writing in, and until next time, don’t let the tax man hit you where the good Lord split you.

Steven Jarvis:     Hi everyone. Quick reminder before you go, while Ben and I feel very strongly about the information we’re sharing on this podcast, it is for educational purposes only and should not be taken as specific tax, investment, or legal advice.

You need to make sure that you are working with a professional to evaluate how these concepts apply to your specific situation before you take action.