Taxes and Social Security

Benjamin Brandt:         Welcome back to The Retirement Tax Podcast. I’m your humble host as always Benjamin Brandt. I’m your humble co-host as always. Steven Jarvis joins us. Steven, how are you doing?

Steven Jarvis:     I’m doing really well. How are you doing today, Ben?

Benjamin Brandt:         I’m doing great. I’m excited to talk about one of the three corners of the stool that every retiree is going to have to make decisions about, and that is Social Security.

Steven Jarvis:     Yeah, love talking about Social Security because contrary to popular belief that it’s going to run out, and no one’s going to get it starting tomorrow, taxes, this is something that ends up impacting most everybody, and so, you get to cast a pretty wide net with who you gets to benefit from this information.

Benjamin Brandt:         Yeah. You think about the wide spectrum of people in this country, 330 million of us approximately, from Jeff Bezos down to the poorest of poor person that’s on Social Security Disability. Social Security affects everybody. Even when you turn 70, the richest guy in the world, he’s still going to get a Social Security benefit. It’s going to it to pay out at some point. So it connects all of us in one way or another.

Steven Jarvis:     Yeah. Yeah. I’m really excited to talk about and share some information around Social Security today. Because again, similar to taxes, this is an area where a lot of people feel like they don’t have a lot of choice or influence on what goes on with their Social Security. That it’s just, “Hey, this is going to happen someday. We’ll just wait and deal with it when it comes.” Some people are surprised to find out that Social Security is in fact taxable. We’ll talk a little bit more about that. And then, I want to make sure that people know where they can take action and where they can make choices around Social Security.

Benjamin Brandt:         Right. Where would be a good place to start? Maybe you can just answer the question for me, Steven, will I pay income taxes on my Social Security?

Steven Jarvis:     Yes. Most definitely, Benjamin. Social Security has this new level of how things get taxed. It comes up quite often of, “Well, what income tax rate am I going to pay?” But for Social Security, we have to take a step back and say, “How much of it is taxable?” And depending on your filing status and what the IRS refers to as combined income, because of course, we need another new definition, somewhere between zero and 85% of your Social Security benefit is going to be taxable to you. And whatever that percentage ends up being, it’s going to be taxed at the same tax rates or using the same tax brackets that your wages, your ordinary income was taxed at, before you started claiming Social Security.

Benjamin Brandt:         Okay. There’s two things that we have to calculate. I’m getting $10,000 a year in Social Security. How much of that am I going to realize as taxable income? Then that number, if it’s 85%, so $8,500, then $8,500 shows up on my income tax. We have to figure out how much applies to me. And then, we roll that up with every other piece of income, capital gains, whatever else I have. Am I understanding that right?

Steven Jarvis:     Yep. Interest, dividends, all those kinds of things, they can get combined to calculate your tax. But that starting point is, “Well, how much of it is taxable?” And there definitely are retirees where this isn’t much of a consideration because you get to that 85% level pretty quickly if you’ve intentionally prepared for retirement. Because for 2022, it actually didn’t change compared to 2021. But for a married filing jointly couple, once you’re over $44,000 of combined income, it’s 85% taxable. And so, for a lot of people who’ve prepared for retirement, it’s going to be 85%.

Benjamin Brandt:         Well, I think it’s been 44,000. I’m trying to think of when I first started teaching the Social Security class at our local community college. I think it’s been 44,000 for a really long time. In fact, I remember, and our listeners can Google this to double-check me, and then email me that I’m wrong. But I think when they first set these rules out in the ’80s, it was still 44,000. I think that was a princely sum back then. And they just never moved the mark. So now, if you’re listening to a podcast about retirement taxes, there’s a really good chance you’re well prepared for retirement, and the levels will just never apply to you. Your income’s going to be over 44,000. Your number’s going to be 85%. Let’s just dial in the withholding properly on Social Security, and you probably don’t have to think about it again.

Steven Jarvis:     Yeah. This springs up just kind of, I guess, interesting tidbit to me about different tax brackets and tax rates. The tax code is just this giant mystery to most of us. But part of the reason for that is that the tax code is written by Congress, not by accountants. And no one sat down one day and said, “Okay, let’s write the whole tax code so that we can have consistent themes for it.”

                           These things get established at different times. And so a lot of aspects of the tax code are indexed for inflation. That’s why the tax brackets for ordinary income rates change every year just a little bit. They did it again in 2022. They shifted just a little bit. But the Social Security percentage that is taxable, when they created the rule for it, it was not indexed for inflation. So to your point, it’s just, it is what it is.

Benjamin Brandt:         Right. Is it an intentional political football like the minimum wage where they’ll never index it to inflation, just so that they can promise to fight about it every five years?

Steven Jarvis:     I wasn’t really paying attention to this stuff back in the ’80s when it was created. So, I couldn’t speak to the political football on that one. All of these things…

Benjamin Brandt:         Were you alive in the ’80s?

Steven Jarvis:     I was alive in the ’80s. I just wasn’t paying attention to tax law.

Benjamin Brandt:         Okay. I was alive for almost all the ’80s. I remember.

Steven Jarvis:     Then you mentioned just almost in passing about, “Okay, great. For a lot of people listening, they’re probably in the 85% bucket, and let’s just dial in the withholdings.” But I want to key in on that because, in my experience, most taxpayers aren’t aware that they can withhold taxes directly from Social Security. That comes as a surprise to them. I’d be curious with your experience with your clients as they get to Social Security age.

Benjamin Brandt:         Yep. Usually, it’s something that… And the numbers don’t fully make sense when we do tax reviews, we’ll talk about, “You’re in the 22% bracket. You’re in the 24% bracket.” But that might not be an option on Social Security Statement. I don’t remember what the four options are, or however many there are. But the numbers, they just don’t really seem numbers that we normally talk about. I think in many cases unless there’s specific guidance given from an advisor, people either pick nothing or they pick that seems appropriate to them.

                           Which I guess can be fine, but then we’re either going to be over withholding or under-withholding on everything else, like our retirement, or our IRA distributions, things like that. So, when we talk marginal rates and we talk effective or average rates, if we don’t have it all dialed into the same radio frequency, it’s not really going to make sense at the end of the year. It’s important. It’s probably like a set it, forget it situation where we do this math once, and then don’t ever deal with it again. But I think it is worth that extra step to go find that form, fill it out, and file it.

Steven Jarvis:     Definitely. And it’s Form W-4V for anyone who is about to claim Social Security, or who’s already claiming Social Security that wants to go and start a withholding. And what’s interesting, really what’s frustrating is that, when you apply for Social Security through the Social Security Administration, there is nowhere in the application process that points you towards having a withholding.

                           If you’re claiming Social Security and you’re thinking, “Ah, how did I miss that?” Don’t feel bad. No one probably ever told you. It certainly wasn’t in the application process. But yeah, Form W-4V. And you’re right, Ben, it is a little goofy. Unlike electing withholdings from your earnings from a job where you can set a percentage or you can pick a dollar amount. You can essentially do whatever you want.

                           For Social Security, there are four percentages to choose from. I always remember the lowest is 7% and the highest is 22%. But you have these four options that maybe at some point they aligned with something, I don’t know. They certainly don’t really match up now to anything that makes sense to me. But from a planning perspective, especially a cash flow planning perspective, I think it’s a great idea for people to go out, and have some kind of withholding done.

                           I usually will encourage clients to air on the high side of what they withhold from their Social Security, just because it gives us more flexibility in other withholdings throughout the year. So that if we have IRA distributions as well, or maybe we still have other sources of income, we have more flexibility in those other areas to adjust throughout the year, instead of having to go back and refile the W-4V and adjust that every year, we can get to our IRA distribution, look at where we’re at for the rest of the year and say, “Okay, how much do we need to withhold?”

Benjamin Brandt:         I like that. It gives us a little bit of a withholding cushion so that the IRS is getting money every single month. And if there’s something big at the end of the year that we didn’t anticipate or something that we don’t want to take out extra withholding, we’ve got a little bit of a cushion of withholding that much less likely we’re going to pay taxes or pay interest penalties, are those things we want to avoid if possible.

Steven Jarvis:     Yeah. And the other comments I’ll make on this is that in my experience, quite often, tax preparers will, for whatever reason, they’ll err on the side of recommending estimated payments, I’ve never entirely understood this. Maybe it’s a disconnect on that side too of, “Hey, this form’s out there, and we can’t have withholdings.” But for clients or for taxpayers who aren’t already making estimated payments, I’m going to try to avoid estimated payments for as long as possible. That’s just one more thing that can go wrong, that can get forgotten, that can get lost. And so, at least my personal preference opinion is that for Social Security, you should definitely be defaulting to let’s have a withholding. Let’s not think of an extra reason to have estimated payments.

Benjamin Brandt:         I do have a couple of clients. One is a former account who insists on making estimated payments. But my philosophy, and I probably, unintentionally, push this on clients is, “You’re retired. I want you in your RV, in Baja, California. I want you on an airplane in Europe. I don’t want you at home thinking about your estimated payments or on a vacation remembering in a panic that you forgot yours. Let’s just get our withholds right. And that’s just one line thing to forget about or worry about.”

                           And there’s been quite a few clients over the years where I’ve shown them like, “Hey, we can be done doing estimates now. We’re just going to withhold.” And you’d think that I blessed them with some sort of miracle of tax planning just because they’ve been doing it for so long. But yeah, let’s make it a goal to get rid of those estimated payments if we can.

Steven Jarvis:     Yeah. And like I said, this does come as a surprise to a lot of people. I was actually working with an advisor recently. I worked with a lot of advisors on tax planning. And when this came up, it was news to him. And then he went and worked with some of his clients on it. And he was telling me just how appreciative, just like you’re sharing, how appreciative they were of just being able to hit the easy button on that. It’s just one less thing to worry about. And like you said, we should be enjoying retirement. We should be enjoying life. Let’s not intentionally leave ourselves more things we have to remember when it comes to end IRS.

Benjamin Brandt:         Exactly. Exactly. What was the name of the form again? And then, where can we go to find it, and submit it?

Steven Jarvis:     Yeah. It’s Form W-4V and you can find that on the IRS’s website. We’ll put a link to it in the show notes if you go to retirementtaxpodcast.com. And so, that’s the form that you need to be able to file or adjust you’re withholdings if for some reason you’d like to change what they are. But it’s definitely something I recommend that everyone take a look at if you’re not having taxes withheld from Social Security.

Benjamin Brandt:         And one thing we talked about while the mics were warming up is kind of an interesting… With your background and my background, to talk about where does that money come from. You go and you claim your Social Security benefits and the money shows up every month, hopefully. Where does that money come from, and should I be concerned that it’s going to run out? Should I claim my benefit as quickly as I possibly can because there is a bucket of money somewhere, and every month we all take out, and it’s going to run out someday? Can we talk a little bit about where that money comes from, and how long it might last?

Steven Jarvis:     Yeah. I mean, I’d like to start by saying that as long as I can remember, I’ve been told, “Hey, you can’t plan on Social Security. It’s going around. It’s going to be gone.” And the first times I remember hearing that it was, “Oh, it’ll be gone in five years. It’ll be gone in 10 years.” And those five and 10-year marks have passed multiple times now.

                           It makes great headlines. Well, great headlines from the clickbait and eyecatching standpoint. So, you see it all the time, “Social Security’s gone. It’s running out. It’s going to disappear.” But yeah, this is actually… All of us who are working still are continually paying into this, so it’s not just a bucket of money that was set up, and we got to hope it doesn’t run out. This is what some of your payroll taxes go to cover.

                           And I’m not an actuary. I’m not an economist. I know that there are concerns and some of them are probably valid. But every time there’s been a concern about when Social Security is going to run out, it gets adjusted. There’s some kind of pivot. I personally don’t take the doomsday approach if it’s gone tomorrow, so I better just ignore it or pretend it’s not there.

Benjamin Brandt:         Right. And depending on the strength of the economy, depending on unemployment, I mean, there’s different amounts that come in. But yeah, it’s a dedicated payroll tax. So you pay a bit, your employer pays a bit. If you’re self-employed, you pay both sides. But there’s new money coming into the system all the time. So really it would be impossible for it to “run out to zero” because I’ve certainly done my part by having six kids. As long as someone is working somewhere, no matter how high unemployment is, there’s going to be new money coming into the system every first and 15th, every other Friday, whatever it is. There’s new money coming in all the time. So right now, the Social Security Administration has something like 70 something cents on the dollar, 65 cents on the dollar where they know how many people are in the country, how many people are working.

                           And so by 2035 or whatever it is, they think that they’ve got that 65, 70 cents a dollar. As the economy improves, that number always goes up, 2035 becomes 2041, whatever it is. Economy gets worse. And you see the headlines of, “It’s 2029,” whatever it is. So there’s no way it can run out. It’s a dedicated payroll tax. There is just a possibility that if they don’t raise payroll taxes or do something, reduce benefits, recalculate how they calculate cost of living, have phase-outs. There’s a phase-out right now at 66 to 67 every other month. There could be a new set of phase-out. There’s a dozen ways that they could fix it, but it is fixable.

                           And I guess the moral of the story of what I’m trying to say is, “Make your Social Security claiming decision based on what’s right for you and your family, based upon your unique circumstances. Don’t simply claim early because you want to get all the gettings good. And you want to get yours before the money runs out. It’s not going to run out. Worst case scenario, it gets dinged slightly. So you don’t want to intentionally reduce your payment for life in anticipation of a reduction, because then you just got cut twice. Make a decision based on your unique circumstances and have confidence that the powers that be will get this figured out. They have thus far, just barely right on time, 12th-hour decision. Probably was going to be this time around, but make a decision on what’s right for you. Focus on what you can control.”

Steven Jarvis:     Yeah. And really, that’s great advice for any area of financial planning tax or otherwise. There’s always a huge list of unknowns. When we’re talking about investing. When we’re talking about taxes, there’s a lot we can’t control and we don’t know, but we got to do the best we can with the information we have. We got to keep moving forward. If we just allow ourselves to not make decisions because there’s an unknown piece, well that’s still taking action or a lack of action. And that’s not how we’re going to set ourselves up for success.

Benjamin Brandt:         Excellent point. And moving on to our listener question. We’ve got a listener question from anonymous, and they write, “What is the best way to avoid taxes in a 401(k)? I’m 57. Do I start making withdrawals and spend down the portfolio or hold off on Social Security until 70?” What are your thoughts, Steven?

Steven Jarvis:     I love getting our listener questions and being able to hear how people are thinking about this. Because there’s so many different things to consider. It gives us a chance to really wade through this, and because the way I read this, it makes it seem like, “Here’s the two alternatives.” But there’s other things to consider. The CPA in me wants to start with, I would reword this to what’s the best way I can lower the taxes on my 401(k)? Again, as a CPA, that avoidance, that seems like we’re hoping the IRS doesn’t notice. There are definitely ways we can intentionally stand off the rough edges. And a lot of it has to do with the timing of when we’re taking these distributions when we’re paying that income tax. Because even at 57, there are things that we can start doing to go ahead and start paying some of that tax now.

                           At 57, we probably wouldn’t want to just directly take the distribution. We’d be paying some penalties and it kind of defeat the point of why we put it in there. But we can start looking at things like Roth conversions to say, “Are we at a place right now where we think our tax rate might be higher in the future? Or even if it’s not going to be higher, that we want to pay the taxes now so that we can take away the IRS’s ability to change the game on us later.” And so, there’s certainly some planning things that we can do. A piece of that could be, delaying when we take Social Security so that we have a few more years where our total income’s a little bit lower, and therefore the tax rate we’re paying is a little bit lower. But those aren’t necessarily mutually exclusive decisions.

Benjamin Brandt:         Yeah, I would wholeheartedly agree. With Roth conversions and things like that, you’re locking in or hopefully locking in the rules that you play by. So I know exactly what taxes I’m paying, and I’ve locked that in, hopefully for life. When I think about 57 and I don’t have any additional information, so I’m going to make up my own. When I think of a 57, I think about someone that’s just right on at the end of their career, their golden years of their career. And what’s often the case with my clients is, they’re in their peak earning years. And they’ve maybe done some retirement planning on their own, maybe five years or 10 years before retirement, but they’ve continued to get raises beyond that initial planning. So maybe their income is 25% higher than it would need to be their first year of retirement because they’ve laid out those plans, but have continued to be rewarded at their work, thus they’re in their peak earning years.

                           Often, when someone is in their last few years of work, we want to be in full deduction mode. We want to find as many deductions as we possibly can. We want to fund future charitable contributions. We want to fund future retirement plans. We want to fund future healthcare plans, whatever it is, squirrel away as much as that money, and get as many tax deductions as we can, because we want to blunt those highest income years, and of course, the affiliated taxes. Then when we’re retired, let’s say, in this case, we’re retired at 62. We can start to undo some of those deductions. We can start to give out a Donor-Advised Fund. We can start to spend out of our HSA. We can start to do Roth conversions, and undo the deduction we took two or three years ago at a significantly lower effective tax rate.

                           To answer this question, I would say, “Look at where your income is now over the next, last year, this year, next year, through your window. Where is it going to be in three years? If it’s going to be quite a bit lower, let’s take as many deductions as we can now, and simply undo them when our income drops.” So I would say, to answer the question, if you should start taking withdrawals and spend down the portfolio, you’ve got to know where you’re going, and what your income levels are, and that’ll help you make that decision.

Steven Jarvis:     Yeah, if we just stop the question at what is the best way to avoid taxes in a 401(k). The answer’s going to start with taking this multi-year approach that you’re describing. You’ve got to understand your relative situation where you’re at now versus when you’re going to retire. And then, not just when Social Security might start, but also when required minimum distributions might start. So those are some key timing things to look at of where’s my income going to be at those different times.

Benjamin Brandt:         Steven, brilliant as always.

Steven Jarvis:     Thank you. One last thing I want to say on Social Security before we wrap up is that, regardless of how close or far away from claiming Social Security you are, you can go ahead and create your account with the Social Security Administration, and as unfortunate as this is, the reason I recommend this to everyone is that it’s a great fraud prevention step, because if you don’t create your account, then, unfortunately, someone else might try to and might try to change the address that those checks are going to, things like that. So if you’re over the age of 18, and working or a citizen or going to be eligible for Social Security someday, go ahead and go create your account. It only takes a few minutes, and it makes sure that no one else can create it later. And that would be my other recommendation that everyone can take if you don’t already have that account set up.

Benjamin Brandt:         I love it. Do that, and do that today. We’re going to talk to you again in two weeks. Make sure that’s done. So, until next time. Don’t let the taxman hit you where the good Lord split you.