@treyfranciscoh a few questions. Is it a Fitbit original clockface i.e. made by Fitbit or 3rd party? Do you have any issues syncing automatically? Fitbit needs regular sync to keep the correct time. It looks like your devices is not syncing regularly. What's your phone model and OS? if you're Android, make sure Fitbit app is not being optimized in battery settings of Android. Ie. the process can run in the background.
My Sense is experiencing the same issue.
https://community.fitbit.com/t5/Sense/Clock-is-constantly-a-few-minutes-behind/m-p/4560587#M5949
I talked to Fitbit support, they asked me to do a few things (restart the device, set it up again, etc.) but nothing helped. I wonder if it's an issue with all Sense watches or just a defect affecting a couple of units.