Scaling issues

john | May 20, 2023, 5:49 p.m. Programming

Any project, no matter how large or small, will at some point probably run into scaling issues.  That is of course, assuming it is growing.

Now it might not be a sort of viral thing that goes through the roof, but instead a website that you potentially made a few years ago and slowly but surely has been growing.  And when it grows its awesome.  Especially when the errors that start occurring because it grows, are not reproducible.

Earlier when I added in a load balancer to this project.  It broke, granted that was because of the fact that I didn't realize the port was wrong.  But now, the issue is odd.  Out of no where, the queue just stops. 

It decides it no longer wants to process more files.  It happened yesterday, and at first, I checked since it is on two different boxes.  There was one extra thing on the box, and I took it down, thinking that was what caused it. 

It wasn't.  Today I wake up, and now both boxes have a queue in the hundreds.  So I decided to update everything, and reboot as Ubuntu was asking.  I checked the Redis logs and found that there was one initialization error that I never took care of.

I fixed that, it was on both boxes.  So if that was the error, it would be odd that it fails, with no error log, but it could be.  So, now instead of getting notified once per day if the queue is growing or not, I set it for an hourly reminder, so just maybe it was that Redis error.

Cause if not, its going to be another scaling issue that takes longer than ever to debug.  I hate when that happens, bad user experiences always suck.  Especially when you're trying to fix it and can't reproduce it.  All I can do is wait till it grows again or happens again.  And see if any monitoring software catches it this time.

If not, maybe I did fix it after all.  Only time will tell.

Follow me on BlueSky @nader.mx