When all else fails

john | Feb. 17, 2023, 5:09 a.m. Uncategorized

It started as would any Thursday.  Fixing a bug or two.  Nothing out of the ordinary, until about 10:13am. 

I had decided to up the worker connections on a box, nothing drastic, to the file limit, about 1024 to see if that would solve the random bottle neck, and then it happened.  CORS.

The mother of all bugs.  CORS, or Cross-origin resource sharing, if not configured right has to be one of the most time wasting bugs to ever solve.  It isn't the first time I have run into a CORS related bug, but this one was different.  This one came out of the blue. Like would a blue whale beaching in a field.

A perfectly functioning server, one change, and then users can no longer send requests, let alone files.  I start digging.  I figure I must of some how misconfigured something, or run a command while searching for another one.  Possibly changing permissions.

But no, nothing.  I check the error logs, nginx shows 404's for every request.  It never even reaches the Django server.  I keep digging again, trying to figure out what was going on.  I try to see the tcp dump.  I ask on stackoverflow.  I even asked chatgpt.  And then Finally, after eight ours I figured it out.  Simply put, I'm an idiot. 

By hour eight, I did not realize I had spent so much time trying to solve this, that it finally occurred to me to just launch another server with my ansible script and see if it had any issues.

I mean it's not like it was a unbackedup database or something, no, it was just a simple long function workhorse server. 

Surprisingly the new server did have issues.  So the next day I spent some time trying to figure out what it is, ended up figuring out it was one of two things.  My app did not launch under supervisor for some reason, or there was an issue with django-header-cors.  So I rolled my own, and it started working, but only when doing everything but the load balancer.

And then, finally... it occured to me.  Days later, it's because since I have a SSL, its port 443, so my load balancer needed to send to ip:443, not just ip, since that sends to port 80. 🤦

Follow me on BlueSky @nader.mx

---