So I was working on our company’s server the other day. I was running a couple of updates - notably to CPAN. Like an idiot, I assumed that updating CPAN wouldn’t affect anything critical. Boy, was I wrong.
After the update, I went to check on something on the Web site and got the message that Drupal couldn’t connect to the database. I figured maybe the CPAN update stopped mysqld, and I just had to restart it. So I restarted it. No luck. So I dug deeper and found that the root user could no longer do a whole lot of work with the databases. I fired up phpmyadmin only to see that root had no privileges. I couldn’t create, delete, or edit any of the databases. This was looking grim. I had this problem once before and never did figure it out.
So I immediately hit Google trying to find an answer.
Two days of Google searches, and I had nothing. So I send out an e-mail to a few lists. Nothing. No one knew what was up. The closest thing I could find was that it could be a MySQL bug.
A bug? Really? A CPAN update causing a critical bug?
Well, after much digging I decided to try to start the mysql daemon with the:
“mysqld_safe –skip-grant-tables –user=root &” command.
I was at least able to get further with phpmyadmin. I still had no privileges, but I could access the tables on the database. Well, it turned out the CPAN update changed some permissions on the USER table of the mysql main database which removed root’s ability to actually do anything.
Hmmmm - I wonder if those that develop for CPAN know of this? I made a mental note to make sure to contact the developers to let them know.
But ultimately, what this did was to remind me that (open source or no) one should never make changes on a production-level machine without first testing those changes on a non-production machine. It was looking as if I would have to go as far as rebuilding the entire server simply because I no longer had access to the databases. And not being a mysql guru - I was a fish out of water. I was very fortunate for one thing: when something is wrong you can always dig a little deeper if you’re using open source software.
Have you had a near-catastrophic incident involving open source software?
This incident also taught me one other valuable lesson: When searching for help within the open source community it’s always best to approach it with more information than you think you need to know…otherwise you’ll be bombarded with RTFM from all angles.
Happy holidays everyone!