This Sunday at 2 AM, I got a call from one client, who complained that cgi scripts on his website stopped working all of a sudden. This being peak season for his business, he was a worried lot and wanted me to give it an immediate attention. I rushed to my workstation. Client was on phone telling me the urls and script names which stopped working. I saw it was throwing “500 Internal Server Error”. My first question to client was if he has made any changes to the scripts or re-uploaded or changed the permissions on the files. He denied doing anything. On the other hand he said that might be something has changed on server. I wasn’t sure as neither data centre housing my servers nor the server management guy, managing my servers, had informed me of any changes. I told client to wait for sometime as I look into it.
My next stop was error log. It was of little help as it showed:
[Sun Dec 02 01:58:02 2007] [error] [client xx.xx.xx.xxx] suexec policy violation: see suexec log for more details [Sun Dec 02 01:58:02 2007] [error] [client xxx.xx.xx.xxx] Premature end of script headers: FrmsMail2.cgi
However, it made one thing sure that some updation / patching of APACHE / Kernel or firewall has taken place on server which broke these cgi scripts. I shoot a mail to my Data Centre as well as the Server Management person and I started looking at suexec.log. Within minutes I had reply from both place. Data centre team said that they will take a look at the issue and the server management guy said that he is already looking at the problem as another client had directly contacted him. But, I needed some solution immediately as the client who called me first hand was calling again and again.
suexec log had me:
target uid/gid (uid-1/gid-1) mismatch with directory (uid-2/gid-2) or program (uid-3/gid-3)
I googled it and came to a decision that it is related to some ownership / permission issues of folders and files. Now I started taking a stock of all files and folders involved. The page calling the offending script was owned by user ‘xxxxxx’ under group ‘psacln’. CGI-BIN folder was owned by same user but under group ‘psaserv’, while the cgi script itself was owned by user ‘xxxxxx’ under group ‘psacln’ (this server uses PLESK hence psaserv and psacln). The cgi file was in root of cgi-bin folder. I wasn’t sure if this could be the problem, however I tried to adopt some error and trial method. Just created another folder under cgi-bin folder, made a copy of cgi script under the new folder and called the script from my browser.
WOW!!!!!!!!! it was working perfect. I looked at the temp folders’ ownership. It was owned by user ‘xxxxxx’ under group ‘psacln’. So, now the page calling the script, the folder housing the script and the script itself was owned by same user under same group and it was working fine without changing a bit of code. I quickly changed the reference of cgi-bin file in the calling page and informed the client about success.
Didn’t forget to thank and inform the data centre team and my server management person and it was just 2.45 am so still I could get some sleep.