MySQL Error: Job failed to start
#1
Its been around 2-3 months and everything was running like charm. But, yesterday Apache crashed and it didn't start by using service apache2 restart. I reinstalled apache and it started to work. But, when I reboot my VPS and I saw error establishing a database connection while browsing to my site. I tried to start MySQL, but, it doesn't start with Start: Job Failed error. Also, I have webmin installed.

MySQL Error Log:

http://pastebin.com/kkFG6UdV


Any help would be appreciated.
#2
I would like to see the recent portions of the apache log..around the time it went down..

From your MySQL logs, it seems like
1) you might have permissions problem on /var/lib/mysql/

do a "ls -al /var/lib/mysql" and see that the mysql user ( most probably mysql) has ownership and write permissions on it.

2) those user and plugin tables are mysql system tables. They are created during first installation by the mysql_install_db script run by the installing package and after every mysql upgrade, the upgrade scripts run a script (it is called mysql_upgrade that checks for any incompatibilities and puts things in sync). In your case either it has not been run or may be those tables are not there ( dropped / deleted ? ) for some reason.

I wonder if you tried to install mysql from source after this problem ? Did you force shutdown mysql or kill the process ? Do you have normal mount options set on that partition where /var/lib/mysql resides? ( cause setting noatime, nodirtime increases performance a lot. But if there is a sudden shutdown or process killing, then you may have problem during normal mysql health check and restore process).

I see you have webmin. I have no idea how webmin handles mysql.

You could try moving the ib tables and reinstall the whole thing and then reinsert those tables. Or may be if you have a backup, i guess that would be the best option at this point.
Many thanks to Freevps, Chris (cw1998), The Guy( ID 4810), optimus, GHP and the other  staff members.
#3
1) Running that showed mysql had no permission:
Code:
root@NodeA:~# ls -al /var/lib/mysql
total 8
drwxr-xr-x  2 root root 4096 Mar  2 06:50 .
drwxr-xr-x 59 root root 4096 Mar  2 03:36 ..

I chown mysql to that folder and now the output is:

Code:
total 28680
drwxr-xr-x  2 mysql mysql     4096 Mar  2 07:01 .
drwxr-xr-x 59 root  root      4096 Mar  2 03:36 ..
-rw-rw----  1 mysql mysql 18874368 Mar  2 06:57 ibdata1
-rw-rw----  1 mysql mysql  5242880 Mar  2 07:01 ib_logfile0
-rw-rw----  1 mysql mysql  5242880 Mar  2 06:57 ib_logfile1
But, still its not starting.

2) Yesterday, I was installing Vesta CP and the connection to the SSH was dropped due to connection error. Could this be the reason of that tables issue?
Unfortunately, I hadn't made any backup of the databases. Is it possible to backup the database now so that I can do a fresh install of the server?
I just want to get back my website online asap.
#4
Now you could try running the init.d script / service and pastebin the new log entries.

these three links will help you in recovery...

http://dba.stackexchange.com/questions/1...corruption
http://rivenlinux.info/how-to-recover-in...for-mysql/
http://serverfault.com/questions/592793/...t-start-up
Many thanks to Freevps, Chris (cw1998), The Guy( ID 4810), optimus, GHP and the other  staff members.
#5
Here's the log: http://pastebin.com/HF0hi4AT

Okay, I will let you know if the link helps.
Thanks.
#6
Hey Neon.. i think you dont have any problems in your data tables.. somehow the privilege / user tables got corrupted .. so here is what i would suggest ..
1) shut down the deamon .. "ps aux | egrep -i mysql" to check.
2) copy the sql db folder .. just do a "cp /var/lib/mysql /var/lib/mysqlbkup1" etc
3) now try to use those links to start in a level and do a mysqldump ( for good measure ).
4) now either .. >> apt-get purge mysql-server mysql-common > mysql autoremove , autoclean clean ( i just love to do .. Tounge )
next install fresh .. then import the tables..

or

you could clean that sql db folder by rm -rf * and install the basic tables and then bypass the privilege tables and set a new password for root.. it will spare you from installing ( i wll provide step by step here when i am back from doc.. 2 -3 hours )


EDIT: as i mentioned above..i would love to see the last sections of your apache log just to see how it shows the errors etc.. if possible that is. Thanks
Many thanks to Freevps, Chris (cw1998), The Guy( ID 4810), optimus, GHP and the other  staff members.
#7
MySQL doesn't start if I set innodb_force_recovery to 1/2/3/4. And, using mysqldump throws:

Code:
mysqldump: Got error: 2002: Can't connect to local MySQL server through socket /var/run/mysqld/mysqld.sock' (2) when trying to connect

Yes, you can see Apache log, http://pastebin.com/tfHAz60Y .
#8
[Tue Mar 01 11:48:02.439997 2016] [mpm_prefork:error] [pid 2909] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

this is what i found in your Apache log. I wonder how much ram do you usually see available when Apache is not running and what would be your per minute visitor count ?
Here is a link on how to increase the setting if need be..
http://serverfault.com/questions/684424/...-04-server

Also if your system supports it, try to use apache mpm_event module and not prefork one if your php app has no issue with that. It is much better performing ( also i had read somewhere that it was now default in apache 2.4 .. !! )

for "zend_mm_heap corrupted" error in apache log, see this one ..
https://serverpilot.io/community/article...upted.html

it is possible that you have copy pasted some config file from somewhere which had settings not applicable to yours. Only you will be able to know what you changed last before the apache crash.



Now to MySQL:
>> Sorry, I fell asleep and forgot about it Tounge


It said cant connect to the socket. Did you check if the mysql process was running ? and did it create the socket ??



Here are the steps to do it fresh ..but do not do it now. Please answer the above questions first..

***********
copy the /var/lib/mysql to another place and then remove everythng in it. rm -rf /var/lib/mysql/*

Run the following command from the /usr directory to install the default tables:
mysql_install_db --user=mysql --ldata=/var/lib/mysql

>> now to setup root user..
mysqld &
mysql_secure_installation  --socket=/path/to../mysqld.sock -u root -p
............OR
/usr/local/mysql/bin/mysqld_safe  --skip-grant-tables &
/usr/local/mysql/bin/mysql -u root mysql
mysql> use mysql;
mysql> UPDATE mysql.user SET Password=PASSWORD('your password here') WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> exit


if you still cant start mysql ..

Try to run MySQL in safemode:
# mysqld_safe --datadir=/var/lib/mysql
And then run:
# mysql_upgrade -u root -p
Reset the root password
Stop mysqld.service. Issue the following command:
# mysqld_safe --skip-grant-tables &
Connect to the mysql server. Issue the following command:
# mysql -u root mysql
Change root password:
mysql> use mysql;
mysql> UPDATE mysql.user SET Password=PASSWORD('your password here') WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> exit
Start mysqld.servic
Many thanks to Freevps, Chris (cw1998), The Guy( ID 4810), optimus, GHP and the other  staff members.
#9
About Apache:
Apache is working after I reinstalled it. That's a small site, so, not much traffic. Visitors/day would be around 70.
I just want to get the DBs of my site and I will do a clean install in a new server with nginx instead of Apache.

About MySQL:
Code:
root@NodeA:~# service mysql status
mysql stop/waiting
Seems MySQL is not running. I don't know about that socket thing.
Should I follow the steps?
#10
No.. not now.
you can see if there is a socket by doing a " ls -al /var/run/mysqld/" .. [[ to be sure .. you could check the path to sock file in my.cnf by " egrep -i "*.sock" /etc/my.cnf" or may be in /etc/mysql/my.cnf ]]
do you see any mysqld.sock there ? I guess not. Then most probably mysql did not start. try " ps aux | egrep -i mysql" and see if your mysql started..

If you dont see any mysqld running, try this again after "service mysql start".

if still not started..

try to run in safe mode and ask it not to look into permissions table..

mysqld_safe --skip-grant-tables &

##your mysqld_safe will most probably be in /usr/bin and you will find them in your path so direct command will work.. i compiled latest mariadb.. thats why my notes showed /usr/local/mysql/bin above.

Now check again if it has started... and reply ..
Many thanks to Freevps, Chris (cw1998), The Guy( ID 4810), optimus, GHP and the other  staff members.




Users browsing this thread: 1 Guest(s)