Support Forum

Advanced Search
Forum Scope


Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
general-topic
SQL Processes Cripple Server After SP Update
Avatar
jim
Here and Now
Member
Pro Subscribers
sp_UserOfflineSmall Offline
Nov 1, 2013 - 5:44 pm

Well. I'm long overdue for an urgent request for critical support so here goes...

Yesterday, I updated to SP to v.534 build 10781 after locking our forums as usual, then proceeded to upgrade WordPress to v 3.71. I immediately started experiencing performance issues on our server and spent the next 6+ hours troubleshooting high loads.

I pinpointed SimplePress as the cause, for whatever reason. With the plugin deactivated all is well with the site and server performance. Upon activation, I can watch server loads immediately start to rise and spin our of control with MySQL processes eating up resources. Deactivate the plugin and loads immediately drop and stabilize.

I have repaired our database and run additional repair and optimize commands on all wp_1_sf* database tables. I uploaded a fresh download of the plugin, and the issue persists—activate plugin, server load and MySQL spin out of control, deactivate and the box quickly stabilizes. 

During troubleshooting, the techs at ServerBeach mentioned the "wp_1_sfposts table is not indexed causing slow queries". Is there a way to index this table via PHP MyAdmin or the shell CLI. I notice there is the rebuild indexes function in the SP Admin, but activating the plugin will cripple our site. The posts table is also 129± MB, so I fear an attempt to do this from the Admin may choke.

I have deactivated all SimplePress plugins (Admin Bar, PM, etc) to rule those out and problem persists upon SP activation.

Any other steps I might take to remedy this situation. Our forums are now offline until I get this resolved. What other information might you need from me to help?

As usual, thanks in advance for any guidance!

TripawdsSimple:Press powers the Tripawds Discussion Forums.

It's better to hop on three legs than to limp on four.

The Tripawds Blogs Community is made possible by The Tripawds Foundation.

Avatar
jim
Here and Now
Member
Pro Subscribers
sp_UserOfflineSmall Offline
Nov 1, 2013 - 5:48 pm

PS: I was able to grab these from the SP Admin Error Log... sorry for the post length and poor formatting, but I thought something here may help.

Date    #    Type    Description 31 October 2013
4:52 pm    2    php    file: s/simple-press/forum/content/sp-search-view-functions.php
line: 61
function: sp_SearchHeaderName
Notice | Undefined variable: label

31 October 2013 3:12 pm    1    php    file: s/simple-press/sp-api/sp-api-timezone.php
line: 53
function: sp_apply_timezone
Notice | Undefined index: timezone

31 October 2013 2:36 pm    9    php    file: ripawds/public_html/wp-content/sp-resources/forum-plugins/admin-bar/library/sp-admin-bar-components.php
line: 564
function: sp_AdminBarDashboardPosts
Notice | Undefined property: stdClass::$topic_id

31 October 2013 2:36 pm    9    php    file: ripawds/public_html/wp-content/sp-resources/forum-plugins/admin-bar/library/sp-admin-bar-components.php
line: 562
function: sp_AdminBarDashboardPosts
Notice | Undefined property: stdClass::$post_count

31 October 2013 2:36 pm    10    php    file: ripawds/public_html/wp-content/sp-resources/forum-plugins/admin-bar/library/sp-admin-bar-components.php
line: 559
function: sp_AdminBarDashboardPosts
Notice | Undefined property: stdClass::$post_count

31 October 2013 5:17 pm    25    php    file: s/simple-press/forum/content/sp-common-view-functions.php
line: 1549
function: sp_BreadCrumbs
Notice | Undefined index: forumname

31 October 2013 5:18 pm    127    php    file: s/simple-press/sp-startup/forum/sp-forum-support-functions.php
line: 635
function: sp_setup_title
Notice | Undefined index: forumname

1 November 2013 2:00 pm    4    php    file: s/simple-press/forum/content/sp-common-view-functions.php
line: 1555
function: sp_BreadCrumbs
Notice | Undefined index: topicname

31 October 2013 4:18 pm    9    php    file: s/simple-press/sp-api/sp-api-filters.php
line: 1414
function: sp_format_display_image
Notice | Undefined offset: 1

1 November 2013 2:00 pm    45    php    file: s/simple-press/sp-startup/forum/sp-forum-support-functions.php
line: 639
function: sp_setup_title
Notice | Undefined index: topicname

31 October 2013 4:16 pm    13    php    file: s/simple-press/forum/content/classes/sp-search-view-class.php
line: 255
function: sp_build_search_url
Notice | Undefined index: forum

31 October 2013 3:25 pm    8    php    file: s/simple-press/sp-startup/forum/sp-forum-support-functions.php
line: 85
function: sp_populate_query_vars
Notice | Undefined index: forum

31 October 2013 4:55 pm    22    php    file: s/simple-press/forum/content/classes/sp-search-view-class.php
line: 81
function: sp_searchview_control
Notice | Undefined index: forum_id

31 October 2013 11:54 am    4    php    file: s/simple-press/admin/panel-options/support/spa-options-save.php
line: 37
function: spa_save_global_data
Notice | Undefined index: blockredirect

31 October 2013 11:54 am    6    php    file: s/simple-press/admin/panel-options/support/spa-options-prepare.php
line: 30
function: spa_get_global_data
Notice | Undefined index: blockroles

1 November 2013 2:00 pm    127    php    file: ripawds/public_html/wp-content/sp-resources/forum-plugins/private-messaging/library/sp-pm-components.php
line: 218
function: sp_pm_do_remove_pms
Notice | Undefined index: sfpmremove

TripawdsSimple:Press powers the Tripawds Discussion Forums.

It's better to hop on three legs than to limp on four.

The Tripawds Blogs Community is made possible by The Tripawds Foundation.

Avatar
Mr Papa
Simi Valley, CA
SP Master
Free Members
sp_UserOfflineSmall Offline
Nov 1, 2013 - 6:16 pm

you can ignore those notices - they are harmless and we fix them as we come across them... starting with 5.3.3 (or 5.3.4) you will see more of them since we found we had actually left some test code in place that was omitting the undefined index notices...  so they were re-enabled which just means sifting through them....  many have already been fixed in 5.3.5...

I can think of no reason why your sfposts table would not have any indexes...  every site when installed creates the tables and the needed indexes... so unless your mysql permissions do not allow adding indexes, unknown how it could happen..  just looked at my couple of multisite test sites and they all have proper indexes on the tables...

yes, you can add them via phpmyadmin...  size should be a matter... I have added them on a site where the table size was over 2 GB...

here is a screen shot of what the standard indexes look like on the sfposts table...  I have not compared those to what the install script does, but should be quite close...

indexes.pngImage Enlarger

 

Avatar
jim
Here and Now
Member
Pro Subscribers
sp_UserOfflineSmall Offline
Nov 2, 2013 - 8:35 pm

Mr Papa said
here is a screen shot of what the standard indexes look like on the sfposts table... 

Interesting. We do have indexes on our wp_1_sfposts tables as shown in the first screenshot below, very similar to the defaults you provide. I do, however, have an additional table and get the following notice:

The indexes user_id_idx and user_idx seem to be equal and one of them could possibly be removed.

Before I do more harm than good, my questions at this point include:

  • Could the "equal" user_id_idx  index be causing problems and could I safely drop it? I never added that so i have no idea how it was created.
  • Is there a way to rebuild or repair these indexes? I see no commands in PHP MyAdmin other than Edit and Drop.
  • Admittedly, I don't understand what "Cardinality" means in this view but it appears the other numbers refer to forums, topics, comments, etc. So should post_content really be 1 when there are 110,317 posts?
  • Anything else look odd in the attached Indexes and Table Info screens? Is 53+ MB normal for the index on a table this size?

Any other suggestions as to what might be causing MySQL resources to climb immediately upon SimplePress activation? Thanks again for any input on this!

wp_1_sfposts-indexes.pngImage Enlargerwp_1_sfposts-info.pngImage Enlarger

TripawdsSimple:Press powers the Tripawds Discussion Forums.

It's better to hop on three legs than to limp on four.

The Tripawds Blogs Community is made possible by The Tripawds Foundation.

Avatar
Yellow Swordfish
Glinton, England
SP Master
sp_UserOfflineSmall Offline
Nov 3, 2013 - 3:05 am

I am going to have to pass on why you have two indices on user_id but yes - I would remove the one named user_id_idx. That will be fine.

Do you regularly optimise your tables? This action will normally correct any indexing issues as well as ensure the table data structures are correct etc. Normally, at the bottom of the table listing in phpmyadmin there is a drop down list of options and from there you can 'check', 'repair' and 'optimise' tables. I would suggest you perform a 'check' operation n all of them to see if anything gets reported as a problem. And then repair and optimise as necessary. This should also rebuild the post_content full text index as well - which should not be empty.

andy-signature.png
YELLOW
SWORDFISH
Avatar
jim
Here and Now
Member
Pro Subscribers
sp_UserOfflineSmall Offline
Nov 3, 2013 - 2:14 pm

Yellow Swordfish said
...I would remove the one named user_id_idx.

Thanks, done.

Do you regularly optimise your tables?

Yes, about every week or so I do regular checks and optmize db tables.

I just ran a check on all sf tables with OK results all around. After deleting the duplicate index, I then ran a repair and optimize on each sf table. Again, Status returned was OK for all tables.

So... I activated SimplePress and the server load quickly climbed to 5.5±, but stabilized long enough for me to unlock the forums and run the Rebuild Indexes command on all forums from the SP Admin. Then, within a couple minutes the load climbed to 20± before I deactivated the plugin. Load then immediately dropped to <0.5.

During this short time, we did receive about 10 guest spam posts. To rule out some sort of forum based spam attack, I waited until loads settled down and reactivated SimplePress. I immediately locked the forums, but continued to watch loads climb to 15+ within seconds. Loads again immediately dropped upon deactivation of the plugin.

FYI: I have ruled out any plugin conflicts by deactivating all other but SimplePress, and same problem persists.

This should also rebuild the post_content full text index as well - which should not be empty.

I realize this should not be empty but I wonder if a Cardinality value of 1 is appropriate for forums our size or if this may be an indication of something wrong.

What steps can I take from here to get our forums back online and stable?

Is there a recommended collation for _sf tables? i may be reaching for straws here, but I'm at my wit's end and Tripawds members are getting restless.confused

TripawdsSimple:Press powers the Tripawds Discussion Forums.

It's better to hop on three legs than to limp on four.

The Tripawds Blogs Community is made possible by The Tripawds Foundation.

Avatar
Yellow Swordfish
Glinton, England
SP Master
sp_UserOfflineSmall Offline
Nov 3, 2013 - 2:40 pm

Jim - could you check your php error log? Not the SP one - the actual, full php log?

andy-signature.png
YELLOW
SWORDFISH
Avatar
jim
Here and Now
Member
Pro Subscribers
sp_UserOfflineSmall Offline
Nov 3, 2013 - 3:06 pm

Yellow Swordfish said
...could you check your php error log?

I've been monitoring the server error log via cPanel in hopes that it would shed some light, but nothing unusual shows up other than a lot of denied access requests because of our firewall settings.

There are no SQL related errors in the log, and the only mentions of "simple-press" include...

[Sun Nov 03 12:27:12 2013] [error] [client 67.142.130.48] File does not exist: /public_html/wp-admin/images/ed-bg-vs.gif, referer: .../wp-admin/admin.php?page=simple-press/admin/panel-options/spa-options.php

This one is interesting since it doesn't appear that image is in the core wordpress download.

[Sun Nov 03 12:27:09 2013] [error] [client 67.142.130.48] File does not exist: /public_html/wp-content/plugins/jetpack-lite/_inc/images/menuicon-sprite.png , referer: .../wp-admin/admin.php?page=simple-press/admin/panel-options/spa-options.php

And this one is interesting since that image file does exist, though it looks like there is a blank space after the filename.

<still perplexed>

TripawdsSimple:Press powers the Tripawds Discussion Forums.

It's better to hop on three legs than to limp on four.

The Tripawds Blogs Community is made possible by The Tripawds Foundation.

Avatar
jim
Here and Now
Member
Pro Subscribers
sp_UserOfflineSmall Offline
Nov 3, 2013 - 3:38 pm

OK, this may reveal a bit more...

I noticed the physical error_log file in our root directory had grown so extraordinarily large that I couldn't view it. So I regenerated the log file, reactivated SimplePress and kept it on for a few minutes this time before I started noticing the loads increase again.

I checked that log file and it's filled with errors. Could you or Steve please review this full error_log file? here are a few interesting excerpts....

[03-Nov-2013 20:10:11 UTC] PHP Notice:  Undefined index: forumname

[03-Nov-2013 20:15:53 UTC] WordPress database error MySQL server has gone away for query UPDATE `wp_1_sfoptions` SET `option_value` = 'a:4:{s:16:"css-sp-theme-css";a:2:{s:8:"modified";i:1378753824;s:4:"type";s:3:"all";}s:3:"ver";a:1:{s:3:"all";i:61;}s:16:"css-sp-admin-bar";a:2:{s:8:"modified";i:1378299212;s:4:"type";s:3:"all";}s:17:"css-sp-share-this";a:2:{s:8:"modified";i:1356538920;s:4:"type";s:3:"all";}}' WHERE `option_name` = 'sp_css_concat' made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/wpmu-nelo-child/page-full-width-forums.php'), get_header, locate_template, load_template, require_once('/themes/wpmu-nelo-child/problem-with-post-edit-buttoneader.php'), wp_head, do_action('wp_head'), call_user_func_array, wp_print_styles, do_action('wp_print_styles'), call_user_func_array, sp_load_plugin_styles, sp_combine_plugin_css_files, sp_update_option

[03-Nov-2013 20:16:22 UTC] WordPress database error MySQL server has gone away for query SELECT  * FROM wp_1_sftopics WHERE topic_slug='chemo-for-osteosarcoma-yes-or-no' made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/wpmu-nelo-child/page-full-width-forums.php'), get_header, locate_template, load_template, require_once('/themes/wpmu-nelo-child/problem-with-post-edit-buttoneader.php'), wp_head, do_action('wp_head'), call_user_func_array, sp_forum_header, sp_setup_meta_tags, sp_get_metadescription, spdb_table, spdb_select

[03-Nov-2013 20:16:22 UTC] WordPress database error MySQL server has gone away for query SHOW TABLES LIKE 'wp_1_sferrorlog' made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/wpmu-nelo-child/page-full-width-forums.php'), get_header, locate_template, load_template, require_once('/themes/wpmu-nelo-child/problem-with-post-edit-buttoneader.php'), wp_head, do_action('wp_head'), call_user_func_array, sp_forum_header, sp_setup_meta_tags, sp_get_metadescription, spdb_table, spdb_select, sp_construct_database_error

"MySQL server has gone away" question frown

Not sure what this means as our sites are still operating and the database is on the same server.

Hopefully something here or in all these errors may help explain where I can go from here.

THANK YOU!

TripawdsSimple:Press powers the Tripawds Discussion Forums.

It's better to hop on three legs than to limp on four.

The Tripawds Blogs Community is made possible by The Tripawds Foundation.

Avatar
Yellow Swordfish
Glinton, England
SP Master
sp_UserOfflineSmall Offline
Nov 3, 2013 - 4:33 pm

The 'server gone away' message can be a hard one to track down. And the problem with it is that once it has gone away it can spawn a lot of error entries..

It is probably best to just ask you to do a search on the term as there is a lot of documentation on this available but basically it means your MySQL Server is timing out and stopping. The difficult question is... why. And the secondary question is why has it started now. You are probably going to need to involve your host support in this one. But some questions.... Has your MySQL Server been updated recently? Reading suggests that some versions can be more prone to this than others. Has the timeout value been altered for any reason? Have any MySQL settings been changed in the server config of late? Actually even the Apache settings may have an effect.

andy-signature.png
YELLOW
SWORDFISH
Forum Timezone: Europe/Stockholm
Most Users Ever Online: 1170
Currently Online:
Guest(s) 1
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Mr Papa: 19448
Ike: 2086
Brandon: 864
kvr28: 804
jim: 650
FidoSysop: 577
Conrad_Farlow: 531
fiddlerman: 358
Stefano Prete: 325
Member Stats:
Guest Posters: 620
Members: 17365
Moderators: 0
Admins: 4
Forum Stats:
Groups: 7
Forums: 17
Topics: 10128
Posts: 79626