do you have any more info?? what is the option name getting created?  by itself, captcha plugin does NOT create any transient records… but wp or sp core may be… just need the full record to try to understand…

this one looks like a post failure message…  which is needed for when the page is reloaded to know the reason for failure so a status message can be displayed…

wordpress handling of transients is far from optimal, but they will get remove fairly quickly – as you notice by the small number of actual records…  1991 entries in the options table is actually quite small…  I have some sites with tens of thousands and no performance hit…

how is that creating a load on the server??

@Mr Papa – thank you for replying. You are correct, it is a post failure message. There are 2 entries per failure, with the naming scheme of _transient_timeout_{ip}post and _transient_{ip}post.

wordpress handling of transients is far from optimal, but they will get remove fairly quickly – as you notice by the small number of actual records…  1991 entries in the options table is actually quite small…  I have some sites with tens of thousands and no performance hit…

When the server was going down and I noticed the huge amount of cpu usage by mysql, I ran show full processlist quite a few times to see what exactly was causing it. When I did, each time I think I saw 6 – 12 of those record being simultaneously inserted. I don’t have a db dump from the day of the actual server outtages unfortunately, but I do have one from the day before they started. The bulk of the records in the table are these entries,  and from then until I locked the forums there were roughly 30k entries added and deleted to the wp_options table. Keep in mind the whole server was down for hours and hours during this time period. While some of those inserts may have been due to other actions based on what I was witnessing it’s a safe bet most were from those failed captcha attempts. In comparison, the forums were locked to new posts on June 20th, and since then there have been only 56k inserts, or roughly 1.4k inserts per day. That is a huge difference.

this one looks like a post failure message…  which is needed for when the page is reloaded to know the reason for failure so a status message can be displayed…

Is there a reason this couldn’t be handled with a session variable? Or do the posting via ajax, and return the error directly instead of having to reload the page and re-query for the error? Can you think of any other suggestions to alleviate the load this is causing?

30k of transients? Or 30k of those very specific transients? WordPress creates huge numbers of transient records and leaves them forever lying around. It is just about he most awful development they ever did. I regularly clean then out. So it is an important distinction.

IF you are genuinely referring to huge numbers of the SP post failure transients all related to the captcha then that suggests to me that either there are a very large number of illiterate users or – and perhaps this is more likely – the captcha plugin is not working correctly. And if it is the latter then surely that is the most important task to address? Is it not?

Yellow Swordfish said
30k of transients? Or 30k of those very specific transients? WordPress creates huge numbers of transient records and leaves them forever lying around. It is just about he most awful development they ever did. I regularly clean then out. So it is an important distinction.

IF you are genuinely referring to huge numbers of the SP post failure transients all related to the captcha then that suggests to me that either there are a very large number of illiterate users or – and perhaps this is more likely – the captcha plugin is not working correctly. And if it is the latter then surely that is the most important task to address? Is it not?

In 1 day the auto increment went up by 30k entries, whereas with the forum locked it went up by 1.4k per day for the next 38 days. Since they are transient and get deleted, no, I cannot say with 100% certainty that is what they were. However, prior to locking the forums and keeping a continuous eye on the mysql queries that were running, those inserts were pretty much all I saw, and I saw them in excess. Locking the forums to new posts immediately resolved the performance issue, and this is on a server with ~160 sites at the time, the bulk of which are WordPress.

IF you are genuinely referring to huge numbers of the SP post failure transients all related to the captcha then that suggests to me that either there are a very large number of illiterate users or – and perhaps this is more likely – the captcha plugin is not working correctly.

Or, even more likely: bots. Looking through the entries the user names were things like “spain jersey”, “WOLFSBURG JERSEY HOME”, “hermes bag”, “iraq jersey”, etc… in other words, the captcha is doing exactly what it is supposed to be doing, which is blocking non-human traffic. However, once the bot traffic hits a certain threshold posting all of those failed captcha to the database causes an issue. Wasting database resources to tell a bot that they entered in an incorrect captcha is very inefficient. With normal captchas on WordPress comments, for instance, I believe the norm is to check the captcha first, before processing anything else, and if failed raise a WordPress error, thus halting execution of the script, not performing any more mysql queries, and saving on resource usage.

Which brings me back to the question… how do I fix this?

Thanks.

not sure what to fix yet.. hence the discussion and trying to fully understand the problem…

so if your exact model were followed in this instance, there would be no feedback to the user that there was a problem with the captcha, the transient is the method that provides that feedback…  obviously not an issue for bogus bots, but valid for a real user… the checking on captcha, just like for comments, occurs on submit… 

and the reason for the transient is to not lose the post data you just entered… we could handle just a captcha check without a transient, but then you would lose the post data… so say you typed in 10,000 words in the post as a valid user but forgot to do the captcha…  if we simply stated post refused because wrong captcha with a page reload, the post content would be lost and have to be retyped over… sessions or other post/get mechanisms dont really lend themselves well to saving of data like that… so a transient is used..  it would be quite annoying and aggravating for real users to lose data like that – and it has and will happen…

so yeah, I can give you a code change to prevent the transients if you want…  but any real users that have that issue will lose the post content on failed save for captcha or other (this is not a captcha issue)…

would you like that change?

this is odd as I still dont understand how that is killing a dedicated machine… wp designed transients (wont get into the quality of – Andy already expressed his frustration) for these kind of activities… and oddly, we have not had any other users with an issue like this… and we have some users with tremendously busy sites (thousands real users on forum at same time)…  so if we are moving slower than you like on it, please hang in there, we cannot make a knee jerk reaction immediately…

if we simply stated post refused because wrong captcha with a page reload, the post content would be lost and have to be retyped over

Is there a way to check the captcha via ajax? This would de-necessitate the need to leave the page to check it, thus saving the content, and should minimize the resources used in blocking bots.

this is odd as I still dont understand how that is killing a dedicated machine

It’s possible there was something else going on with the server as well, but I can only go based on what my investigation showed, which I described above. Also, yes, this is a dedicated machine for my client, but he has 160 other clients on there, so this was not the only site by a longshot. Also, it’s possible that this was just due to a particularly nasty wave of bots during that time period. The forum is not new, and appears to have been running fine for quite some time. However, the unfortunate thing is that to test that theory requires putting the server at risk of hours of downtime again, which is not something we really want to do to be honest.

we have thoughts of changing posting to be ajax driven… its not just a captcha change – its just a plugin for simple press (yes, plugin for plugin)…  it would require the whole sp posting mechanism to change… but that is a significant rewrite of the current logic…

hours of downtime? sorry, wont ask why…

but try this…  find file simple-press/forum/library/sp-post.php…  way down near the bottom, in function sp_return_to_post(), you will find this line of code:

    sp_add_transient(1, $failure);

you can comment it out and the post failure transients wont be logged…  should let you do some testing if you wanted without fear of lots of wp transients…  but as mentioned, you lose some post failure feedback…

I have been thinking about this more. I have not implemented the above solution yet, but thank you for that. My guess is that those entries are symptoms of the massive amount of bots trying to post, and that it is the overall processing on each attempt that was causing the load and not the entries themselves. Perhaps I should have mentioned before that due to the nature of the forum the client feels it is important to allow people to post anonymously, without needing to register, so that captcha is the only thing stopping the bots, and it is only stopping the posts from actually going through, not the processing needed on each attempt.

My thought process is that odds are the bots are most likely not visiting the forum before attempting each post, and that the captcha is failing because it is empty. Would the error be the same for an empty captcha as an incorrect one? If so, is there a way to simply halt all execution if the captcha is not included with the post? Would it be in the same file, simple-press/forum/library/sp-post.php ?

My second thought was including a randomly generated hash in a hidden form field on the post form, one that includes the timestamp in it, and look for and decrypt that field before doing anything else. If the field is missing, invalid, or more than 1 hour old, assume that the post is from a bot, and again, just kill the page.

Any thoughts on either of those methods, or their impact?

sorry, bit hectic day getting 5.3 released… and Andy is not around today…  we started a discussion about ajax for the captcha…  very problematic…

as to guest posting, couple of other ideas…  are posts moderated?  and do you use the admin bar plugin for simple press?  if using the admin bar,  you can have all posts run through akismet for spam prevention…  seems to do pretty well…

have to think on your other ideas this weekend when the 5.3 flurry of activity for a release dies down…

I just went in to try and edit the sp-post.php with the intention of checking to see if the captcha was even filled out, and if it was blank halt execution there. This should kill the majority of the bots while causing a minimal impact on real people. However, I am running into an issue trying to figure out exactly what I should be checking for. Tracing through the code it appears as if there are 3 checks, 1) a honeypot for whether or not the url field has been filled out, 2) a built in math captcha, and 3) an addon for a capthca located in wp-content/sp-resources/forum-plugins/captcha/library/sp-captcha.php.

Based on the actual errors getting submitted to the database it appears that it is the addon that is blank when the form is submitted. I cannot tell though if this means that it is getting checked first, or if 1 & 2 are being checked and pass and then it fails on 3. My guess is that it is the former. Where should I be looking to implement an emptiness test (or filled in test in the case of the url) for these fields, if I want them to be checked before anything else happens? What exactly should I be checking and where should I put it in the code?

Thanks.

I am not sure where you got some of what you said… so lets walk through it…

a post gets submitted via the submit button… there is some js that runs there and does a few initial checks such as is there any post content?  if it passes the js validation, the form gets submitted…

then we go through and validate that the user making the post actually has permission to post in that forum/topic…  the post can be rejected at this point…  currently, we dont fire any kind of hook here, but we have already added one into the next version of SP, so theoretically you could hook in here and do your checks…

then we go through and validate the data…  this is where we make sure the data meets a bunch of other requirements to be posted, ie number of links, valid guest name, not a duplicate post, etc…  again, no hook here, but there is one already in next version so someplace else you could hook into the process…

at this point, the post class is happy with the post.. but we still need to open it up to other plugins to add any needed checks… so we fire the sph_new_forum_post hook for this… 

This is where captcha is hooked into for checking…  so captcha will actually be the first check…  It hooks in with normal priority of 10… so you could hook in with higher priority and do your checks first…

any other plugins hooked in to that hook will also run their checks…

if all those pass, then we run the math spam check if you the user permission requires it… the math spam check may not be needed if users have permission to bypass the check…

assuming all the passes, we save the data to the db, perform post saving stuff… and fire on last hook to for those that want to know a new post is official – sph_post_create…

and that is it…  I am not sure what you mean by #1… we dont do a honeypot… or have a hidden url field… wonder if there is some other plugin at work here?  or perhaps I am misunderstanding what you are seeing…

when the captcha runs, it verifies that the $_POST[‘captcha’] fields matches… so you should be able to hook in at higher priority on hook sph_new_forum_post, check that field and if empty, return an error and bail…

the hook passes a $newpost array…  just set $newpost[‘error’] element to some error string, return the array and the post saving will be aborted…

we dont do a honeypot… or have a hidden url field… wonder if there is some other plugin at work here?  or perhaps I am misunderstanding what you are seeing…

In sp-api/sp-api-primitives.php, function sp_spamcheck(), first if{} block:

    # Check dummy input field
    if (array_key_exists ('url', $_POST)) {
        if (!empty($_POST['url'])) {
            $spamcheck[0] = true;
            $spamcheck[1] = sp_text('Form not filled by human hands!');
            return $spamcheck;
        }
    }

That corresponds to a hidden field on the form named “url”, which is, from what I can tell, generated on line 401 of sp-startup/site/sp-site-support-functions.php:

        $out = '<input type="hidden" size="30" name="url" value="" /></p>'."n";

That is, as far as I know, in the core SimplePress, and not in a plugin.

What I want to do is if it is not a human, halt all further processing, and I need to put this as high up in the code as possible. I don’t know whether or not the bots are filling out the url field, since it’s not getting that far, so I would like to create a check for the empty captcha at the top, prior to even checking if the user has permissions, in order to prevent these bots from using up the server resources. What exactly would I need to check for, and where should I put this check? I am ok with a non-graceful die() at that point as well, so no need for any pretty error messages.

 *note: I stated that I wanted to put this check in sp-post.php based on the earlier suggestion on where to block the transient entries, and because the naming scheme suggests that file is what is initially invoked when a post is submitted. However, if there is a more efficient (eg, further up in the process) place where I should place this check please let me know, thanks.

the line numbers have clearly changed in sp-site-support.php but I assume you are referring to the function spa_register_math() which is where this is still used. It is not used in the post editor form. That is used in the WP registration form if the user wishes to add the ‘math’ question to it.

If you interrupt the process in the sp-post.php file then really that is not going to give you much as it still means a page load – and a die() in there would just leave things in limbo.

If you REALLY want to stop the processing then you need to get something into the JavaScript validation routine that is called on the submit of the editor form. Have you looked at that?

At the moment it is hard, of course, to change JavaScript without changes being lost on the next plugin update but as I believe the WordPress team are talking about introducing a filter/action type hook interface for JavaScript code and if that does get into WP core then that is the opportunity to do this safely.

Ok, this is getting really frustrating…

Changing Javascript will not stop the bots. The bots are, I am fairly certain, not loading the page to begin with. What I need to do is this:

If on a post submission the captcha is empty, not process anything. What do I look for as far as the form fields to make this check, and where in the files should I put it for maximum effectiveness?