Support Forum

Advanced Search
Forum Scope


Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
general-topic
Weird unicode characters appear when editing with the TinyMCE editor
Avatar
Helping Outreach
Member
Free Members
sp_UserOfflineSmall Offline
Aug 12, 2016 - 10:08 pm

So I've recently upgraded an older (4.x) Siimple:Press installation to the latest version (as of yesterday, anyway, 5.6.5), and have noticed that some single and double quotes are being garbled.

Let's say someone entered a sentence with a nice typographical apostrophe and quotes back on the old 4.x forum, like this:

I'd like it if my "friend" stopped using "quotes" inappropriately.

Well, this at some point went into the database as:

I& #39;d like it if my “friend” stopped using “quotes” inappropriately.

(Except for the space between the ampersand and the #39; bit -- pretend it's not there; I actually had to add that space because otherwise it just displays here on this forum as an apostrophe. Arrgggh)

Okay, fine, so in the older version of Simple:Press it saved the HTML codes for things. No worries, because on the old forum, when I edited this post, it would pop up just fine in the rich text editor. (In case anyone is wondering, the database table wp_sf_posts is set to be utf8_general_ci, and was in the previous version as well.)

After upgrading, it still displays fine in the forum. But now, when I try to edit the exact same post in the visual editor (TinyMCE plugin), it's running it through a filter and converting it. So now, it shows up in the edit post editor as this:

I& #39;d like it if my “friend” stopped using “quotes” inappropriately.

Oops! and when I click the <> button to reveal the "source code," it shows that, sure enough, TinyMCE has run it through "htmlspecialchars," double-converting the ampersands so it's actually treating the post as if it were:

I&#39;d like it if my &ldquo;friend&rdquo; stopped using &ldquo;quotes&rdquo; inappropriately.

Uhhhh, thanks TinyMCE, but no thanks.

Why is it doing this now when it worked fine before? Is this a known issue?

Do I have to run some kind of global database search and replace for all these special characters? Thank you!

Avatar
Mr Papa
Simi Valley, CA
SP Master
Free Members
sp_UserOfflineSmall Offline
Aug 13, 2016 - 12:05 am

as of yesterday, the current version was 5.6.7 was bit out of date...and today, we released 5.7..  but I highly doubt it has anything to do with it...

when did you convert from 4.x???  that was many years ago (4+?)...

afraid there is no known issue here...  we dont (and dont remember) convert entities like that... but that was a long time ago...  we did not have issues like this with our user base that upgraded from 4.x to 5.x back in the day...  so some questions to gather more info...

Are these quotes common to all posts/posters?  or do they perhaps come from a few users that might be potentially using other charset quotes?

I think its somehow related to how its stored in your database...  did you use tinymce in 4.x? or could the posts have been created via another editor?

I am also a bit confused as to the problem...  is this just an issue for editing? couple quick tests show them displaying fine...

Forum Timezone: Europe/Stockholm
Most Users Ever Online: 1170
Currently Online:
Guest(s) 1
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Mr Papa: 19448
Ike: 2086
Brandon: 864
kvr28: 804
jim: 650
FidoSysop: 577
Conrad_Farlow: 531
fiddlerman: 358
Stefano Prete: 325
Member Stats:
Guest Posters: 620
Members: 17365
Moderators: 0
Admins: 4
Forum Stats:
Groups: 7
Forums: 17
Topics: 10128
Posts: 79626