Non latin characters don't work

[krko]

Hello there,

First please let me say that form tools is a great product. I've been using it some time now and I was pleased to install the latest 2 version.

I have a small problem though. Until now I have used the version 1.x and all worked well. It's version 2 I'm having problem with.

I'm using the form tools on a non-latin language website (Hebrew). The website is in utf-8. I installed the form tools, created my form and when I submitted the test submission all was ok. The submission was received properly, the form tools recognized all the entries and all entries were stored correctly. I completed my form, went on to submit a real submission and instead of Hebrew letters in form submissions all I see is question marks. The same goes for Excel and emails. In al of them I see question marks instead of text. Latin characters are however stored ok and displayed properly.

I must say that I have had this problem with version 1.x also but I realized that it was due to too small field size value. When I changed the field size to Very Large (MEDIUMBLOB) all worked fine. In version 2 however, this doesn't work. Of course in version 2 I chose Very Large (MEDIUMTEXT).

Please help me sort this out as I'd hate to go back to version 1.x now that its end of life has been announced.

Thank you all in advance,
krko

---

I just noticed looking in the db in phpmyadmin that the following tables have latin1_swedish_ci collation:

ft_form_1 - I assume that's the form I created
ft_module_pages - don't know what's this one
ft_module_swift_mailer_email_template_fields - email fields?
ft_sessions - don't know ehat's this one

All others have utf8_general_ci collation.

Default MySQL charset on the server is: UTF-8 Unicode (utf8).
Default MySQL connection collation is: utf-8_unicode_ci

So, what should I do?

Thank you.

krko

---

Hello there,

First please let me say that form tools is a great product. I've been using it some time now and I was pleased to install the latest 2 version.

I have a small problem though. Until now I have used the version 1.x and all worked well. It's version 2 I'm having problem with.

I'm using the form tools on a non-latin language website (Hebrew). The website is in utf-8. I installed the form tools, created my form and when I submitted the test submission all was ok. The submission was received properly, the form tools recognized all the entries and all entries were stored correctly. I completed my form, went on to submit a real submission and instead of Hebrew letters in form submissions all I see is question marks. The same goes for Excel and emails. In al of them I see question marks instead of text. Latin characters are however stored ok and displayed properly.

I must say that I have had this problem with version 1.x also but I realized that it was due to too small field size value. When I changed the field size to Very Large (MEDIUMBLOB) all worked fine. In version 2 however, this doesn't work. Of course in version 2 I chose Very Large (MEDIUMTEXT).

Please help me sort this out as I'd hate to go back to version 1.x now that its end of life has been announced.

Thank you all in advance,
krko

---

I just noticed looking in the db in phpmyadmin that the following tables have latin1_swedish_ci collation:

ft_form_1 - I assume that's the form I created
ft_module_pages - don't know what's this one
ft_module_swift_mailer_email_template_fields - email fields?
ft_sessions - don't know ehat's this one

All others have utf8_general_ci collation.

Default MySQL charset on the server is: UTF-8 Unicode (utf8).
Default MySQL connection collation is: utf-8_unicode_ci

So, what should I do?

Thank you.

krko

I also changed the collation manually to all tables to utf8_unicode_ci, made a new submission and still same question marks... I double checked that the new collation for all tables is as set but it doesn't work.

[Ben]

Hi krko,

That's very strange...! I just checked and you're quite right, some tables don't have the correct collation. Just so you know, you can safely change the collations without messing with any of Form Tools code. I'd set it to utf8_general_ci like the other tables. I'll log this as a bug and address it in the next release.

Things to check:

- Check your /global/config.php file doesn't contain a $g_unicode = false; line and your /global/library.php does contain a $g_unicode = true; line (unicode is enabled by default, so unless you edited it this should be fine).

- When you look at the database directly (e.g. via phpMyAdmin or something), does the newly inputted data (i.e. inputted since you changed the collation type on the field) look correct? Or does it appear as question marks / nonsense chars there as well? If so, it looks like the problem is with the information getting to the database correctly. Either way, it's a big clue.

Also, is your form a POST or an API form?

- Ben

[krko]

Hi krko,

That's very strange...! I just checked and you're quite right, some tables don't have the correct collation. Just so you know, you can safely change the collations without messing with any of Form Tools code. I'd set it to utf8_general_ci like the other tables. I'll log this as a bug and address it in the next release.

Things to check:

- Check your /global/config.php file doesn't contain a $g_unicode = false; line and your /global/library.php does contain a $g_unicode = true; line (unicode is enabled by default, so unless you edited it this should be fine).

- When you look at the database directly (e.g. via phpMyAdmin or something), does the newly inputted data (i.e. inputted since you changed the collation type on the field) look correct? Or does it appear as question marks / nonsense chars there as well? If so, it looks like the problem is with the information getting to the database correctly. Either way, it's a big clue.

Also, is your form a POST or an API form?

- Ben

Hi Ben,

I think I have a temporary solution for the problem. BEFORE installing form tools, changing the database collation while it's empty and has nothing in it to unicode solves the problem. If you do that, all tables will be created with unicode collation and everything will work.

Me happy so far

Btw, the form is POST and config.php is fine and I haven't edited it.

krko

[Ben]

Good to know! However, Form Tools should still explicitly create all database tables as utf8, so I'll fix that much at least - this *should* do away with the problem.

All the best -

Ben

[pmdkrex]

Add <input name="cheat" type="hidden" value="ﭒ"> to your form and accept-charset="utf-8" after process.php

[Ben]

Excellent tip!