Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.


I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.


Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?


Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.


Even then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.


I disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: