Package outils

Class CharsetToolkit


  • public class CharsetToolkit
    extends java.lang.Object

    Utility class to guess the encoding of a given byte array. The guess is unfortunately not 100% sure. Especially for 8-bit charsets. It's not possible to know which 8-bit charset is used. Except through statistical analysis. We will then infer that the charset encountered is the same as the default standard charset.

    On the other hand, unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are easy to find. For UTF-8 files with no BOM, if the buffer is wide enough, it's easy to guess.

    Tested against a complicated UTF-8 file, Sun's implementation does not render bad UTF-8 constructs as expected by the specification. But with a buffer wide enough, the method guessEncoding() did behave correctly and recognized the UTF-8 charset.

    A byte buffer of 4KB or 8KB is sufficient to be able to guess the encoding.

    Usage:

     // guess the encoding
     Charset guessedCharset = com.glaforge.i18n.io.CharsetToolkit.guessEncoding(file, 4096);
    
     // create a reader with the charset we've just discovered
     FileInputStream fis = new FileInputStream(file);
     InputStreamReader isr = new InputStreamReader(fis, guessedCharset);
     BufferedReader br = new BufferedReader(isr);
    
     // read the file content
     String line;
     while ((line = br.readLine())!= null)
     {
         System.err.println(line);
     }
     

    Date: 18 juil. 2002

    • Constructor Summary

      Constructors 
      Constructor Description
      CharsetToolkit​(byte[] buf)
      Constructor of the com.glaforge.i18n.io.CharsetToolkit utility class.
      CharsetToolkit​(byte[] buf, java.nio.charset.Charset defCharset)
      Constructor of the com.glaforge.i18n.io.CharsetToolkit utility class.
    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static java.nio.charset.Charset[] getAvailableCharsets()
      Retrieves all the available Charsets on the platform, among which the default charset.
      java.nio.charset.Charset getDefaultCharset()
      Retrieves the default Charset
      static java.nio.charset.Charset getDefaultSystemCharset()
      Retrieve the default charset of the system.
      boolean getEnforce8Bit()
      Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
      java.nio.charset.Charset guessEncoding()
      Guess the encoding of the provided buffer.
      static java.nio.charset.Charset guessEncoding​(java.io.File f, int bufferLength)
      essaie de deviner l'encodage du fichier f
      static java.nio.charset.Charset guessEncoding​(java.io.File f, int bufferLength, java.nio.charset.Charset defaultCharset)
      essaie de deviner l'encodage du fichier f
      static void main​(java.lang.String[] args)  
      void setDefaultCharset​(java.nio.charset.Charset defCharset)
      Defines the default Charset used in case the buffer represents an 8-bit Charset.
      void setEnforce8Bit​(boolean enforce)
      If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • CharsetToolkit

        public CharsetToolkit​(byte[] buf)
        Constructor of the com.glaforge.i18n.io.CharsetToolkit utility class.
        Parameters:
        buf - the byte buffer of which we want to know the encoding.
      • CharsetToolkit

        public CharsetToolkit​(byte[] buf,
                              java.nio.charset.Charset defCharset)
        Constructor of the com.glaforge.i18n.io.CharsetToolkit utility class.
        Parameters:
        buf - the byte buffer of which we want to know the encoding.
        defCharset - the default Charset to use in case an 8-bit charset is recognized.
    • Method Detail

      • setDefaultCharset

        public void setDefaultCharset​(java.nio.charset.Charset defCharset)
        Defines the default Charset used in case the buffer represents an 8-bit Charset.
        Parameters:
        defCharset - the default Charset to be returned by guessEncoding() if an 8-bit Charset is encountered.
      • setEnforce8Bit

        public void setEnforce8Bit​(boolean enforce)
        If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII. It might be a file without any special character in the range 128-255, but that may be or become a file encoded with the default charset rather than US-ASCII.
        Parameters:
        enforce - a boolean specifying the use or not of US-ASCII.
      • getEnforce8Bit

        public boolean getEnforce8Bit()
        Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.
        Returns:
        a boolean representing the flag of use of US-ASCII.
      • getDefaultCharset

        public java.nio.charset.Charset getDefaultCharset()
        Retrieves the default Charset
        Returns:
        defaultCharset
      • guessEncoding

        public java.nio.charset.Charset guessEncoding()

        Guess the encoding of the provided buffer.

        If Byte Order Markers are encountered at the beginning of the buffer, we immidiately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.

        If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).

        It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.

         UCS-4 range (hex.)        UTF-8 octet sequence (binary)
         0000 0000-0000 007F       0xxxxxxx
         0000 0080-0000 07FF       110xxxxx 10xxxxxx
         0000 0800-0000 FFFF       1110xxxx 10xxxxxx 10xxxxxx
         0001 0000-001F FFFF       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
         0020 0000-03FF FFFF       111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
         0400 0000-7FFF FFFF       1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
         

        With UTF-8, 0xFE and 0xFF never appear.

        Returns:
        the Charset recognized.
      • guessEncoding

        public static java.nio.charset.Charset guessEncoding​(java.io.File f,
                                                             int bufferLength)
                                                      throws java.io.FileNotFoundException,
                                                             java.io.IOException
        essaie de deviner l'encodage du fichier f
        Parameters:
        f - le fichier dont on doit déterminer l'encodage
        bufferLength - longueur du buffer
        Returns:
        le charset détecté
        Throws:
        java.io.FileNotFoundException - fichier f non trouvé
        java.io.IOException - erreur de lecture du fichier f
      • guessEncoding

        public static java.nio.charset.Charset guessEncoding​(java.io.File f,
                                                             int bufferLength,
                                                             java.nio.charset.Charset defaultCharset)
                                                      throws java.io.FileNotFoundException,
                                                             java.io.IOException
        essaie de deviner l'encodage du fichier f
        Parameters:
        f - le fichier dont on doit déterminer l'encodage
        bufferLength - longueur du buffer
        defaultCharset - le charset par défaut à utiliser
        Returns:
        le charset détecté
        Throws:
        java.io.FileNotFoundException - fichier f non trouvé
        java.io.IOException - erreur de lecture du fichier f
      • getDefaultSystemCharset

        public static java.nio.charset.Charset getDefaultSystemCharset()
        Retrieve the default charset of the system.
        Returns:
        the default Charset.
      • getAvailableCharsets

        public static java.nio.charset.Charset[] getAvailableCharsets()
        Retrieves all the available Charsets on the platform, among which the default charset.
        Returns:
        an array of Charsets.
      • main

        public static void main​(java.lang.String[] args)
                         throws java.io.FileNotFoundException,
                                java.io.IOException
        Parameters:
        args - arguments de la fonction main
        Throws:
        java.io.FileNotFoundException - fichier non trouvé
        java.io.IOException - erreur de lecture du fichier