Viewing file:  charset.htm (9.87 KB) -rw-r--r--
Select action/file-type:
 (+) |  (+) |  (+) | Code (+) | Session (+) |  (+) | SDB (+) |  (+) |  (+) |  (+) |  (+) |  (+) |
OpenSP - Character sets
Handling of character sets in OpenSP
The following description applies only to the multi-byte version of
OpenSP. In the single-byte version of OpenSP, each character is represented
both internally and in storage objects by a single byte equal to the
number of the character in the document character set.
OpenSP's entity manager converts the bytes comprising a storage object
into a sequence of characters.
This conversion is determined by the encoding associated with
the storage object.
An encoding may be specified using the name of a mapping from
sequences of characters to sequences of bytes.
An encoding can also be specified relative to the document character
set. This kind of an encoding maps a sequence of characters in the
repertoire of the document character set into a sequence of bytes by
-
mapping each character to its bit
combination in the document character set, and then
-
applying a transformation that maps sequences of bit combinations
to sequences of bytes.
The transformation applied in the second step is called a bit
combination transformation format (BCTF).
A document character set relative encoding is specified by giving
the name of a BCTF.
An application receives characters from OpenSP represented as non-negative
integers.
The mapping from characters to integers is determined by OpenSP's internal
character set.
OpenSP can operate in a mode in which the internal character set is the
same as the document character set.
(Versions of SP up to 1.1.1 always operated in this mode.)
The multibyte version of OpenSP can also operate in a mode in which the
internal character set does not vary with the document character set,
but is always a fixed character set, known as the system character set;
this mode of operation is called fixed character set mode.
Environment
SP's character set handling is controlled by the following
environment variables:
-
SP_CHARSET_FIXED
-
If this variable is 1 or YES, then OpenSP will operate in fixed character set mode.
-
SP_SYSTEM_CHARSET
-
This identifies the system character set. When in fixed character set
mode, this character set is used as the internal character set. When
not in fixed character set mode this character set is used as the
internal character set until the document character set has been read,
at which point the document character set is used as the internal
character set.
The only currently recognized value for this is JIS.
This refers to a character set which combines JIS X 0201, JIS X 0208
and JIS X 0212 by adding 0x8080 to the codes of characters in JIS X
0208 and 0x8000 to the codes of characters in JIS X 0212.
The default system character set is Unicode 2.0.
-
SP_ENCODING
-
This specifies the default encoding when operating in fixed character set mode.
The value must be the name of an available encoding.
The default encoding cannot be document character set relative
when operating in fixed character set mode.
-
SP_BCTF
-
This specifies the default encoding when not operating in fixed character set mode.
The value must be the name of an available BCTF.
When not operating in fixed character set mode, the default encoding is
the document character set relative encoding with this BCTF.
The default encoding is required to be document character set relative
when not operating in fixed character set mode.
The default encoding is used for file input and output, and, except
under Windows 95 and Windows NT, for all other interfaces with the
operating system including filenames, environment varable names,
environment variable values and command line arguments.
Under Windows 95 and Windows NT there are no restrictions on the
default encoding. Note that in order for non-ASCII characters to be
correctly displayed on your console you must select a TrueType font,
such as Lucida Console, as your console font. (This seems to work
only on Windows NT.)
Under other operating systems, the default encoding must be one in
which ASCII characters are represented by a single byte.
Applications built with OpenSP may require fixed character set mode and a
particular system character set; such applications will ignore the
SP_SYSTEM_CHARSET and SP_CHARSET_FIXED environment variables.
Encoding names are case insensitive.
The following named encodings are available:
-
utf-8
-
Each character is represented by a variable number of bytes
according to UCS Transformation Format 8 defined in Annex P to be
added by the first proposed drafted amendment (PDAM 1) to ISO/IEC
10646-1:1993.
-
utf-16
-
Each character is represented by a variable number of bytes
according to UCS Transformation Format 16 defined in Annex O to be
added by the first proposed drafted amendment (PDAM 1) to ISO/IEC
10646-1:1993.
-
ucs-2
-
iso-10646-ucs-2
-
This is ISO/IEC 10646 with the UCS-2 transformation format.
Each character is represented by 2 bytes.
No special treatment is given to the byte order mark character.
-
ucs-4
-
iso-10646-ucs-4
-
utf-32
-
This is ISO/IEC 10646 with the UCS-4 transformation format.
Each character is represented by 4 bytes.
-
unicode
-
Each character is represented according to the utf-16 encoding.
The bytes representing the entire storage object may be preceded by a pair of
bytes representing the byte order mark character (0xFEFF). The bytes
representing each character are in the system byte order, unless
the byte order mark character is present, in which case the order of
its bytes determines the byte order. When the storage object is read,
any byte order mark character is discarded.
-
euc-jp
-
This is equivalent to
the Extended_UNIX_Code_Packed_Format_for_Japanese Internet charset.
Each character is encoded by a variable length sequence of octets.
-
euc-kr
-
This is ASCII and KSC 5601 encoded with the EUC encoding
as defined by KS C 5861-1992.
-
euc-cn
-
cn-gb
-
gb2312
-
This is ASCII and GB 2312-80 encoded with the EUC encoding.
It is equivalent to the CN-GB MIME charset defined in RFC 1922.
-
sjis
-
shift_jis
-
This is equivalent to the Shift_JIS Internet charset.
Each character is encoded by a variable length sequence of octets.
This is Microsoft's standard encoding for Japanese.
-
big5
-
cn-big5
-
This is equivalent to the CN-Big5 MIME charset defined in RFC 1922.
-
is8859-n
-
iso-8859-n
-
n can be any single digit other than 0. Each
character in the repertoire of ISO 8859-n is represented
by a single byte.
-
koi8-r
-
koi8
-
The koi8-r encoding as defined in RFC 1489.
-
xml
-
On input, this uses XML's rules to determine the encoding.
On output, this uses UTF-8.
The following additional encodings are supported under Windows 95
and Windows NT:
-
windows
-
Specify this encoding when a storage object is encoded using your
system's default Windows character set.
This uses the so-called ANSI code page.
-
wunicode
-
This uses the unicode encoding if the storage object starts
with a byte order mark and otherwise the windows encoding.
If you are working with Unicode, this is probably the best value
for SP_ENCODING.
-
ms-dos
-
Specify this encoding when a storage object (file) uses the OEM code page.
The OEM code-page for a particular
machine is the code-page used by FAT file-systems on that machine and
is the default code-page for MS-DOS consoles.
The following BCTFs are available:
-
identity
-
Each bit combination is represented by a single byte.
-
fixed-2
-
Each bit combination is represented by exactly 2 bytes,
with the more significant byte first.
-
fixed-4
-
Each bit combination is represented by exactly 4 bytes,
with the more significant byte first.
-
utf-8
-
Each bit combination is represented by a variable number of bytes
according to UCS Transformation Format 8 defined in Annex P to be
added by the first proposed drafted amendment (PDAM 1) to ISO/IEC
10646-1:1993.
-
euc
-
Each bit combination is represented by a variable number of bytes
depending on the values of the 0x80 and 0x8000 bits:
-
if neither bits are set, then the bit combination is
represented by a single byte equal to the bit combination;
-
bit combinations with both bits set, are represented by the MSB of the
bit combination followed by the LSB of the bit combination;
-
bit combinations with just the 0x80 bit set are represented
by 0x8E followed by a byte equal to the bit combination;
-
bit combinations with just the 0x8000 bit set are represented
by 0x8F followed by the MSB of the bit combination followed
by the LSB of the bit combination.
-
sjis
-
A bit combination between 0 and 127 or between 161 and
223 is encoded as a single byte with the same value as the bit combination.
A bit combination with the 0x8000 and 0x80 bits set is encoded by the
sequence of bytes with which the SJIS encoding encodes the character
whose number in JIS X 0208 added to 0x8080 is equal to the bit
combination.
-
big5
-
A bit combination less than 0x80 is encoded as a single byte.
A bit combination with the 0x8000 bit set is encoded as two bytes,
the MSB of the bit combination followed by the LSB of the bit
combination.
|