20060301
4 Number of Digits and Precision
k
Number of Digits
Standard Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic
Format dialog box is not selected.
• Up to 611 digits are stored in memory for integer values.
• Decimal values up to 15 digits are converted to fraction format and saved in memory. When
a mathematical expression cannot be converted to fraction format, the result is displayed in
decimal format.
• Values stored in memory are displayed as-is, regardless of how [Number Format] settings
(Normal 1, Normal 2, Fix 0 – 9, Sci 0 – 9) are configured (except when a decimal value is
displayed).
Decimal Mode
The following applies when the check box next to the “Decimal Calculation” item on the Basic
Format dialog box is selected.
• Values stored in Ans memory and values assigned to variables have the same number of
digits as defined for Standard mode values.
• Values are displayed in accordance with how [Number Format] settings (Normal 1, Normal 2,
Fix 0 – 9, Sci 0 – 9) are configured.
• Displayed values are rounded to the appropriate number of decimal places.
• Some applications store values using a mantissa up to 15 digits long and a 3-digit
exponent.
k
Precision
• Internal calculations are performed using 15 digits.
• The error for a single mathematical expression (Decimal mode calculation error) is
±
1 at the
10th digit. In the case of exponential display, calculation error is
±
1 at the least significant
digit. Note that performing consecutive calculations causes error to be cumulative. Error
is also cumulative for internal consecutive calculations performed for: ^(
x
y
),
x
,
x
!,
n
P
r
,
n
C
r
, etc.
• Error is cumulative and tends to be larger in the vicinity of a function’s singular point(s) and
inflection point(s), and the vicinity of zero. With sinh(
x
) and tanh(
x
), for example, the
inflection point occurs when
x
= 0. In this vicinity, error is cumulative and precision is poor.
α
-4-1
Number of Digits and Precision
To Next Page
Loading...
