Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
314 views
in Technique[技术] by (71.8m points)

excel - Compare double in VBA precision problem

I have trouble comparing 2 double in Excel VBA

suppose that I have the following code

Dim a as double
Dim b as double
a = 0.15
b = 0.01

After a few manipulations on b, b is now equal to 0.6

however the imprecision related to the double data type gives me headache because

if a = b then
 //this will never trigger
end if

Do you know how I can remove the trailing imprecision on the double type?

Question&Answers:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You can't compare floating point values for equality. See this article on "Comparing floating point numbers" for a discussion of how to handle the intrinsic error.

It isn't as simple as comparing to a constant error margin unless you know for sure what the absolute range of the floats is beforehand.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...