Jump to content
j00ey

losing decimal precision in string to float conversion

Recommended Posts

I have another problem now. Not really my day!

I have some geometry on which I've stored some height information as string detail attributes, eg '45.40000152587890625 45.450000762939453125 45.479999542236328125 45.529998779296875 45.5699996948242187'

Now I want to split each string into an array of strings, convert each of those to a float and store those in arrays. The problem is that when I do so, I end up with eg '45.40 45.45 45.47 45.52 45.56'. It's storing only 2 decimal places and I can't figure out why.

You can see in the screengrab the difference. Attached is a test file

Thanks for looking

 

odforce_decimal_precision.hiplc

decimal_problem.jpg

Edited by j00ey
typo

Share this post


Link to post
Share on other sites

edit, nevermind, looked at the wrong node

the debug node is returning the right data

Edited by holycause

Share this post


Link to post
Share on other sites

Ah, I was just about to fire up an old version. Thanks for looking anyway.

Share this post


Link to post
Share on other sites

I do not actually have an answer but just wanted to share a thing: I'm not sure the lost of precision is coming from the conversion from string.

if you run

@base = 123456789;
@div = pow(10, 9);

@RESULT =  @base / @div;

RESULT will return 0.1234567 instead of 0.123456789

not sure where to go from there though.

Share this post


Link to post
Share on other sites

also if you try

@result = atof("12.3456789");

the @result will be 12.3457

so there is precision rounding in this function.

Share this post


Link to post
Share on other sites

weirdly though if you do

@result = 12.3456789;

the @result is 12.3457, so the rounding isn't even happening in the atof function. Note also this is 4 decimal places but... if you do as iamyog says

@base = 123456789;
@div = pow(10, 9);

@RESULT =  @base / @div;

the result is 0.123457, ie 6 decimal places. Both though have 6 non zero digits. In my results from the original posts I was getting a maximum of 4 non zero digits.

Very odd

Share this post


Link to post
Share on other sites

interesting...

@result = 0.123456789;

= 0.123457

@result = 1.123456789;

= 1.12346

@result = 12.123456789;

= 12.1235

...

@result = 123456.123456789;

= 123456.1

...

and so on.

wonder if this is what shows or what really is stored in the attrib

 

edit:

@result = 123456.123456789;
@test = @result - 123456;

 

returns 0.123457

so i guess this is just visualisation rounding

Edited by sasho78

Share this post


Link to post
Share on other sites

this is just single floating point accuracy, which is generally accepted to be 6 digits I think.

even though Houdini floats are really 32 bit by default (in wrangles), which can represent more digits accurately, I guess its just the same baseline that is chosen.

Share this post


Link to post
Share on other sites

I see, but I'm only getting 4 digits in my arrays. You can see in the jpg attachment on the original post.

It's actually fine for my purposes, I was just curious as to why it happened.

Thanks for the info.

Share this post


Link to post
Share on other sites

1) Houdini displays floats in "g" form, which is shorter representation.

1.23456789       1.23457
12.3456789       12.3457
123.456789       123.457
1234.56789       1234.57
1234567.89       1.23457e+06
0.0000123456789  1.23457e-05

2) It seems that VEX ignores different precision and always converts floats to 32 bits. It's the case for other Houdini nodes, but not all of them. I lost precision with Point SOP, but Attribute works well. I guess, it depends on value types used inside HDK.

precision.hipnc

3) That number "45.479999542236328125" looks like a single precision float stored in "double-precision" style. So, it is just a 32-bit value of 45.48. And half of the digits are rounding error garbage and may vary across different computers (if string generated again). You will loose nothing restoring it as 32-bit floats. But if the numbers looks as precise as doubles, like "45.479999999999997", you loose precision.

It's all only my thoughts, I may be wrong somewhere.

Edited by f1480187

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×