Jump to content

losing decimal precision in string to float conversion


Recommended Posts

I have another problem now. Not really my day!

I have some geometry on which I've stored some height information as string detail attributes, eg '45.40000152587890625 45.450000762939453125 45.479999542236328125 45.529998779296875 45.5699996948242187'

Now I want to split each string into an array of strings, convert each of those to a float and store those in arrays. The problem is that when I do so, I end up with eg '45.40 45.45 45.47 45.52 45.56'. It's storing only 2 decimal places and I can't figure out why.

You can see in the screengrab the difference. Attached is a test file

Thanks for looking




Edited by j00ey
Link to comment
Share on other sites

I do not actually have an answer but just wanted to share a thing: I'm not sure the lost of precision is coming from the conversion from string.

if you run

@base = 123456789;
@div = pow(10, 9);

@RESULT =  @base / @div;

RESULT will return 0.1234567 instead of 0.123456789

not sure where to go from there though.

Link to comment
Share on other sites

weirdly though if you do

@result = 12.3456789;

the @result is 12.3457, so the rounding isn't even happening in the atof function. Note also this is 4 decimal places but... if you do as iamyog says

@base = 123456789;
@div = pow(10, 9);

@RESULT =  @base / @div;

the result is 0.123457, ie 6 decimal places. Both though have 6 non zero digits. In my results from the original posts I was getting a maximum of 4 non zero digits.

Very odd

Link to comment
Share on other sites


@result = 0.123456789;

= 0.123457

@result = 1.123456789;

= 1.12346

@result = 12.123456789;

= 12.1235


@result = 123456.123456789;

= 123456.1


and so on.

wonder if this is what shows or what really is stored in the attrib



@result = 123456.123456789;
@test = @result - 123456;


returns 0.123457

so i guess this is just visualisation rounding

Edited by sasho78
Link to comment
Share on other sites

this is just single floating point accuracy, which is generally accepted to be 6 digits I think.

even though Houdini floats are really 32 bit by default (in wrangles), which can represent more digits accurately, I guess its just the same baseline that is chosen.

Link to comment
Share on other sites

1) Houdini displays floats in "g" form, which is shorter representation.

1.23456789       1.23457
12.3456789       12.3457
123.456789       123.457
1234.56789       1234.57
1234567.89       1.23457e+06
0.0000123456789  1.23457e-05

2) It seems that VEX ignores different precision and always converts floats to 32 bits. It's the case for other Houdini nodes, but not all of them. I lost precision with Point SOP, but Attribute works well. I guess, it depends on value types used inside HDK.


3) That number "45.479999542236328125" looks like a single precision float stored in "double-precision" style. So, it is just a 32-bit value of 45.48. And half of the digits are rounding error garbage and may vary across different computers (if string generated again). You will loose nothing restoring it as 32-bit floats. But if the numbers looks as precise as doubles, like "45.479999999999997", you loose precision.

It's all only my thoughts, I may be wrong somewhere.

Edited by f1480187
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...