size of Houdini Alembic mesh exports larger than expected

[motionprojects]

I exported an animated mesh from Maya for use in Houdini. Maya writes out the .abc file with a size of 139.5 mb.

I import this into Houdini (with Alembic loader inside a geo node) and add some sops to scale it down and clean up attributes.

I'm only dealing with P, vertex normals, and UV.

When I rexport from Houdini using Alembic rop, the .abc file is now 955.9 mb!

If I reimport the new file back into Houdini, I see that a point id attribute is in the data now. I have no idea why, and still can't fathom why the file is so much larger.

Anyone find this to be the case with meshes?

thanks.

[motionprojects]

Replying to my own topic...

I tested out the Alembic Ocotopus sample and did not find any file size bloat when exporting from Houdini. So it must be my file.

In the case of my file, I had a low-poly crowd of walking figures, all baked into one mesh. All with normals, and uv. All sitting in one primitive group.

If I scaled the crowd using a geometry Transform Sop the resulting export would be about 9x the size, and slowish to export.

When I scale the crowd at the Object level, and allow Alembic to collapse transforms, the file size was correct, and export was quick.

Some of my alembic files come back in with a point attribute id attached (all at zero), and I can't account for that.

I solved the problem with the gigantic sizes, but don't have insights as to causes the aberrations.

Edited November 28, 2012 by motionprojects

[ela1995]

recently I'm working on a fluid scene, I have my fluid mesh cache out as bgeo.sc, 1 frame is 132,140 kB; but when it cache 1 frame of abc. it's 316,627 kB. Is there a way to reduce the .abc file? I plane to import the whole sequence into Maya.

[davpe]

motionprojects if the file size is bigger it must mean there is some additional data that wasn't there before, or the data representation is different. for example moving ball can be saved as a sphere with an animation curve over the entire timeline length, or it can be saved as deforming mesh, modifying topology change at each frame. that would naturally take much more space. also, alembic supports instancing which can make a huge difference in file size. any additional attributes and groups bloats the file too. i guess it'll be one of these things, but it's hard to say just like that.   

ela1995 bgeo.sc will be always smaller file than abc 

[ela1995]

is there a way to reduce the file size? it's too heavy to load in maya

[davpe]

23 minutes ago, ela1995 said:

is there a way to reduce the file size? it's too heavy to load in maya

attributes and groups makes the file larger. delete everything you dont need. fluid sims in general tend to be really big. no way around it I'm afraid, it's simply a lot of data. 300 mb per frame dosen't seem too much to me thou. personally I've never done anything in maya but i guess it should handle it no problem if you've got enough ram.

[anim]

most renderers have alembic procedural, in which case there should be no need to import the geo to Maya, you should be able to render it directly as a stand in or proxy 

also depending on the renderer you may be able to import .abc file sequence rather than single stitched .abc file, therefore avoiding single massive file, since in case of fluid sim mesh there is not much data deduplication that can happen anyways

[davpe]

12 minutes ago, anim said:

most renderers have alembic procedural, in which case there should be no need to import the geo to Maya, you should be able to render it directly as a stand in or proxy 

also depending on the renderer you may be able to import .abc file sequence rather than single stitched .abc file, therefore avoiding single massive file, since in case of fluid sim mesh there is not much data deduplication that can happen anyways

splitting the abc is a good idea actually. will it output file per frame if you just use $F4 in filename on ROP? strange i never tried to do that

[anim]

41 minutes ago, davpe said:

will it output file per frame if you just use $F4 in filename on ROP?

yes, that should be enough, it also depends on whether you can read such sequence in renderer, Vray for Maya for sure can for example

there is also checkbox Render Full Range, which you can try turning off if it causes issues during per frame .abc export, currently I don't see the difference, and I don't recall what exactly it's meant to do

Edited April 20, 2020 by anim

[makah21803]

Houdini abc export seems to be way bigger than from maya.

I recently decided to recreate an old Maya project in Houdini. For that I had to export animated mesh from M to H so I used alembic. The 960f was 119Mb I did some modifications and tried exporting it as bgeo but it exploded to 106Gb so I tried abc which turned out to be 35Gb. And yes i added some attributes, subdivided few things and stuff, but it still seemed too much for the changes I made so I decided to explore.

I exported animated abc from M of just 11f (step 0.25f) and that was 94Mb. I opened the abc in H with basic alembic node (default H18 settings) and export that with rop_alembic_output node straight from the alembic read node, with the same frame range & step. I've tried different formats (HDF5 and Ogawa (used by Maya) ) and settings here but all had pretty much the same size results of 1.5Gb. I didnt find any settings that would relay to compression or anything like that.

Does anyone have an explanation of this and perhaps a solution/idea how to reduce the abc size from H?

[davpe]

13 minutes ago, makah21803 said:

Houdini abc export seems to be way bigger than from maya.

I recently decided to recreate an old Maya project in Houdini. For that I had to export animated mesh from M to H so I used alembic. The 960f was 119Mb I did some modifications and tried exporting it as bgeo but it exploded to 106Gb so I tried abc which turned out to be 35Gb. And yes i added some attributes, subdivided few things and stuff, but it still seemed too much for the changes I made so I decided to explore.

I exported animated abc from M of just 11f (step 0.25f) and that was 94Mb. I opened the abc in H with basic alembic node (default H18 settings) and export that with rop_alembic_output node straight from the alembic read node, with the same frame range & step. I've tried different formats (HDF5 and Ogawa (used by Maya) ) and settings here but all had pretty much the same size results of 1.5Gb. I didnt find any settings that would relay to compression or anything like that.

Does anyone have an explanation of this and perhaps a solution/idea how to reduce the abc size from H?

as was stated before - if the file is bigger it means either there is some additional data, or the data has been saved in a different manner. Alembic is just a framework, and is always the same regardless the application you're using.

Just by looking at your numbers - you said 960f takes 119mb from Maya. now calc this: 119mb * 960 frames =~ 114,000 mb, which roughly matches your 106gb. the calculation i did is the difference between saving non-deforming geometry (geo is saved once and then transformed on every frame) and deforming geometry (saves full ~100mb for each frame as it presumes topology is changing). I suspect that is your issue.

cheers.  

[makah21803]

54 minutes ago, davpe said:

as was stated before - if the file is bigger it means either there is some additional data, or the data has been saved in a different manner. Alembic is just a framework, and is always the same regardless the application you're using.

Just by looking at your numbers - you said 960f takes 119mb from Maya. now calc this: 119mb * 960 frames =~ 114,000 mb, which roughly matches your 106gb. the calculation i did is the difference between saving non-deforming geometry (geo is saved once and then transformed on every frame) and deforming geometry (saves full ~100mb for each frame as it presumes topology is changing). I suspect that is your issue.

cheers.  

Thanks for quick response

See the thing is that even setting the 'Packed Transform' option to Transform Geometry or Merge With Parent Transform (rather than the default Deform Geometry) doesn't reduce the size to the size M exports - it still remains about 5 times larger.

As far as additional data goes, I'm literally just reading the abc from M and directly exporting it. There is just P, N, uv and path all of which is originally there too. And both M and H is set to use the same format (Ogawa). I've tried unpacking it before exporting, but that only made the file size larger.

The closest I got in size is 94Mb for 11f/0.25step animation from M and 481Mb for the same 11f/0.25step animation from H (using Transform Geometry ..which apparently Maya optimises under the hood).

 

I just tested a different geometry (organic animation) that uses Deforming Geo rather than Transform and the result is much closer of 2.2Gb from M and 2.4Gb from H (960f/0.25).

So it seems there is just something about M settings under the hood for non-deforming geometry. Single frame (of the non-deforming robotic animation) from M is 93Mb and from H 116Mb, which is more that 10% difference but not really significant. It seems its the way H stores the transitions that differs, since from single f to 11f/0.25 H makes the file about 4* larger (116Mb -> 480Mb) and M just 1.005* larger (93.5Mb -> 93.9Mb).

// PS: Že přijde pomoc z Čech jsem nečekal

Edited May 13, 2020 by makah21803

[davpe]

17 minutes ago, makah21803 said:

Thanks for quick response

See the thing is that even setting the 'Packed Transform' option to Transform Geometry or Merge With Parent Transform (rather than the default Deform Geometry) doesn't reduce the size to the size M exports - it still remains about 5 times larger.

As far as additional data goes, I'm literally just reading the abc from M and directly exporting it. There is just P, N, uv and path all of which is originally there too. And both M and H is set to use the same format (Ogawa). I've tried unpacking it before exporting, but that only made the file size larger.

The closest I got in size is 94Mb for 11f/0.25step animation from M and 481Mb for the same 11f/0.25step animation from H (using Transform Geometry ..which apparently Maya optimises under the hood).

well, hard to say then. then there must be some other reason why it yields several times bigger output. perhaps if you can share an example scene?

btw did you also try File>Export>Alembic? (instead of ROP Alembic output). just out of interest  

[makah21803]

52 minutes ago, davpe said:

well, hard to say then. then there must be some other reason why it yields several times bigger output. perhaps if you can share an example scene?

btw did you also try File>Export>Alembic? (instead of ROP Alembic output). just out of interest  

Ive tried  File>Export>Alembic and it doesnt allow sub frames and still made the file size a bit larger (521Mb ..rop_abc was 480Mb). I tried rop_geometry and fileCache nodes but they arent able to write animation in a single alembic file (if they dont have $F in file path they just keep overriding it each frame).

I also tested a different geometry (organic animation) that uses Deforming Geo rather than Transform and the result is much closer of 2.2Gb from M and 2.4Gb from H (960f/0.25).

So it seems there is just something about M settings under the hood for non-deforming geometry. Single frame (of the non-deforming robotic animation) from M is 93Mb and from H 116Mb, which is more that 10% difference but not really significant. It seems its the way H stores the transitions that differs, since from single f to 11f/0.25 H makes the file about 4* larger (116Mb -> 480Mb) and M just 1.005* larger (93.5Mb -> 93.9Mb).

// PS: Že přijde pomoc z Čech jsem nečekal 

[makah21803]

Ive uploaded example files: the abc and hip (just read -> rop). Frame range is 500-510

abcTest.hiplc

testM.abc