Jump to content
motionprojects

size of Houdini Alembic mesh exports larger than expected

Recommended Posts

I exported an animated mesh from Maya for use in Houdini. Maya writes out the .abc file with a size of 139.5 mb.

I import this into Houdini (with Alembic loader inside a geo node) and add some sops to scale it down and clean up attributes.

I'm only dealing with P, vertex normals, and UV.

When I rexport from Houdini using Alembic rop, the .abc file is now 955.9 mb!

If I reimport the new file back into Houdini, I see that a point id attribute is in the data now. I have no idea why, and still can't fathom why the file is so much larger.

Anyone find this to be the case with meshes?

thanks.

Share this post


Link to post
Share on other sites

Replying to my own topic...

I tested out the Alembic Ocotopus sample and did not find any file size bloat when exporting from Houdini. So it must be my file.

In the case of my file, I had a low-poly crowd of walking figures, all baked into one mesh. All with normals, and uv. All sitting in one primitive group.

If I scaled the crowd using a geometry Transform Sop the resulting export would be about 9x the size, and slowish to export.

When I scale the crowd at the Object level, and allow Alembic to collapse transforms, the file size was correct, and export was quick.

Some of my alembic files come back in with a point attribute id attached (all at zero), and I can't account for that.

I solved the problem with the gigantic sizes, but don't have insights as to causes the aberrations.

Edited by motionprojects

Share this post


Link to post
Share on other sites

recently I'm working on a fluid scene, I have my fluid mesh cache out as bgeo.sc, 1 frame is 132,140 kB; but when it cache 1 frame of abc. it's 316,627 kB. Is there a way to reduce the .abc file? I plane to import the whole sequence into Maya.

Share this post


Link to post
Share on other sites

motionprojects if the file size is bigger it must mean there is some additional data that wasn't there before, or the data representation is different. for example moving ball can be saved as a sphere with an animation curve over the entire timeline length, or it can be saved as deforming mesh, modifying topology change at each frame. that would naturally take much more space. also, alembic supports instancing which can make a huge difference in file size. any additional attributes and groups bloats the file too. i guess it'll be one of these things, but it's hard to say just like that.   

ela1995 bgeo.sc will be always smaller file than abc 

Share this post


Link to post
Share on other sites

is there a way to reduce the file size? it's too heavy to load in maya

Share this post


Link to post
Share on other sites
23 minutes ago, ela1995 said:

is there a way to reduce the file size? it's too heavy to load in maya

attributes and groups makes the file larger. delete everything you dont need. fluid sims in general tend to be really big. no way around it I'm afraid, it's simply a lot of data. 300 mb per frame dosen't seem too much to me thou. personally I've never done anything in maya but i guess it should handle it no problem if you've got enough ram.

Share this post


Link to post
Share on other sites

most renderers have alembic procedural, in which case there should be no need to import the geo to Maya, you should be able to render it directly as a stand in or proxy 

also depending on the renderer you may be able to import .abc file sequence rather than single stitched .abc file, therefore avoiding single massive file, since in case of fluid sim mesh there is not much data deduplication that can happen anyways

  • Like 1

Share this post


Link to post
Share on other sites
12 minutes ago, anim said:

most renderers have alembic procedural, in which case there should be no need to import the geo to Maya, you should be able to render it directly as a stand in or proxy 

also depending on the renderer you may be able to import .abc file sequence rather than single stitched .abc file, therefore avoiding single massive file, since in case of fluid sim mesh there is not much data deduplication that can happen anyways

splitting the abc is a good idea actually. will it output file per frame if you just use $F4 in filename on ROP? strange i never tried to do that :)

Share this post


Link to post
Share on other sites
41 minutes ago, davpe said:

will it output file per frame if you just use $F4 in filename on ROP?

yes, that should be enough, it also depends on whether you can read such sequence in renderer, Vray for Maya for sure can for example

there is also checkbox Render Full Range, which you can try turning off if it causes issues during per frame .abc export, currently I don't see the difference, and I don't recall what exactly it's meant to do

Edited by anim
  • Thanks 1

Share this post


Link to post
Share on other sites

Houdini abc export seems to be way bigger than from maya.

I recently decided to recreate an old Maya project in Houdini. For that I had to export animated mesh from M to H so I used alembic. The 960f was 119Mb I did some modifications and tried exporting it as bgeo but it exploded to 106Gb so I tried abc which turned out to be 35Gb. And yes i added some attributes, subdivided few things and stuff, but it still seemed too much for the changes I made so I decided to explore.

I exported animated abc from M of just 11f (step 0.25f) and that was 94Mb. I opened the abc in H with basic alembic node (default H18 settings) and export that with rop_alembic_output node straight from the alembic read node, with the same frame range & step. I've tried different formats (HDF5 and Ogawa (used by Maya) ) and settings here but all had pretty much the same size results of 1.5Gb. I didnt find any settings that would relay to compression or anything like that.

Does anyone have an explanation of this and perhaps a solution/idea how to reduce the abc size from H?

Share this post


Link to post
Share on other sites
13 minutes ago, makah21803 said:

Houdini abc export seems to be way bigger than from maya.

I recently decided to recreate an old Maya project in Houdini. For that I had to export animated mesh from M to H so I used alembic. The 960f was 119Mb I did some modifications and tried exporting it as bgeo but it exploded to 106Gb so I tried abc which turned out to be 35Gb. And yes i added some attributes, subdivided few things and stuff, but it still seemed too much for the changes I made so I decided to explore.

I exported animated abc from M of just 11f (step 0.25f) and that was 94Mb. I opened the abc in H with basic alembic node (default H18 settings) and export that with rop_alembic_output node straight from the alembic read node, with the same frame range & step. I've tried different formats (HDF5 and Ogawa (used by Maya) ) and settings here but all had pretty much the same size results of 1.5Gb. I didnt find any settings that would relay to compression or anything like that.

Does anyone have an explanation of this and perhaps a solution/idea how to reduce the abc size from H?

as was stated before - if the file is bigger it means either there is some additional data, or the data has been saved in a different manner. Alembic is just a framework, and is always the same regardless the application you're using.

Just by looking at your numbers - you said 960f takes 119mb from Maya. now calc this: 119mb * 960 frames =~ 114,000 mb, which roughly matches your 106gb. the calculation i did is the difference between saving non-deforming geometry (geo is saved once and then transformed on every frame) and deforming geometry (saves full ~100mb for each frame as it presumes topology is changing). I suspect that is your issue.

cheers.  

Share this post


Link to post
Share on other sites
54 minutes ago, davpe said:

as was stated before - if the file is bigger it means either there is some additional data, or the data has been saved in a different manner. Alembic is just a framework, and is always the same regardless the application you're using.

Just by looking at your numbers - you said 960f takes 119mb from Maya. now calc this: 119mb * 960 frames =~ 114,000 mb, which roughly matches your 106gb. the calculation i did is the difference between saving non-deforming geometry (geo is saved once and then transformed on every frame) and deforming geometry (saves full ~100mb for each frame as it presumes topology is changing). I suspect that is your issue.

cheers.  

Thanks for quick response :)

See the thing is that even setting the 'Packed Transform' option to Transform Geometry or Merge With Parent Transform (rather than the default Deform Geometry) doesn't reduce the size to the size M exports - it still remains about 5 times larger.

As far as additional data goes, I'm literally just reading the abc from M and directly exporting it. There is just P, N, uv and path all of which is originally there too. And both M and H is set to use the same format (Ogawa). I've tried unpacking it before exporting, but that only made the file size larger.

The closest I got in size is 94Mb for 11f/0.25step animation from M and 481Mb for the same 11f/0.25step animation from H (using Transform Geometry ..which apparently Maya optimises under the hood).

 

I just tested a different geometry (organic animation) that uses Deforming Geo rather than Transform and the result is much closer of 2.2Gb from M and 2.4Gb from H (960f/0.25).

So it seems there is just something about M settings under the hood for non-deforming geometry. Single frame (of the non-deforming robotic animation) from M is 93Mb and from H 116Mb, which is more that 10% difference but not really significant. It seems its the way H stores the transitions that differs, since from single f to 11f/0.25 H makes the file about 4* larger (116Mb -> 480Mb) and M just 1.005* larger (93.5Mb -> 93.9Mb).

// PS: Že přijde pomoc z Čech jsem nečekal :D

Edited by makah21803

Share this post


Link to post
Share on other sites
17 minutes ago, makah21803 said:

Thanks for quick response :)

See the thing is that even setting the 'Packed Transform' option to Transform Geometry or Merge With Parent Transform (rather than the default Deform Geometry) doesn't reduce the size to the size M exports - it still remains about 5 times larger.

As far as additional data goes, I'm literally just reading the abc from M and directly exporting it. There is just P, N, uv and path all of which is originally there too. And both M and H is set to use the same format (Ogawa). I've tried unpacking it before exporting, but that only made the file size larger.

The closest I got in size is 94Mb for 11f/0.25step animation from M and 481Mb for the same 11f/0.25step animation from H (using Transform Geometry ..which apparently Maya optimises under the hood).

well, hard to say then. then there must be some other reason why it yields several times bigger output. perhaps if you can share an example scene?

btw did you also try File>Export>Alembic? (instead of ROP Alembic output). just out of interest  

Share this post


Link to post
Share on other sites
52 minutes ago, davpe said:

well, hard to say then. then there must be some other reason why it yields several times bigger output. perhaps if you can share an example scene?

btw did you also try File>Export>Alembic? (instead of ROP Alembic output). just out of interest  

Ive tried  File>Export>Alembic and it doesnt allow sub frames and still made the file size a bit larger (521Mb ..rop_abc was 480Mb). I tried rop_geometry and fileCache nodes but they arent able to write animation in a single alembic file (if they dont have $F in file path they just keep overriding it each frame).

I also tested a different geometry (organic animation) that uses Deforming Geo rather than Transform and the result is much closer of 2.2Gb from M and 2.4Gb from H (960f/0.25).

So it seems there is just something about M settings under the hood for non-deforming geometry. Single frame (of the non-deforming robotic animation) from M is 93Mb and from H 116Mb, which is more that 10% difference but not really significant. It seems its the way H stores the transitions that differs, since from single f to 11f/0.25 H makes the file about 4* larger (116Mb -> 480Mb) and M just 1.005* larger (93.5Mb -> 93.9Mb).

// PS: Že přijde pomoc z Čech jsem nečekal :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×