Hi, I don't think you can get rid of this issue on the edges by increasing resolution, pixel samples or changing filter type.
here I attached the way I do separating of objects and collecting matte information of objects as separate passes in .exr format which gives a good result when you wish composting them and relighting, grading elements,
the benefit of using this way is you render your scene one time and get all passes you want in one go.
the zip is including your hip file which is edited (a houdini comp is added) , and a nuke composting sample is also in the zip file , hope it helps
comp.zip