Diminishing returns from Compiled Blocks?

[MJNorris]

Hi,

I've just got a question about compile blocks.

I'm working on a relatively simple process that I need to run on extremely heavy data sets (groom curves), and I've noticed something odd.

I need to do a check on each curve, and for reasons I need to do it on every curve in isolation so I'm putting my loop inside a compile block, If I loop over 100'000 curves and run this process then I get extremely fast results for the first 10'000 or so curves, however after about the halfway point I see my CPU usage drops right off and instead of seeing it calculate hundreds of curves per second it's down to 10-20 curves per second.

It calculates the first half of the data set in about 4 mins but takes 30 mins to do the second half!

At first I thought that this was due to the data set getting progressively heavier, however if that was true I would expect the CPU usage to remain high, despite the progress slowing down? I also had a go at randomising the sort order too, to see if that changed the result and it was the same.

So I guess my question is this:

What factors, apart from the size of the data-set could cause a compiled block process to slow down over time?

Any insights are very welcome!

Thanks in advance!

M