Jump to content

[Solved] Custom LiDAR importer with python


sant0s81

Recommended Posts

13 hours ago, sant0s81 said:

What do you mean loading under a second to load?

 

Julien means the same thing I already mentioned. Loading and building attributes in Python is much slower than doing same thing in C++ most of the time. Lidar Import SOP is a fast C++ node compared to pylas. Loading your *.las example takes 20ms with Lidar Import SOP on my machine but same file takes almost 6 seconds with Python SOP with pylas  (3 seconds just for transforming numpy nd array to serialized form and another 2.5 seconds for setPointFloatAttribValues method).

If you do have enough memory and you do end up reading most of the file anyway it's faster just to use Lidar Import SOP to load all points fast and then use Python SOP to add only additional data that Lidar Import SOP cannot read (like classification) and blast what you don't need. Like this:

laspy_lidar_faster_import.thumb.png.933c90d966b9028ce7c04befee05a755.png

 

 

 

  • Like 1
Link to comment
Share on other sites

Okay, wow....thats cool :)

But anyway, you must have crazy machines... I load at leas 20 till 25 seconds for the big .las.

@pezetko

One thing I just realised, I cannot load .laz - only .las.
Is that a restriction?
I can convert them, no problem - but somewhere I saw it may work witz .laz, too.

Edited by sant0s81
Link to comment
Share on other sites

Another quick test - so the HF is on maximum for UE4, means 8129 x 8129.
Of course there is only one texture now and it looks shitty - but was only to test with the resolution.

The LiDAR file out of the box, combined with super little erosion to keep the main shape, alreadygives some crazy details on the ground.
The trees are also on the "correct" positions filtered by the classification.

Tomorrow I will try to make it with more layers of the HF and the different classifications from the LiDAR.

Have a good evening everyone.

 

Edited by sant0s81
  • Like 1
Link to comment
Share on other sites

Nice,

I think fast ssd is much more important nowadays.

I didn't try *.laz with laspy, may worth a try. I know I submitted a few RFEs for Lidar Import SOP, one for *.laz support as well as one for supporting a newer version of *.las format. If you find some of those features important, submit yours, more "votes" (RFEs) doesn't hurt ;)

 

Btw: if you replace

np.concatenate(array)

with

np.ravel(array)

the second one is 10x faster.

 

And this is even faster than the second one (but only few ms).

array.reshape(-1)

 

Edited by pezetko
better code example
  • Like 1
Link to comment
Share on other sites

Just a few more optimizations to native python types, and testing on bigger las file.

Python laspy SOP is 1.2 - 1.6x slower than Import Lidar SOP (1.6x if I apply scale, offset and add classification attribute what Import Lidar SOP does not perform), not bad.

This is the code:

from laspy.file import File
import numpy as np

node = hou.pwd()
geo = node.geometry()

file_path = geo.attribValue("file_path")

def load_color(inFile):
    missing_color = ["red", "green", "blue"]
    for spec in inFile.point_format:
        if spec.name in missing_color:
            missing_color.remove(spec.name)

    if missing_color:
        return None
    color = np.vstack((inFile.red, inFile.green, inFile.blue)).transpose()
    return (color / 255.0).reshape(-1) # transform from 1-255 to 0.0-1.0 range)


with File(file_path, mode='r') as inFile:
    # --- load point position
    coords = np.vstack((inFile.X, inFile.Z, inFile.Y)).transpose()  # 632 ms
    scale = inFile.header.scale	# should be already np.array
    offset = inFile.header.offset # there is no offset in simple.las example from laspy library

    # --- compute scaled and offseted positions and transform in 1d array
    pos = (coords*scale+offset).reshape(-1)     # 300 ms
    geo.setPointFloatAttribValues("P", pos.tolist())     # 2203 ms

    # --- add classification attribute
    geo.addAttrib(hou.attribType.Point, "classification", 0, False, False)
    geo.setPointFloatAttribValues("classification", inFile.Classification.tolist())     # 450 ms

    # --- load color
    colors = load_color(inFile)
    if colors is not None:
        geo.addAttrib(hou.attribType.Point, "Cd", (1.0,1.0,1.0), False, False)  # add color atttribute
        geo.setPointFloatAttribValues("Cd", colors.tolist())
        
    # --- load intensity
    geo.addAttrib(hou.attribType.Point, "intensity", 0.0, False, False)  # add intensity atttribute
    geo.setPointFloatAttribValues("intensity", (inFile.intensity / 512.0).tolist()) # transform from 1-255 to 0.0-1.0 range)

 

laspy_lidar_comparsion.png.e9a8e754e2cb048a26a83f7bdefe2995.png

Edited by pezetko
fixed inFile.x to inFile.X
  • Like 1
Link to comment
Share on other sites

16 hours ago, pezetko said:

I didn't try *.laz with laspy, may worth a try. I know I submitted a few RFEs for Lidar Import SOP, one for *.laz support as well as one for supporting a newer version of *.las format. If you find some of those features important, submit yours, more "votes" (RFEs) doesn't hurt ;)

@pezetko

Awesome upadtes, thanks!

What do you mean with REFs?
References?

I can upload some more *.*las and also *.*laz to try.
I guess the wetransfer link will expire soon - shall I gonna upload on google drive or something?

And you can download very detailed LiDAR of Thueringen/Germany here: https://www.geoportal-th.de/de-de/Downloadbereiche/Download-Offene-Geodaten-Thüringen/Download-Höhendaten
Choose the orange LiDAR date, they are soooo crazy detailed... :D

And I have a 12,xGB LiDAR file here if you are interested to test with crazy amount of data.
Here is a screenshot - just a little cutout, combined with real world texture and some HF textures - and of course some quick and dirty scattered trees ^^
steiger_wip.thumb.jpg.5e9324c45b9b1bacf25deeb087020d6e.jpg

Or do you mean with votes, sending that feature to SideFX?
Already did. :)

Edited by sant0s81
Link to comment
Share on other sites

Just realized that there is a difference between inFile.x (x position with scale and offset applied) and inFile.X (X position without any scale or offset). So Lidar Import SOP produces scaled and offset position from las.

 

And for the laz there is laszip executable from https://rapidlasso.com/laszip/ that you have to have somewhere on your PATH. There is also c++ based laz-perf https://pypi.org/project/lazperf/

Edited by pezetko
laz
  • Like 1
Link to comment
Share on other sites

  • 10 months later...

Solved: This is just what I've been looking for ! Although, I have limited experience with python, as such, upon loading the  pz_load_las_with_python_classification_attribute_memory_fix.hipnc from pezetko, on the load_las_and_read_point_count, I receive this error: 


Error 
Python error: Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ImportError: No module named laspy.file

Solved the above... Although I am getting an inversion in the point cloud... the whole point cloud is inverted like looking in a mirror...it backwards. *see attached the foliage heavy area in the real world is left rather than the output with it on the right... how might I invert the result? 

Also have any of you made further progress on the pipeline from LiDAR - Houdini - Ue4... cheers

inverted from actual scan.PNG

ok I resolved the above ... sort of ... by changing the

 load_point_attributes_from_file node from try:
 

  # --- load point position
    coords = np.vstack((inFile.x, inFile.z, inFile.y)).transpose()  # 632 ms

to 

try:
    # --- load point position
    coords = np.vstack((inFile.x, inFile.y, inFile.z)).transpose()  # 632 ms
 

then adding a 'transform' node, clicking 'move origin to centroid' and in the transform node offsetting the 'rotate',  x,  by -90.

---------------

I am waiting on moderator to ok my comments and in the meantime have resolved the questions I had. Although I am very happy to find a few of you who know how to work on the exact problems I am trying to solve. Have you all made any progress in improving or utilizing the LiDAR data from Houdini into UE4. I would greatly like to improve a pipeline from LiDAR to Houdini to UE4 in a project much like that of Santos81.


   

Edited by knightvid
resolved but with remaining question
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...