[Solved] Custom LiDAR importer with python

[sant0s81]

Nothing fancy, just to show how easy it is to go super simple from LiDAR to UE4 with that cool tool now

[pezetko]

Hi, no, this is just default comment from Python SOP.

3 hours ago, sant0s81 said:

# Add code to modify contents of geo. # Use drop down menu to select examples.

 

In the latest version, https://forums.odforce.net/applications/core/interface/file/attachment.php?id=56259 classification is just an integer attribute on the points.

 

 

 

[pezetko]

13 hours ago, sant0s81 said:

What do you mean loading under a second to load?

 

Julien means the same thing I already mentioned. Loading and building attributes in Python is much slower than doing same thing in C++ most of the time. Lidar Import SOP is a fast C++ node compared to pylas. Loading your *.las example takes 20ms with Lidar Import SOP on my machine but same file takes almost 6 seconds with Python SOP with pylas  (3 seconds just for transforming numpy nd array to serialized form and another 2.5 seconds for setPointFloatAttribValues method).

If you do have enough memory and you do end up reading most of the file anyway it's faster just to use Lidar Import SOP to load all points fast and then use Python SOP to add only additional data that Lidar Import SOP cannot read (like classification) and blast what you don't need. Like this:

 

 

 

[sant0s81]

Okay, wow....thats cool

But anyway, you must have crazy machines... I load at leas 20 till 25 seconds for the big .las.

@pezetko

One thing I just realised, I cannot load .laz - only .las.
Is that a restriction?
I can convert them, no problem - but somewhere I saw it may work witz .laz, too.

Edited July 12, 2020 by sant0s81

[sant0s81]

Another quick test - so the HF is on maximum for UE4, means 8129 x 8129.
Of course there is only one texture now and it looks shitty - but was only to test with the resolution.

The LiDAR file out of the box, combined with super little erosion to keep the main shape, alreadygives some crazy details on the ground.
The trees are also on the "correct" positions filtered by the classification.

Tomorrow I will try to make it with more layers of the HF and the different classifications from the LiDAR.

Have a good evening everyone.

 

Edited July 13, 2020 by sant0s81

[pezetko]

Nice,

I think fast ssd is much more important nowadays.

I didn't try *.laz with laspy, may worth a try. I know I submitted a few RFEs for Lidar Import SOP, one for *.laz support as well as one for supporting a newer version of *.las format. If you find some of those features important, submit yours, more "votes" (RFEs) doesn't hurt

 

Btw: if you replace

np.concatenate(array)

with

np.ravel(array)

the second one is 10x faster.

 

And this is even faster than the second one (but only few ms).

array.reshape(-1)

 

Edited July 13, 2020 by pezetko
better code example

[pezetko]

Just a few more optimizations to native python types, and testing on bigger las file.

Python laspy SOP is 1.2 - 1.6x slower than Import Lidar SOP (1.6x if I apply scale, offset and add classification attribute what Import Lidar SOP does not perform), not bad.

This is the code:

from laspy.file import File
import numpy as np

node = hou.pwd()
geo = node.geometry()

file_path = geo.attribValue("file_path")

def load_color(inFile):
    missing_color = ["red", "green", "blue"]
    for spec in inFile.point_format:
        if spec.name in missing_color:
            missing_color.remove(spec.name)

    if missing_color:
        return None
    color = np.vstack((inFile.red, inFile.green, inFile.blue)).transpose()
    return (color / 255.0).reshape(-1) # transform from 1-255 to 0.0-1.0 range)


with File(file_path, mode='r') as inFile:
    # --- load point position
    coords = np.vstack((inFile.X, inFile.Z, inFile.Y)).transpose()  # 632 ms
    scale = inFile.header.scale	# should be already np.array
    offset = inFile.header.offset # there is no offset in simple.las example from laspy library

    # --- compute scaled and offseted positions and transform in 1d array
    pos = (coords*scale+offset).reshape(-1)     # 300 ms
    geo.setPointFloatAttribValues("P", pos.tolist())     # 2203 ms

    # --- add classification attribute
    geo.addAttrib(hou.attribType.Point, "classification", 0, False, False)
    geo.setPointFloatAttribValues("classification", inFile.Classification.tolist())     # 450 ms

    # --- load color
    colors = load_color(inFile)
    if colors is not None:
        geo.addAttrib(hou.attribType.Point, "Cd", (1.0,1.0,1.0), False, False)  # add color atttribute
        geo.setPointFloatAttribValues("Cd", colors.tolist())
        
    # --- load intensity
    geo.addAttrib(hou.attribType.Point, "intensity", 0.0, False, False)  # add intensity atttribute
    geo.setPointFloatAttribValues("intensity", (inFile.intensity / 512.0).tolist()) # transform from 1-255 to 0.0-1.0 range)

 

Edited July 15, 2020 by pezetko
fixed inFile.x to inFile.X

[sant0s81]

16 hours ago, pezetko said:

I didn't try *.laz with laspy, may worth a try. I know I submitted a few RFEs for Lidar Import SOP, one for *.laz support as well as one for supporting a newer version of *.las format. If you find some of those features important, submit yours, more "votes" (RFEs) doesn't hurt

@pezetko

Awesome upadtes, thanks!

What do you mean with REFs?
References?

I can upload some more *.*las and also *.*laz to try.
I guess the wetransfer link will expire soon - shall I gonna upload on google drive or something?

And you can download very detailed LiDAR of Thueringen/Germany here: https://www.geoportal-th.de/de-de/Downloadbereiche/Download-Offene-Geodaten-Thüringen/Download-Höhendaten
Choose the orange LiDAR date, they are soooo crazy detailed...

And I have a 12,xGB LiDAR file here if you are interested to test with crazy amount of data.
Here is a screenshot - just a little cutout, combined with real world texture and some HF textures - and of course some quick and dirty scattered trees ^^

Or do you mean with votes, sending that feature to SideFX?
Already did.

Edited July 14, 2020 by sant0s81

[sant0s81]

@pezetko

Here is a link to that 12GB LiDAR file:

https://drive.google.com/drive/folders/1zupkunKr0YW3WIfzkBee0V-ttXCzfGAo?usp=sharing

Could you give me feedback, if the link works?
Dont have that much experience with google drive.

[pezetko]

7 hours ago, sant0s81 said:

What do you mean with REFs?

Requests for Enhancement: https://www.sidefx.com/bugs/submit/

I was able to download the file from google drive but my old K4000 can't display it in Houdini.

 

[pezetko]

Just realized that there is a difference between inFile.x (x position with scale and offset applied) and inFile.X (X position without any scale or offset). So Lidar Import SOP produces scaled and offset position from las.

 

And for the laz there is laszip executable from https://rapidlasso.com/laszip/ that you have to have somewhere on your PATH. There is also c++ based laz-perf https://pypi.org/project/lazperf/

Edited July 15, 2020 by pezetko
laz

[knightvid]

Solved: This is just what I've been looking for ! Although, I have limited experience with python, as such, upon loading the  pz_load_las_with_python_classification_attribute_memory_fix.hipnc from pezetko, on the load_las_and_read_point_count, I receive this error: 

Error 
Python error: Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ImportError: No module named laspy.file

Solved the above... Although I am getting an inversion in the point cloud... the whole point cloud is inverted like looking in a mirror...it backwards. *see attached the foliage heavy area in the real world is left rather than the output with it on the right... how might I invert the result? 

Also have any of you made further progress on the pipeline from LiDAR - Houdini - Ue4... cheers

ok I resolved the above ... sort of ... by changing the

 load_point_attributes_from_file node from try:
 

  # --- load point position
    coords = np.vstack((inFile.x, inFile.z, inFile.y)).transpose()  # 632 ms

to 

try:
    # --- load point position
    coords = np.vstack((inFile.x, inFile.y, inFile.z)).transpose()  # 632 ms
 

then adding a 'transform' node, clicking 'move origin to centroid' and in the transform node offsetting the 'rotate',  x,  by -90.

---------------

I am waiting on moderator to ok my comments and in the meantime have resolved the questions I had. Although I am very happy to find a few of you who know how to work on the exact problems I am trying to solve. Have you all made any progress in improving or utilizing the LiDAR data from Houdini into UE4. I would greatly like to improve a pipeline from LiDAR to Houdini to UE4 in a project much like that of Santos81.

   

Edited May 26, 2021 by knightvid
resolved but with remaining question