question

KeiichiOta-6948 avatar image
0 Votes"
KeiichiOta-6948 asked KeiichiOta-6948 commented

Azure Remote Rendering doesn't show textured models

I am trying to get the open data provided by the Ministry of Land, Infrastructure, Transport and Tourism of Japan and display the city model of Tokyo using Holorens 2.
I went through several processes to combine multiple small FBX files, create the combined FBX data, convert the files for Azure Remote Rendering, and place them in the cloud. The file is described below.

arrAsset : https://arrdemo20211108.blob.core.windows.net/arrt-convertedmodels/533946.arrAsset
info.json : https://arrdemo20211108.blob.core.windows.net/arrt-convertedmodels/533946.info.json
result.json : 160419-hololens2.jpghttps://arrdemo20211108.blob.core.windows.net/arrt-convertedmodels/533946.result.json

This model is about 1 million polygons. The model also includes the texture of the roof and walls of the building.
I tried to watch this model on HoloLens 2, but unfortunately I couldn't see it.
So I created a model without texture. As a result, the model could be seen on HoloLens 2. I was also able to see a model when two small FBX models were combined.
Therefore, it seems that the reason why it cannot be displayed is due to the amount of data of the texture that has become bloated as a result of combining.

Question 1
Is there a way to display a file containing this texture data on HoloLens 2 using Azure Remote Rendering?

Question 2
The instruction manual for using Azure Remote Rendering clearly states the restrictions on the number of polygons, but unlike this time, the amount of texture data is not stated. Are there any restrictions other than the number of polygons?

Question 3
I think that it can be displayed with HoloLens 2 if multiple models are displayed as they are, without combining the models.
However, in that case, I find it difficult to align small models. This is because the coordinate data of FBX disappears when you convert to Azure Remote Rendering.
Is there a way to keep the coordinate data of FBX when converting to Azure Remote Rendering? Even if I erase it, is there a way to reset it?


hololens-developmentazure-remote-rendering
hololens2.jpg (95.5 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FlorianBorn-2377 avatar image
1 Vote"
FlorianBorn-2377 answered FlorianBorn-2377 commented

Hello and thank you for providing detailed information!

I have downloaded the arrAsset and I'm analyzing it now.

Looking at the output.json, the first thing that comes to mind is that the model's box is far off-center:
"boundingBox": {
"min": [
-76.05452728271485,
-0.04086558520793915,
342.1785583496094
],
"max": [
-41.80239486694336,
2.1324777603149416,
370.4200439453125
]

So the center of the scene is somewhere at ~350 meter in z-direction and that's behind the (default) far clip plane of the renderer.
Are you compensating for this offset in your app by transforming the model? Anyway, I'm not sure if that is the real issue here, but I'll find out.

To answer your other questions:

Q1: No, we don't expose this kind of debug functionality out-of-the-box. (Unless of course you do this fully manually through the client API: load the model, find the node, find the material, grab the material's albedo texture handle, then load a small model programmatically, and assign the texture handle to the small model's material)

Q2: There is no hard limit on the number of textures that can be used. There is only a limit on the maximum size of a single texture (16k x 16k), which is in fact a GPU hardware limit. I think I can make the docs more specific here.

Q3: ARR conversion should keep the coordinates as-is, unless you specify the "recenter to origin" option. In that case it's indeed harder to align multiple models because each one recenters to its own average position. However you can look into this section of the json output:

     "recenteringOffset": [
         0.0,
         0.0,
         0.0
     ],

...to see by which amount the object had been moved during conversion (it is (0,0,0) when re-centering was disabled, like in this case). Aligning should be about re-applying this offset to every model.

I'll keep you updated on my findings with the arrAsset.
Let me know if you have more questions!

Cheers,
Florian

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @KeiichiOta-6948 ,

Did you have a look at @FlorianBorn-2377 answers? Did it help you continuing with your project successfully?

Thank you!


.Remember:
- Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
- Want a reminder to come back and check responses? Here is how to subscribe to a notification.

1 Vote 1 ·

Sorry for the late contact due to the New Year holidays.

Is there a sample program that you could display using my fbx model that I sent you the other day?
If so, please let me know.

As you pointed out, I have confirmed that the fbx model I sent you the other day is far from the origin of ARR.
So now I've tried to improve it and make it appear.

I'm using the ShowCase App in the Microsoft tutorial.
I'm also trying out QuickStart, which was included in the tutorial package.

1 Vote 1 ·

Hi,

Is there a sample program that you could display using my fbx model

I have only displayed the converted asset using ARR yet, I haven't tested the original fbx.

The challenge is to find an external viewer program that can handle big .fbx files. I don't know a particular one, but I can ask in the team if there are any recommendations.
I'll answer in this thread when I have something.

Cheers, Florian
2 Votes 2 ·
Show more comments
FlorianBorn-2377 avatar image
1 Vote"
FlorianBorn-2377 answered

Quick update: I was able to see the model when I transformed it to compensate for the offset as indicated by the output bounds.
Specifically, I applied the following translation to the root object after loading: (59, -1, -356).

However the scene graph looks a bit malformed and maybe that is already part of the input fbx: There are thousands of scene graph nodes under the root with no meshes applied, and only the very last node has a mesh that references everything.

Can you confirm that this is a problem with the initial offset?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KeiichiOta-6948 avatar image
0 Votes"
KeiichiOta-6948 answered

Sorry for the confusing message.
My problem was that I couldn't see "533946.arrAsset" when using the sample program named "Showcase" on GitHub.

[Showcase]
https://github.com/Azure/azure-remote-rendering/tree/master/Unity/Showcase
[533946.arrAsset]
https://arrdemo20211108.blob.core.windows.net/arrt-convertedmodels/533946.arrAsset

I understand that the origin of the model you pointed out is offset, and I'm sure it still can't be displayed.
If you can, would you please try displaying "533946.arrAsset" in "Showcase"?

Then I tried a sample program named "Quickstart".

[Quickstart]
https://github.com/Azure/azure-remote-rendering/tree/master/Unity/Quickstart

As a result, "Quickstart" was able to display "533946.arrAsset".
From these results, I thought that the reason why "533946.arrAsset" could not be displayed this time was not on the remote rendering service side but on the displaying program side.

So I'm going to compare "Showcase" and "Quickstart" to find out why the model couldn't be displayed.
I will tell you as soon as the result is known.

We are very grateful for the progress made in your research.164761-%E3%83%95%E3%83%A9%E3%83%88%E3%83%BClod2%E3%83%86%E3%82%AF%E3%82%B9%E3%83%81%E3%83%A3%E3%83%BC%E3%81%82%E3%82%8A.jpg



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FlorianBorn-2377 avatar image
1 Vote"
FlorianBorn-2377 answered KeiichiOta-6948 commented

OK, I see now what you mean.
The difference is indeed between QuickStart and Showcase, not the service.
Quickstart has some code in there that places the object automatically near you when the box extent is far off the origin. The code in question is in file RemoteRendering.cs, line 216 ff:


 [...]
     var aabb = (await loadModelResult.Root.QueryLocalBoundsAsync()).toUnity();
     bool tooBig = aabb.extents.magnitude > Camera.main.farClipPlane;
     bool tooFar = aabb.center.magnitude > Camera.main.farClipPlane;
     float scaleFactor = 1.0f;
     String modelMessage = "Model loaded";
     if (tooBig)
     {
         scaleFactor = (2.0f / aabb.extents.magnitude);
         rootGO.transform.localScale = (rootGO.transform.localScale * scaleFactor);
         modelMessage += $", too big (scaled to {(scaleFactor).ToString("P2", CultureInfo.InvariantCulture)})";
     }
     rootGO.transform.localPosition = (rootGO.transform.localPosition - aabb.center * scaleFactor);
     if (tooFar)
     {
         modelMessage += $", center too far (moved by {-aabb.center})";
     }

specifically the line that sets the object's transform to the middle of its bounding box:

 rootGO.transform.localPosition = ....


Showcase on the other hand does not have this kind of code, because the placement happens with the hand controllers when you place the blue animated blob. And that position is always near the origin.
Does this explain it? Maybe we can think about how to improve this - as a temporary solution you may add something similar to the Showcase code?

Cheers,
Florian




· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for your quick and informative reply.
Thank you for understanding this situation again.
I haven't tried the method described yet, so I'd definitely like to try it from now on.
I will let you know the result.

Best Regards,
Keiichi Ota

1 Vote 1 ·

Thank you for your advice.
Following that advice, I tried adding a "Quick Start" process to the "Showcase". Unfortunately, it didn't work.
For additional information, I've told you that I can't see the model so far. To be precise, in this regard, the blue blob animation model is placed and the model is fully loaded, but the model disappears and appears, and so on. It is occurring.
Also, when it is displayed, if you try to move or enlarge the model, the model will disappear. The operation is very unstable.
I recorded the operation status at that time. Can you see it if you like?
https://youtu.be/T8o0uCxak90

My internet environment was good and both download and upload were over 100Mbps. I don't think the internet environment is the cause, as "Quickstart" doesn't cause any problems in this internet environment.

0 Votes 0 ·

OK, I will try to reproduce this.
Do I need a new .arrAsset for that or can I use the one from your very first post?
I ask because if I remember correctly I did try a modified Showcase and I did not see the bug. But anyway, I will try with your model regardless and apply some showcase source changes.

Florian

1 Vote 1 ·
Show more comments

I'm still investigating the problem, however I have the first idea that came to mind when testing with the model locally.
Apparently applying a scaling factor to your model is a very expensive operation that causes the physics engine to stall for multiple seconds. A stall will then cause the client application to discard the old video images and show nothing. The object re-appears after a few seconds when the scaling is applied.
My question is, does your interaction with the model apply a different scaling?

1 Vote 1 ·

While we are still investigating, there is a way to mitigate this on your side:
- you can re-convert the model with a "static" scene graph as opposed to default "dynamic". This will use a different scaling code path internally which does not show this problem.
See configure-model-conversion for the "sceneGraphMode" conversion property.
- you can re-convert the model without physics altogether, but then raycasts won't work anymore, so you'd have limited interaction. If that is not a limitation, I'd recommend to turn off physics at conversion time, because it will decrease file size and improve loading times a lot. See this section: configure-model-conversion

The first option (sceneGraphMode:static) should not have any downsides for you, because you can still transform/scale the scene and even move single pieces.

Would it be possible to test this?
Cheers,
Florian

1 Vote 1 ·
Show more comments