Skip to content

[models] glTF not loaded correctly, mesh transform related #1930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chriscamacho opened this issue Aug 18, 2021 · 11 comments
Closed

[models] glTF not loaded correctly, mesh transform related #1930

chriscamacho opened this issue Aug 18, 2021 · 11 comments

Comments

@chriscamacho
Copy link
Contributor

Please, before submitting a new issue verify and check:

  • [X ] I tested it on latest raylib version from master branch
  • [X ] I checked there is no similar issue already reported
  • [X ] My code has no errors or misuse of raylib

Issue description

Previously loading gltf2 models, I saw the same in blender as I did in raylib

Environment

Void Linux, Mesa Intel(R) HD Graphics 5500 (BDW GT2), GL version 4.6

Issue Screenshot

gltf2inBlender
gltf2inRaylib

Code Example

As per my "template" on bedroomcoders.co.uk

@raysan5 raysan5 changed the title [modules] gltf2 regression [models] gltf2 regression, not loading correctly Aug 20, 2021
@raysan5
Copy link
Owner

raysan5 commented Aug 20, 2021

@chriscamacho I'm afraid glTF has suffered several changes lately, it was reviewed on #1849 by @object71 but I'm not sure if he can keep maintaining it.

@chriscamacho
Copy link
Contributor Author

given OBJ loading appears broken (again) currently the only way I've been able to get 3d media into raylib is via my own scratch written OBJ loader...

I'd be inclined to investigate some other options for getting 3d media into raylib, ideally that can be used as an art path from blender (as most can afford the purchase price!)

@object71
Copy link
Contributor

Indeed I cannot continue working at this point. I am occupied with other personal projects. I may review code regarding GLTF though.

When I last worked on the feature 3-4 months ago the loading and display of models was correct. I've cleaned up the code and put comments to describe the process of loading models. The only thing which did not work was animations because they seemed to stretch wierdly when played. There is an open issue about it and I've commented my observations in it.

@630Studios
Copy link
Contributor

given OBJ loading appears broken (again)

In what manner is the OBJ loader broken. I have just tested several OBJ models, as well as the one that was previous listed as causing a crash in RayLib, and there was no issues.

@raysan5 raysan5 changed the title [models] gltf2 regression, not loading correctly [models] glTF not loaded correctly, mesh transform related Aug 27, 2021
@MrDiver
Copy link
Contributor

MrDiver commented Aug 28, 2021

So I will leave a little update on my investigation here!

Model Loading

After a while of testing, I came to the realization that the data loaded by raylib is not wrong. It is actually exactly the same as in the model file that you provided (a direct link would've been helpful :D )

Raylib Model Data
BoneCount: 5
Name: y-axis
Translation:             
[[ 0    0.55 0    ]]
Rotation:
[[ 0 0 0 1 ]]
Scale:
[[ 1 1 1 ]]

Name: z-axis
Translation:
[[ 0.5  0.55 0    ]]
Rotation:
[[ 0 0 0 1 ]]
Scale:
[[ 1 1 1 ]]

Name: x-axis
Translation:
[[ 0    0    0.55 ]]
Rotation:
[[ 0.707 0     0     0.707 ]]
Scale:
[[ 1 1 1 ]]

Name: Cube.001
Translation:
[[ 0.25 0.25 0.25 ]]
Rotation:
[[ 0 0 0 1 ]]
Scale:
[[ 1 1 1 ]]

Name: Cube.002
Translation:
[[ 0 0 0 ]]
Rotation:
[[ 0 0 0 1 ]]
Scale:
[[ 0.2 0.2 0.2 ]]

This data exactly corresponds to what your model data is providing on a base level.

Corresponding glTF2 Snippet of your model
   "nodes" : [
        {
            "mesh" : 0,
            "name" : "y-axis",
            "translation" : [
                0,
                0.550000011920929,
                0
            ]
        },
        {
            "mesh" : 1,
            "name" : "z-axis",
            "translation" : [
                0.5,
                0.550000011920929,
                0
            ]
        },
        {
            "mesh" : 2,
            "name" : "x-axis",
            "rotation" : [
                0.7071068286895752,
                0,
                0,
                0.7071068286895752
            ],
            "translation" : [
                0,
                0,
                0.550000011920929
            ]
        },
        {
            "mesh" : 3,
            "name" : "Cube.001",
            "translation" : [
                0.25,
                0.25,
                0.25
            ]
        },
        {
            "mesh" : 4,
            "name" : "Cube.002",
            "scale" : [
                0.20000000298023224,
                0.20000000298023224,
                0.20000000298023224
            ]
        }
    ],

Part of the problem

As someone with a little knowledge of how these transformations work, you might notice that your x, y, z axis objects don't have the transformations you would expect! This is because you transformed them on a Vertex level instead of in world space, e.g. edit mode in Blender in your case.

State/Current Conclusion

So with that said, the missing transformation data must lie in the vertex positions and I need further investigation to exactly find the point where the data differs from the Model.

@MrDiver
Copy link
Contributor

MrDiver commented Aug 28, 2021

Further Update

for(int i=0; i<model.meshCount; i++){
    Vector3 tl = model.bindPose[i].translation;
    Quaternion rot = model.bindPose[i].rotation;
    Vector3 scale = model.bindPose[i].scale;

    Matrix mt = MatrixTranslate(tl.x,tl.y,tl.z);
    Matrix mr = QuaternionToMatrix(rot);
    Matrix ms = MatrixScale(scale.x,scale.y,scale.z);
    Matrix t = MatrixMultiply(MatrixMultiply(mr,mt), ms);

    DrawMesh(model.meshes[i],model.materials[model.meshMaterial[i]],t);
}

Applying the bone transformations manually instead of relying on DrawModel results in the correct model display.

image

Problem

The bone transforms are not applied in the DrawModelEx function, which is called by DrawModel. The function only applies the world space transformation known for the model as seen in line 3047

raylib/src/models.c

Lines 3023 to 3050 in 936199d

void DrawModelEx(Model model, Vector3 position, Vector3 rotationAxis, float rotationAngle, Vector3 scale, Color tint)
{
// Calculate transformation matrix from function parameters
// Get transform matrix (rotation -> scale -> translation)
Matrix matScale = MatrixScale(scale.x, scale.y, scale.z);
Matrix matRotation = MatrixRotate(rotationAxis, rotationAngle*DEG2RAD);
Matrix matTranslation = MatrixTranslate(position.x, position.y, position.z);
Matrix matTransform = MatrixMultiply(MatrixMultiply(matScale, matRotation), matTranslation);
// Combine model transformation matrix (model.transform) with matrix generated by function parameters (matTransform)
model.transform = MatrixMultiply(model.transform, matTransform);
for (int i = 0; i < model.meshCount; i++)
{
Color color = model.materials[model.meshMaterial[i]].maps[MATERIAL_MAP_DIFFUSE].color;
Color colorTint = WHITE;
colorTint.r = (unsigned char)((((float)color.r/255.0)*((float)tint.r/255.0))*255.0f);
colorTint.g = (unsigned char)((((float)color.g/255.0)*((float)tint.g/255.0))*255.0f);
colorTint.b = (unsigned char)((((float)color.b/255.0)*((float)tint.b/255.0))*255.0f);
colorTint.a = (unsigned char)((((float)color.a/255.0)*((float)tint.a/255.0))*255.0f);
model.materials[model.meshMaterial[i]].maps[MATERIAL_MAP_DIFFUSE].color = colorTint;
DrawMesh(model.meshes[i], model.materials[model.meshMaterial[i]], model.transform);
model.materials[model.meshMaterial[i]].maps[MATERIAL_MAP_DIFFUSE].color = color;
}
}

Setting of the values in bones and bindPose happens here for glTF.

static void InitGLTFBones(Model* model, const cgltf_data* data)

Next Steps

I am not quite sure what the intended use of the bones and bindPose is. So I cannot really say that we should just add the bone transforms to the DrawModelEx because there are probably a lot of Models that don't use the bones.

raylib/src/raylib.h

Lines 375 to 388 in 936199d

typedef struct Model {
Matrix transform; // Local transform matrix
int meshCount; // Number of meshes
int materialCount; // Number of materials
Mesh *meshes; // Meshes array
Material *materials; // Materials array
int *meshMaterial; // Mesh material number
// Animation data
int boneCount; // Number of bones
BoneInfo *bones; // Bones information (skeleton)
Transform *bindPose; // Bones base transformation (pose)
} Model;

I am not really sure about all the different kinds of model loading, but I think that it might be a solution to add meshTransforms besides the meshCount and *meshes. This would only make sense if all the different model types deal with that, but I need to look further in to the process of the models to give an opinion about that.
The default value for Matrix* meshTransforms would be an array of size meshCount where all the elements would be the identity matrix.

I hope this helps a little for now~

--Appended 2 hours Later
This I think more difficult than thought before. I hope someone could explain the API a little. For now, I can't seem to find a way to get a clear distinction between the meshes and bones, which is unfortunate, but I am hopeful that we will find a solution. (Still learning how cgltf handles all of the Data)

@raysan5
Copy link
Owner

raysan5 commented Aug 28, 2021

@MrDiver Very nice! Thank you very much for investigating this issue! I think there are two separate issues here:

  • ISSUE 1: Mesh transformations are not properly applied. glTF exports mesh transform separated of vertex data and when mesh is loaded on raylib, that transformation is not applied.

    • SOLUTION 1: Static meshes should be exported with the transformations already applied to all vertex. This could be a requirement for raylib.
    • SOLUTION 2: Apply the mesh transformation provided to all vertex when loading the mesh.
  • ISSUE 2: Mesh transformations are provided as part of an animated model, using bones. In that case, model.bindPose defines the base bones transformations to be applied to all vertex data following every mesh animation info:

    mesh.boneIds -> the bone identifier per vertex, supporting up to 4 bones per vertex.
    mesh.boneWeights -> the weight of every bone over every vertex, supporting up to 4 bones per vertex.

    • SOLUTION 1: Actually, if model contains bones and a bindPose it should be applied on model loading (IQM applies it in LoadIQM() and InitGLTFBones() does the same in LoadGLTF()... or it should...).
    • SOLUTION 2: If bones and bindPose data is provided, apply it on DrawModelEx(). This is a very costly process done in CPU so, at this moment, it requires UpdateModelAnimation() for user control.

In any case, this issue and probably #1799 are both related to missing transformations that should be applied.

@MrDiver
Copy link
Contributor

MrDiver commented Aug 28, 2021

A new update on my investigations -> #1964

This is only solving a part of the problem, but it is at least half the job done for the mesh transforms. More explanation in the PR.

@chriscamacho
Copy link
Contributor Author

@630Studios please see #1966

@raysan5
Copy link
Owner

raysan5 commented Sep 5, 2021

@MrDiver Do you think this issue could be closed at this point?

@MrDiver
Copy link
Contributor

MrDiver commented Sep 5, 2021

If I'm not mistaken, everything related to this issue should be solved now !

@raysan5 raysan5 closed this as completed Sep 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants