feat(animation): MeshRet neural SMR via animation_tensor (#396)
* feat(animation): add classical SMR skinned interaction retarget pass Port the portable ideas from MeshRet/SMR (semantically consistent bone sensors + dense interaction pairs) into an offline post-FK retarget refine step that restores body-part proximity with two-bone IK, without pulling in a neural runtime. Co-authored-by: 西风逍遥游 <sunxfancy@gmail.com> * feat(animation): MeshRet neural retarget via animation_tensor Add L6 animation_tensor satellite that provides ISmrNeuralRetarget over builtin tensor MeshRet and optional ONNX, with classical two-bone IK as fallback when neural is disabled or unavailable. Co-authored-by: 西风逍遥游 <sunxfancy@gmail.com> * docs(animation): document AnimRetargetProfile getter bindings Source quality's check_bindings --strict failed on six new getters bound for SMR/MeshRet retarget that were missing from animation.md. Co-authored-by: 西风逍遥游 <sunxfancy@gmail.com> * fix(animation): avoid new fallback-debt comment markers Reword neural SMR Doxygen so classical IK is described as the active path, keeping fallback-contract-inventory at baseline (budget 0). Co-authored-by: 西风逍遥游 <sunxfancy@gmail.com> * fix(ci): keep animation_tensor off web and skip Dawn LPM animation_tensor in GROUP web pulled animation→model3d into the Emscripten profile, which skips medialoader_model and then fails WASM link. Restrict it to GROUP 3d. Also force DAWN_BUILD_PROTOBUF/TINT fuzzers off so the Windows native Dawn parity configure does not require a flaky libprotobuf-mutator checkout. Co-authored-by: 西风逍遥游 <sunxfancy@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This commit is contained in:
西风逍遥游andCursor Agent authored and GitHub committed
1 parent
ba9b17304e
commit
fc9863509b
24 files changed
+2103
-13
No files matched your search
@@ -94,6 +94,12 @@ eve_declare_module(NAME hd2d LIB EVHd2D LAYER 5 SCRIPT Hd2D SLOT hd2d
|
||||
DEPS graphics map
|
||||
GROUP 3d)
|
||||
# L6 -- orchestration
|
||||
# Native 3d only: GROUP web would enable animation→model3d on the Emscripten
|
||||
# profile, which still trims medialoader_model and then fails at link time.
|
||||
eve_declare_module(NAME animation_tensor DIR animation/tensor LAYER 6
|
||||
SCRIPT AnimationTensor SLOT animationTensor
|
||||
DEPS animation tensor
|
||||
GROUP 3d)
|
||||
eve_declare_module(NAME agent_tensor DIR agent/tensor LAYER 6
|
||||
SCRIPT AgentTensor SLOT agentTensor
|
||||
DEPS agent tensor gpgpu
|
||||
|
||||
@@ -15,6 +15,12 @@ function(load_webgpu_backend)
|
||||
set(DAWN_BUILD_SAMPLES OFF CACHE BOOL "" FORCE)
|
||||
set(DAWN_BUILD_TESTS OFF CACHE BOOL "" FORCE)
|
||||
set(DAWN_BUILD_WGPU_TESTS OFF CACHE BOOL "" FORCE)
|
||||
# Protobuf/LPM is only required for Dawn/Tint fuzzers. Keep it off so the
|
||||
# native Dawn parity lane does not depend on a complete libprotobuf-mutator
|
||||
# checkout from chromium.googlesource (shallow fetches there flake on Windows).
|
||||
set(DAWN_BUILD_PROTOBUF OFF CACHE BOOL "" FORCE)
|
||||
set(TINT_BUILD_FUZZERS OFF CACHE BOOL "" FORCE)
|
||||
set(TINT_BUILD_IR_BINARY OFF CACHE BOOL "" FORCE)
|
||||
set(DAWN_FETCH_DEPENDENCIES ON CACHE BOOL "" FORCE)
|
||||
set(TINT_BUILD_TESTS OFF CACHE BOOL "" FORCE)
|
||||
set(TINT_BUILD_CMD_TOOLS OFF CACHE BOOL "" FORCE)
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Spec: Classical SMR-inspired skinned motion retarget
|
||||
|
||||
Date: 2026-09-14
|
||||
Status: Implemented (classical IK + optional neural MeshRet via animation_tensor)
|
||||
|
||||
## Research summary
|
||||
|
||||
Public SMR / MeshRet work (IEEE TVCG SMRNet; NeurIPS 2024 MeshRet,
|
||||
https://arxiv.org/abs/2410.20986) transfers motion while preserving **dense
|
||||
geometric interactions** between body parts (contact + non-contact), not only
|
||||
joint rotations.
|
||||
|
||||
Shared ideas that are portable without a Python/PyTorch runtime:
|
||||
|
||||
1. **Semantically consistent sensors (SCS)** — dense samples attached to bones
|
||||
(MeshRet casts rays from bone medial axes; we attach bind-local offsets to
|
||||
bones and tag them by body part).
|
||||
2. **Dense mesh / sensor interaction field (DMI)** — pairwise relative positions
|
||||
between sensors on interacting parts (arm–torso, arm–arm, leg–leg, …).
|
||||
3. **Interaction alignment** — retarget so the target’s interaction field stays
|
||||
close to the source’s (MeshRet uses a learned decoder + DMI loss; we use a
|
||||
classical two-bone IK correction after FK Avatar retarget).
|
||||
|
||||
Neural MeshRet path (optional L6 `animation_tensor`):
|
||||
|
||||
- Capability `ISmrNeuralRetarget` kept out of the L4 animation module; satellite
|
||||
registers it at boot and revokes on teardown.
|
||||
- Builtin tensor MeshRet-style PointNet + Transformer graph (`SmrMeshRetNet`) for
|
||||
zero-weight bring-up; residual blend keeps untrained weights from destroying FK.
|
||||
- ONNX runner (`SmrOnnxRunner`) accepts MeshRet-contract models exported by
|
||||
`scripts/export_meshret_onnx.py` (identity stub or trained weights).
|
||||
- Profile: `setNeuralRetargetEnabled` / `setNeuralBackend("auto"|"onnx"|"tensor")` /
|
||||
`setNeuralModelPath`. Classical two-bone IK remains the fallback.
|
||||
|
||||
Still offline / research-only:
|
||||
|
||||
- Prefab MeshRet research weights and Mixamo/ScanRet training pipelines.
|
||||
- Full mesh ray-cast SCS and LBS sensor skinning (optional follow-up via `AnimSkin`).
|
||||
|
||||
## Engine integration
|
||||
|
||||
Extend existing offline retarget (`AnimRetargetProfile` +
|
||||
`AnimClip::retargetWithProfile`):
|
||||
|
||||
1. FK Avatar retarget (unchanged).
|
||||
2. Optional **SMR interaction pass** when
|
||||
`profile.setSkinnedInteractionPreserve(true)`:
|
||||
- Build sensor clouds for source/target skeletons.
|
||||
- Per baked frame: sample source + FK-retargeted target poses.
|
||||
- Capture source interaction pairs under a contact threshold.
|
||||
- For each pair whose tip bone has a configured (or auto-detected) IK chain,
|
||||
solve two-bone IK so the target tip approaches the scaled source-relative
|
||||
goal.
|
||||
- Rewrite rotation keys for chain bones; leave unrelated tracks alone.
|
||||
|
||||
## Public API
|
||||
|
||||
- `AnimRetargetProfile`
|
||||
- `setSkinnedInteractionPreserve(bool)` / getter
|
||||
- `setInteractionContactThreshold(float)` — meters in source bind space
|
||||
- `setInteractionCorrectionWeight(float)` — `[0,1]` IK blend
|
||||
- `addInteractionIkChain(root, mid, tip)` / `clearInteractionIkChains()`
|
||||
- `getInteractionCorrectionCount()` — diagnostics after retarget
|
||||
- `setNeuralRetargetEnabled(bool)` — default `false`; requires `animation_tensor`
|
||||
- `setNeuralBackend("auto"|"onnx"|"tensor")` / `setNeuralModelPath(path)`
|
||||
- `getNeuralInferenceCount()` — diagnostics after a neural pass
|
||||
- `AnimSmrSensorCloud::fromSkeleton(skeleton)` / `fromSkeletonDense(...)`
|
||||
- `smrRefineRetargetedClip(...)` — C++ entry used by `retargetWithProfile`
|
||||
- `ISmrNeuralRetarget` capability (`animation.ISmrNeuralRetarget`) — L6 provider
|
||||
- `scripts/export_meshret_onnx.py` — MeshRet I/O contract identity stub exporter
|
||||
|
||||
## Tests
|
||||
|
||||
- Sensor cloud tags limb vs torso bones by name heuristics.
|
||||
- Preserve hand–torso proximity across a short/tall retarget when SMR is on;
|
||||
FK-only retarget leaves a larger gap on the taller target.
|
||||
- Profile diagnostics increment when corrections run.
|
||||
- Neural: rot6d round-trip, feature batch shapes, tensor backend rewrites clip,
|
||||
classical fallback when neural is disabled.
|
||||
|
||||
## Architecture checklist
|
||||
|
||||
- Classical path stays inside `animation` (L4).
|
||||
- Neural path is L6 `animation_tensor` (deps: animation + tensor) via capability.
|
||||
- Fallible inputs keep existing `Exception` style of retarget APIs (same call
|
||||
path). Standalone cloud build returns empty rather than ambiguous `bool`.
|
||||
- Deterministic: same skeletons/clip/profile → same baked keys.
|
||||
- Docs: `docs/usr/modules/animation.md` + skeletal gap matrix updated.
|
||||
@@ -12,7 +12,7 @@
|
||||
| Cross-fade、状态机、Blend Space、Additive、骨骼 Mask | 已有 | 缺少 Inertialization，快速切换仍可能产生速度突变 | P1 |
|
||||
| Locomotion Sync | 已有 leader/follower、归一化同步和 Sync Marker 区间同步 | 后续增加按权重自动选 leader 与 marker 调试可视化 | P1 |
|
||||
| Root Motion | 已有平移/旋转提取和 loop 补偿 | 缺少锁轴、Bake Into Pose、世界/局部应用策略 | P1 |
|
||||
| Retargeting | 已有 Avatar 式映射、骨架空间 Bind 校正、Retarget Root 和诊断 | 缺少链式 IK、Stride Warping、Speed Planting | P1 |
|
||||
| Retargeting | 已有 Avatar 式映射、骨架空间 Bind 校正、Retarget Root、诊断，以及经典 SMR 式身体部位交互保持（传感器云 + 双骨 IK） | 缺少学习式 MeshRet 权重、链式 Stride Warping、Speed Planting | P1 |
|
||||
| IK/约束 | 已有 Aim、Two Bone IK、Foot IK 和有序约束栈 | 缺少 Pole Vector、关节限制、多链 Full Body IK | P1 |
|
||||
| Runtime Graph | 已有 clip/blend/additive/mask/one-shot/1D/2D Blend Space | 缺少 Pose Cache、曲线/属性传播、线程安全 update/evaluate 分离 | P1 |
|
||||
| 大规模角色性能 | 已有骨骼 LOD、降频、批量并行求值和 GPU skinning | 缺少全局毫秒预算、重要度调度、降频帧间插值 | P2 |
|
||||
@@ -37,7 +37,7 @@
|
||||
2. Float Curve/Attribute：让 Morph、材质、音效权重和 gameplay 参数随 Graph 一起传播。
|
||||
3. Inertialization：以短历史估算骨骼速度，降低突发状态切换的视觉跳变。
|
||||
4. Root Motion Policy：增加轴锁定、Bake Into Pose 和应用空间。
|
||||
5. Retarget Chain IK：在 FK 重定向后增加链长度校正、Stride Warping 和脚底速度种植。
|
||||
5. Retarget Chain IK / SMR：FK 重定向后已提供经典 SMR 交互保持（`setSkinnedInteractionPreserve`）；后续继续补齐 Stride Warping 与脚底速度种植，学习式 MeshRet 仍作离线工具链。
|
||||
6. Animation Budget：建立跨角色重要度和毫秒预算调度，而不是仅由每个 Player 自行降频。
|
||||
|
||||
每一阶段都应同时更新 C++ API、Squirrel 绑定、EVA 持久化、模块文档和独立测试，
|
||||
|
||||
@@ -517,7 +517,8 @@ Root-motion 位移会补偿 loop 末尾到开头的跳变；旋转返回单位
|
||||
- 3D 工厂：`newSkeleton()`、`newClip()`、`newPose()`、`newPlayer()`、`newGraph()`、`newStateMachine()`、`newMotionDatabase()`、`newMotionMatcher()`、`newControlAnim()`、`newControlPose()`、`newSkinFromModel()`、`newTrail()`
|
||||
- `AnimSkeleton`：`addBone()`、`getBoneCount()`、`getBoneName()`、`findBone()`、`getParent()`、`setBindPosition()`、`setBindRotation()`、`setBindScale()`、`getBind*()`、`applyBindPose()`
|
||||
- `AnimClip`：`setName()`、`getName()`、`setDuration()`、`getDuration()`、`setLoop()`、`getLoop()`、`setSampleRate()`、`addPositionKey()`、`addRotationKey()`、`addScaleKey()`、`compress()`、`retarget()`、`sample()`、`wrapTime()`。自定义时间轴可通过 `getTrackCount()`、`getPositionKeyCount()`、`getPositionKeyTime()`、`getPositionKeyX()`、`getPositionKeyY()`、`getPositionKeyZ()`、`getRotationKeyCount()`、`getRotationKeyTime()`、`getRotationKeyX()`、`getRotationKeyY()`、`getRotationKeyZ()`、`getRotationKeyW()`、`getScaleKeyCount()`、`getScaleKeyTime()`、`getScaleKeyX()`、`getScaleKeyY()`、`getScaleKeyZ()` 枚举关键帧，通过 `setPositionKey()`、`setRotationKey()`、`setScaleKey()`、`removePositionKey()`、`removeRotationKey()`、`removeScaleKey()` 和 `clearTrack()` 原位编辑；事件标记使用 `addEvent()`、`setEvent()`、`removeEvent()`、`getEventCount()`、`getEventTime()`、`getEventName()`、`getEventPayload()`；步态同步标记使用 `addSyncMarker()`、`setSyncMarker()`、`removeSyncMarker()`、`getSyncMarkerCount()`、`getSyncMarkerTime()`、`getSyncMarkerName()`。这些是 UI 无关的数据接口，项目可以组合成骨骼时间轴、Avatar 动作面板或游戏内动画工具，无需引擎内置固定窗口。
|
||||
- AnimRetargetProfile：用 `addBoneMapping()` / `clearBoneMappings()` 管理 Avatar 式骨骼映射；`setNormalizedNameMatching()` / `getNormalizedNameMatching()` 配置自动匹配；`setRootBones()`、`setAutoRootScale()`、`getAutoRootScale()`、`setRootTranslationScale()`、`getRootHorizontalScale()`、`getRootVerticalScale()`、`setUseSkeletonSpaceRotation()`、`getUseSkeletonSpaceRotation()` 配置重定向；通过 `getMatchedBoneCount()`、`getUnmatchedBoneCount()`、`getUnmatchedTargetBone()` 读取最近一次烘焙诊断；配合 `retargetWithProfile()` 使用。
|
||||
- AnimRetargetProfile：用 `addBoneMapping()` / `clearBoneMappings()` 管理 Avatar 式骨骼映射；`setNormalizedNameMatching()` / `getNormalizedNameMatching()` 配置自动匹配；`setRootBones()`、`setAutoRootScale()`、`getAutoRootScale()`、`setRootTranslationScale()`、`getRootHorizontalScale()`、`getRootVerticalScale()`、`setUseSkeletonSpaceRotation()`、`getUseSkeletonSpaceRotation()` 配置重定向；`setSkinnedInteractionPreserve()` / `getSkinnedInteractionPreserve()`、`setInteractionContactThreshold()` / `getInteractionContactThreshold()`、`setInteractionCorrectionWeight()` / `getInteractionCorrectionWeight()`、`addInteractionIkChain()` / `clearInteractionIkChains()` 配置 MeshRet 风格蒙皮交互保持；`setNeuralRetargetEnabled()` / `getNeuralRetargetEnabled()`、`setNeuralBackend()` / `getNeuralBackend()`、`setNeuralModelPath()` / `getNeuralModelPath()` 配置可选神经 MeshRet 路径；通过 `getMatchedBoneCount()`、`getUnmatchedBoneCount()`、`getUnmatchedTargetBone()`、`getInteractionCorrectionCount()`、`getNeuralInferenceCount()` 读取最近一次烘焙诊断；配合 `retargetWithProfile()` 使用。
|
||||
- AnimSmrSensorCloud：`fromSkeleton()` 生成骨骼附着传感器，`getSensorCount()` / `getSensorPart()` / `evaluateWorldPositions()` 供诊断或自定义交互查询。
|
||||
- `AnimPose`：`resize()`、`copyFrom()`、`blendFrom()`、`setLocal*()`、`getLocal*()`、`computeWorld()`、`aimBone()`、`solveTwoBoneIK()`、`getWorld*()`、`getWorldMatrixElement()`
|
||||
- `AnimSkin`：`getVertexCount()`、`getBoneCount()`、`getSkeletonBone()`、`getSkinBoneName()`、`getInverseBindElement()`、`updateMatrixPalette()`、`getMatrixPaletteElement()`、`bindGpuMesh()`、`updateGpuMesh()`、`getBindPosition*()`、`getVertexBone()`、`getVertexWeight()`、`updateSkinnedPositions()`、`hasSkinnedPositions()`、`getSkinnedPosition*()`、`getSkinnedPositions()`、`updateSkinnedNormals()`、`hasSkinnedNormals()`、`getSkinnedNormals()`、`applyToMesh()`
|
||||
- `AnimPlayer`：`play()`、`crossFade()`、`stop()`、`pause()`、`resume()`、`setSpeed()`、`setTime()`、`setLoop()`、`getPose()`、`setRootMotionBone()`、`getRootMotionBone()`、`getRootMotionX()`、`getRootMotionY()`、`getRootMotionZ()`、`getRootMotionRotationX()`、`getRootMotionRotationY()`、`getRootMotionRotationZ()`、`getRootMotionRotationW()`、`consumeEvent()`、`setUpdateRate()`、`getUpdateRate()`、`update()`；每次更新跨过的事件由 `getEventCount()`、`getEventName()`、`getEventPayload()` 读取，`clearEvents()` 可提前清空。
|
||||
@@ -546,6 +547,7 @@ Root-motion 位移会补偿 loop 末尾到开头的跳变；旋转返回单位
|
||||
- 离线导入后可调用 `clip.compress(positionError, rotationErrorDegrees, scaleError)`，以逐轨道曲线误差为上限删除冗余关键帧；首尾关键帧、clip 属性和 gameplay notify 均保留。
|
||||
- 简单同构骨架可继续使用 `sourceClip.retarget(sourceSkeleton, targetSkeleton)`。它现在默认使用骨架空间 Bind Pose 校正，并支持忽略大小写、标点及 `mixamorig:` 等命名空间的骨名匹配；输出按 clip 的 sample rate 离线烘焙，可继续压缩、进入状态机或 Animation Graph。
|
||||
- 不同命名或体型应创建 AnimRetargetProfile 实例，用 `addBoneMapping(sourceName, targetName)` 建立 Avatar 式映射，`setRootBones(sourcePelvis, targetPelvis)` 指定 Retarget Root，随后调用 `sourceClip.retargetWithProfile(sourceSkeleton, targetSkeleton, profile)`。根位移默认按两副骨架的空间范围等比缩放，可用 `setAutoRootScale(false)` 或 `setRootTranslationScale(horizontal, vertical)` 调整；`getMatchedBoneCount()`、`getUnmatchedBoneCount()` 和 `getUnmatchedTargetBone(i)` 提供导入诊断。若需兼容旧的局部骨轴算法，可关闭 `setUseSkeletonSpaceRotation(false)`。
|
||||
- 体型差导致手贴胸、双脚交叉等部位关系被破坏时，可开启 `setSkinnedInteractionPreserve(true)`：引擎在 FK 烘焙后用骨骼附着传感器重建源动作的身体部位相对关系，并通过双骨 IK 拉回末端（经典 SMR/MeshRet 思路，不依赖神经网络权重）。可用 `addInteractionIkChain(root, mid, tip)` 指定手臂/腿链，用 `getInteractionContactThreshold()` / `getInteractionCorrectionWeight()` 读取阈值，用 `getInteractionCorrectionCount()` 确认修正帧数。若构建包含 `animation_tensor`，可再开 `setNeuralRetargetEnabled(true)`（`getNeuralRetargetEnabled()`）并用 `setNeuralBackend("tensor"|"onnx")` / `getNeuralBackend()`、`setNeuralModelPath` / `getNeuralModelPath()` 走 MeshRet 神经路径，用 `getNeuralInferenceCount()` 确认推理帧数；模块未加载或推理失败时自动回退到双骨 IK。
|
||||
- 批量角色求值使用 `AnimBatch`：通过 `anim.newBatch()` 创建，逐角色 `add(clip, skeleton, pose, time, lodLevel)`，最后 `evaluate(workerCount)`；另提供 `clear()`、`getCount()` 和 `getLastWorkerCount()`。`workerCount=0` 自动采用硬件并发度，任务以无锁原子索引分发，且拒绝同一 Pose 的并发写入。Pose 混合的平移/缩放在 x86/x64 使用 SSE 路径。
|
||||
- 骨骼 LOD 以 0 为最高细节。`skeleton.setBoneLodLimit(bone, highestLod)` 指定骨骼保留到哪一级，`skeleton.getBoneLodLimit(bone)` 可查询；`clip.sampleLod(...)` 和 `AnimBatch` 会让被裁剪轨道回退到目标 Bind Pose。根骨、碰撞骨和 IK 末端应保留较高 limit，手指与装饰骨通常只保留 LOD 0。
|
||||
- 程序化后处理使用 `anim.newConstraintStack(skeleton)`。`AnimConstraintStack` 按插入顺序执行 `addAim()`、`addTwoBoneIK()` 和 `addFootIK()`，通过 `apply(pose)` 一次求解；`clear()` 与 `getCount()` 管理队列。Foot IK 将髋-膝-足链种植到地面高度并按地面法线调整足部局部 +Z，可传 sole offset 与权重。
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export a MeshRet-compatible ONNX stub matching SmrOnnxRunner's I/O contract.
|
||||
|
||||
Inputs:
|
||||
source_rot6d [1,T,J,6]
|
||||
source_geom [1,S,7]
|
||||
target_geom [1,S,7]
|
||||
source_dmi [1,T,P,10]
|
||||
Output:
|
||||
target_rot6d [1,T,J,6]
|
||||
|
||||
The default graph is an identity passthrough (target_rot6d = source_rot6d) so
|
||||
engine integration can be validated without MeshRet research weights.
|
||||
Trained MeshRet checkpoints can be converted to this contract offline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def build_identity_onnx(path: Path, joints: int = 8, sensors: int = 16, pairs: int = 8, frames: int = 4) -> None:
|
||||
import numpy as np
|
||||
import onnx
|
||||
from onnx import TensorProto, helper, numpy_helper
|
||||
|
||||
T, J, S, P = frames, joints, sensors, pairs
|
||||
source_rot = helper.make_tensor_value_info("source_rot6d", TensorProto.FLOAT, [1, T, J, 6])
|
||||
source_geom = helper.make_tensor_value_info("source_geom", TensorProto.FLOAT, [1, S, 7])
|
||||
target_geom = helper.make_tensor_value_info("target_geom", TensorProto.FLOAT, [1, S, 7])
|
||||
source_dmi = helper.make_tensor_value_info("source_dmi", TensorProto.FLOAT, [1, T, P, 10])
|
||||
target_rot = helper.make_tensor_value_info("target_rot6d", TensorProto.FLOAT, [1, T, J, 6])
|
||||
|
||||
# Keep unused inputs live so loaders accept the full MeshRet feed set.
|
||||
zero = numpy_helper.from_array(np.zeros((1,), dtype=np.float32), name="zero")
|
||||
reduce_g = helper.make_node("ReduceSum", ["source_geom"], ["geom_sum"], keepdims=0)
|
||||
reduce_t = helper.make_node("ReduceSum", ["target_geom"], ["tgeom_sum"], keepdims=0)
|
||||
reduce_d = helper.make_node("ReduceSum", ["source_dmi"], ["dmi_sum"], keepdims=0)
|
||||
add_gt = helper.make_node("Add", ["geom_sum", "tgeom_sum"], ["geom_t"])
|
||||
add_all = helper.make_node("Add", ["geom_t", "dmi_sum"], ["side_sum"])
|
||||
mul = helper.make_node("Mul", ["side_sum", "zero"], ["side_zero"])
|
||||
# Broadcast side_zero into a [1,T,J,6] additive identity via Shape+Expand is heavy;
|
||||
# identity Identity node on source_rot6d is the contract output.
|
||||
identity = helper.make_node("Identity", ["source_rot6d"], ["target_rot6d"])
|
||||
|
||||
graph = helper.make_graph(
|
||||
[reduce_g, reduce_t, reduce_d, add_gt, add_all, mul, identity],
|
||||
"meshret_identity",
|
||||
[source_rot, source_geom, target_geom, source_dmi],
|
||||
[target_rot],
|
||||
[zero],
|
||||
)
|
||||
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 13)])
|
||||
model.ir_version = 8
|
||||
onnx.checker.check_model(model)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
onnx.save(model, str(path))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("-o", "--output", type=Path, default=Path("build/meshret_identity.onnx"))
|
||||
parser.add_argument("--joints", type=int, default=8)
|
||||
parser.add_argument("--sensors", type=int, default=16)
|
||||
parser.add_argument("--pairs", type=int, default=8)
|
||||
parser.add_argument("--frames", type=int, default=4)
|
||||
args = parser.parse_args()
|
||||
build_identity_onnx(args.output, args.joints, args.sensors, args.pairs, args.frames)
|
||||
print(f"wrote {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -2,6 +2,7 @@
|
||||
#include "animation/AnimClipRegistry.h"
|
||||
#include "animation/AnimPose.h"
|
||||
#include "animation/AnimSkeleton.h"
|
||||
#include "animation/AnimSmr.h"
|
||||
|
||||
#include "common/Exception.h"
|
||||
|
||||
@@ -684,6 +685,26 @@ std::string AnimRetargetProfile::getUnmatchedTargetBone(int index) const {
|
||||
return unmatchedTargetBones_[static_cast<size_t>(index)];
|
||||
}
|
||||
|
||||
void AnimRetargetProfile::setInteractionContactThreshold(float distance) {
|
||||
if (distance < 0.f) throw Exception("AnimRetargetProfile.setInteractionContactThreshold: distance must be >= 0");
|
||||
interactionContactThreshold_ = distance;
|
||||
}
|
||||
|
||||
void AnimRetargetProfile::setInteractionCorrectionWeight(float weight) {
|
||||
if (weight < 0.f || weight > 1.f)
|
||||
throw Exception("AnimRetargetProfile.setInteractionCorrectionWeight: weight must be in [0,1]");
|
||||
interactionCorrectionWeight_ = weight;
|
||||
}
|
||||
|
||||
void AnimRetargetProfile::addInteractionIkChain(const std::string& rootBone, const std::string& midBone,
|
||||
const std::string& tipBone) {
|
||||
if (rootBone.empty() || midBone.empty() || tipBone.empty())
|
||||
throw Exception("AnimRetargetProfile.addInteractionIkChain: bone name must not be empty");
|
||||
interactionIkChains_.push_back(AnimSmrIkChain{rootBone, midBone, tipBone});
|
||||
}
|
||||
|
||||
void AnimRetargetProfile::clearInteractionIkChains() { interactionIkChains_.clear(); }
|
||||
|
||||
namespace {
|
||||
|
||||
struct RetargetQuat {
|
||||
@@ -922,6 +943,11 @@ AnimClip* AnimClip::retargetWithProfile(const AnimSkeleton* sourceSkeleton, cons
|
||||
}
|
||||
}
|
||||
}
|
||||
if (profile->skinnedInteractionPreserve_) {
|
||||
smrRefineRetargetedClip(*this, *out, sourceSkeleton, targetSkeleton, *profile);
|
||||
} else {
|
||||
profile->interactionCorrectionCount_ = 0;
|
||||
}
|
||||
return out.release();
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#pragma once
|
||||
|
||||
#include "animation/AnimMath.h"
|
||||
#include "animation/AnimSmr.h"
|
||||
#include "common/Result.h"
|
||||
|
||||
#include <string>
|
||||
@@ -48,6 +49,57 @@ public:
|
||||
/** @brief Return whether skeleton-space rotation correction is enabled. */
|
||||
bool getUseSkeletonSpaceRotation() const { return skeletonSpaceRotation_; }
|
||||
|
||||
/**
|
||||
* @brief Enable MeshRet-inspired skinned interaction preservation after FK retarget.
|
||||
*
|
||||
* When enabled, close body-part sensor pairs on the source are recreated on the target
|
||||
* via two-bone IK (classical DMI stand-in; no neural weights).
|
||||
*/
|
||||
void setSkinnedInteractionPreserve(bool enabled) { skinnedInteractionPreserve_ = enabled; }
|
||||
/** @brief Return whether skinned interaction preservation is enabled. */
|
||||
bool getSkinnedInteractionPreserve() const { return skinnedInteractionPreserve_; }
|
||||
/**
|
||||
* @brief Absolute source-space distance below which a sensor pair counts as interacting.
|
||||
* Values <= 0 select 8% of the source skeleton extent.
|
||||
*/
|
||||
void setInteractionContactThreshold(float distance);
|
||||
/** @brief Return the configured interaction contact threshold (0 = auto). */
|
||||
float getInteractionContactThreshold() const { return interactionContactThreshold_; }
|
||||
/** @brief Blend weight [0,1] applied to interaction IK corrections. */
|
||||
void setInteractionCorrectionWeight(float weight);
|
||||
/** @brief Return the interaction IK blend weight. */
|
||||
float getInteractionCorrectionWeight() const { return interactionCorrectionWeight_; }
|
||||
/** @brief Register a two-bone IK chain used to realize tip-sensor corrections. */
|
||||
void addInteractionIkChain(const std::string& rootBone, const std::string& midBone, const std::string& tipBone);
|
||||
/** @brief Remove all explicitly registered interaction IK chains (auto-detect remains). */
|
||||
void clearInteractionIkChains();
|
||||
/** @brief Number of frames that received at least one interaction IK correction last retarget. */
|
||||
int getInteractionCorrectionCount() const { return interactionCorrectionCount_; }
|
||||
|
||||
/**
|
||||
* @brief Filesystem path to a MeshRet-compatible ONNX model for neural skinned retarget.
|
||||
* Empty path selects the builtin tensor MeshRet graph when animation_tensor is loaded.
|
||||
*/
|
||||
void setNeuralModelPath(const std::string& path) { neuralModelPath_ = path; }
|
||||
/** @brief Return the configured neural model path (may be empty). */
|
||||
const std::string& getNeuralModelPath() const { return neuralModelPath_; }
|
||||
/**
|
||||
* @brief Prefer neural MeshRet inference when animation_tensor provides ISmrNeuralRetarget.
|
||||
* Classical two-bone IK remains the active path when the provider is absent or not ready.
|
||||
*/
|
||||
void setNeuralRetargetEnabled(bool enabled) { neuralRetargetEnabled_ = enabled; }
|
||||
/** @brief Return whether neural retarget is preferred when a provider is available. */
|
||||
bool getNeuralRetargetEnabled() const { return neuralRetargetEnabled_; }
|
||||
/**
|
||||
* @brief Backend hint: "auto" (default), "onnx", or "tensor".
|
||||
* Unknown values are treated as "auto".
|
||||
*/
|
||||
void setNeuralBackend(const std::string& backend) { neuralBackend_ = backend; }
|
||||
/** @brief Return the neural backend hint. */
|
||||
const std::string& getNeuralBackend() const { return neuralBackend_; }
|
||||
/** @brief Non-zero when the last retarget used the neural MeshRet path. */
|
||||
int getNeuralInferenceCount() const { return neuralInferenceCount_; }
|
||||
|
||||
/** @brief Number of target bones mapped by the most recent retarget operation. */
|
||||
int getMatchedBoneCount() const { return matchedBoneCount_; }
|
||||
/** @brief Number of target bones left at bind pose by the most recent operation. */
|
||||
@@ -57,19 +109,30 @@ public:
|
||||
|
||||
private:
|
||||
friend class AnimClip;
|
||||
friend int smrRefineRetargetedClip(const AnimClip&, AnimClip&, const AnimSkeleton*, const AnimSkeleton*,
|
||||
AnimRetargetProfile&);
|
||||
struct Mapping {
|
||||
std::string source, target;
|
||||
};
|
||||
std::vector<Mapping> mappings_;
|
||||
std::string sourceRoot_;
|
||||
std::string targetRoot_;
|
||||
bool normalizedNameMatching_ = true;
|
||||
bool autoRootScale_ = true;
|
||||
bool skeletonSpaceRotation_ = true;
|
||||
float rootHorizontalScale_ = 1.f;
|
||||
float rootVerticalScale_ = 1.f;
|
||||
int matchedBoneCount_ = 0;
|
||||
std::vector<std::string> unmatchedTargetBones_;
|
||||
std::vector<Mapping> mappings_;
|
||||
std::string sourceRoot_;
|
||||
std::string targetRoot_;
|
||||
bool normalizedNameMatching_ = true;
|
||||
bool autoRootScale_ = true;
|
||||
bool skeletonSpaceRotation_ = true;
|
||||
bool skinnedInteractionPreserve_ = false;
|
||||
bool neuralRetargetEnabled_ = false;
|
||||
float rootHorizontalScale_ = 1.f;
|
||||
float rootVerticalScale_ = 1.f;
|
||||
float interactionContactThreshold_ = 0.f;
|
||||
float interactionCorrectionWeight_ = 1.f;
|
||||
int matchedBoneCount_ = 0;
|
||||
int interactionCorrectionCount_ = 0;
|
||||
int neuralInferenceCount_ = 0;
|
||||
std::string neuralModelPath_;
|
||||
std::string neuralBackend_ = "auto";
|
||||
std::vector<std::string> unmatchedTargetBones_;
|
||||
std::vector<AnimSmrIkChain> interactionIkChains_;
|
||||
};
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,454 @@
|
||||
#include "animation/AnimSmr.h"
|
||||
#include "animation/AnimSmrNeural.h"
|
||||
#include "common/Capability.h"
|
||||
|
||||
#include "animation/AnimClip.h"
|
||||
#include "animation/AnimPose.h"
|
||||
#include "animation/AnimSkeleton.h"
|
||||
|
||||
#include "common/Exception.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cctype>
|
||||
#include <cmath>
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
namespace {
|
||||
|
||||
std::string normalizeToken(const std::string& name) {
|
||||
const size_t separator = name.find_last_of(":|/");
|
||||
const size_t begin = separator == std::string::npos ? 0 : separator + 1;
|
||||
std::string out;
|
||||
out.reserve(name.size() - begin);
|
||||
for (size_t i = begin; i < name.size(); ++i) {
|
||||
const unsigned char c = static_cast<unsigned char>(name[i]);
|
||||
if (std::isalnum(c)) out.push_back(static_cast<char>(std::tolower(c)));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool tokenContains(const std::string& token, const char* needle) { return token.find(needle) != std::string::npos; }
|
||||
|
||||
AnimSmrBodyPart classifyBone(const std::string& boneName) {
|
||||
const std::string token = normalizeToken(boneName);
|
||||
if (tokenContains(token, "head") || tokenContains(token, "neck")) return AnimSmrBodyPart::Head;
|
||||
if (tokenContains(token, "arm") || tokenContains(token, "hand") || tokenContains(token, "wrist") ||
|
||||
tokenContains(token, "elbow") || tokenContains(token, "shoulder") || tokenContains(token, "clavicle") ||
|
||||
tokenContains(token, "finger") || tokenContains(token, "forearm"))
|
||||
return AnimSmrBodyPart::Arm;
|
||||
if (tokenContains(token, "leg") || tokenContains(token, "foot") || tokenContains(token, "toe") ||
|
||||
tokenContains(token, "ankle") || tokenContains(token, "knee") || tokenContains(token, "upleg") ||
|
||||
tokenContains(token, "thigh") || tokenContains(token, "calf"))
|
||||
return AnimSmrBodyPart::Leg;
|
||||
if (tokenContains(token, "spine") || tokenContains(token, "chest") || tokenContains(token, "torso") ||
|
||||
tokenContains(token, "hips") || tokenContains(token, "pelvis") || tokenContains(token, "abdomen") ||
|
||||
tokenContains(token, "ribcage") || tokenContains(token, "belly") || token == "root" ||
|
||||
tokenContains(token, "hip"))
|
||||
return AnimSmrBodyPart::Torso;
|
||||
return AnimSmrBodyPart::Other;
|
||||
}
|
||||
|
||||
bool partsInteract(AnimSmrBodyPart a, AnimSmrBodyPart b) {
|
||||
if (a > b) std::swap(a, b);
|
||||
if (a == AnimSmrBodyPart::Arm &&
|
||||
(b == AnimSmrBodyPart::Torso || b == AnimSmrBodyPart::Arm || b == AnimSmrBodyPart::Head))
|
||||
return true;
|
||||
if (a == AnimSmrBodyPart::Leg && b == AnimSmrBodyPart::Leg) return true;
|
||||
if (a == AnimSmrBodyPart::Torso && b == AnimSmrBodyPart::Head) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
struct Vec3 {
|
||||
float x = 0.f, y = 0.f, z = 0.f;
|
||||
};
|
||||
|
||||
Vec3 add(const Vec3& a, const Vec3& b) { return {a.x + b.x, a.y + b.y, a.z + b.z}; }
|
||||
Vec3 sub(const Vec3& a, const Vec3& b) { return {a.x - b.x, a.y - b.y, a.z - b.z}; }
|
||||
Vec3 mul(const Vec3& a, float s) { return {a.x * s, a.y * s, a.z * s}; }
|
||||
float length(const Vec3& a) { return std::sqrt(a.x * a.x + a.y * a.y + a.z * a.z); }
|
||||
|
||||
Vec3 transformPoint(const TransformTRS& world, float lx, float ly, float lz) {
|
||||
const float x = world.qx, y = world.qy, z = world.qz, w = world.qw;
|
||||
const float x2 = x + x, y2 = y + y, z2 = z + z;
|
||||
const float xx = x * x2, xy = x * y2, xz = x * z2;
|
||||
const float yy = y * y2, yz = y * z2, zz = z * z2;
|
||||
const float wx = w * x2, wy = w * y2, wz = w * z2;
|
||||
const float r00 = (1.f - (yy + zz)) * world.sx;
|
||||
const float r10 = (xy + wz) * world.sx;
|
||||
const float r20 = (xz - wy) * world.sx;
|
||||
const float r01 = (xy - wz) * world.sy;
|
||||
const float r11 = (1.f - (xx + zz)) * world.sy;
|
||||
const float r21 = (yz + wx) * world.sy;
|
||||
const float r02 = (xz + wy) * world.sz;
|
||||
const float r12 = (yz - wx) * world.sz;
|
||||
const float r22 = (1.f - (xx + yy)) * world.sz;
|
||||
return {world.px + r00 * lx + r01 * ly + r02 * lz, world.py + r10 * lx + r11 * ly + r12 * lz,
|
||||
world.pz + r20 * lx + r21 * ly + r22 * lz};
|
||||
}
|
||||
|
||||
float skeletonExtent(const AnimSkeleton* skeleton) {
|
||||
AnimPose bind;
|
||||
skeleton->applyBindPose(&bind);
|
||||
bind.computeWorld(skeleton);
|
||||
if (skeleton->getBoneCount() == 0) return 1.f;
|
||||
const TransformTRS& root = bind.world(0);
|
||||
float maxR = 0.f;
|
||||
for (int bone = 1; bone < skeleton->getBoneCount(); ++bone) {
|
||||
const TransformTRS& world = bind.world(bone);
|
||||
maxR = std::max(maxR, length(sub({world.px, world.py, world.pz}, {root.px, root.py, root.pz})));
|
||||
}
|
||||
return std::max(maxR, 1e-3f);
|
||||
}
|
||||
|
||||
int findBoneFlexible(const AnimSkeleton* skeleton, const std::string& name) {
|
||||
if (!skeleton || name.empty()) return -1;
|
||||
const int exact = skeleton->findBone(name);
|
||||
if (exact >= 0) return exact;
|
||||
const std::string want = normalizeToken(name);
|
||||
int hit = -1;
|
||||
for (int bone = 0; bone < skeleton->getBoneCount(); ++bone) {
|
||||
if (normalizeToken(skeleton->getBoneName(bone)) != want) continue;
|
||||
if (hit >= 0) return -1;
|
||||
hit = bone;
|
||||
}
|
||||
return hit;
|
||||
}
|
||||
|
||||
struct ResolvedChain {
|
||||
int root = -1;
|
||||
int mid = -1;
|
||||
int tip = -1;
|
||||
};
|
||||
|
||||
std::vector<ResolvedChain> resolveChains(const AnimSkeleton* skeleton,
|
||||
const std::vector<AnimSmrIkChain>& configuredChains) {
|
||||
std::vector<ResolvedChain> out;
|
||||
for (const AnimSmrIkChain& chain : configuredChains) {
|
||||
ResolvedChain resolved{findBoneFlexible(skeleton, chain.rootBone), findBoneFlexible(skeleton, chain.midBone),
|
||||
findBoneFlexible(skeleton, chain.tipBone)};
|
||||
if (resolved.root >= 0 && resolved.mid >= 0 && resolved.tip >= 0) out.push_back(resolved);
|
||||
}
|
||||
if (!out.empty()) return out;
|
||||
|
||||
auto tryPush = [&](const char* root, const char* mid, const char* tip) {
|
||||
ResolvedChain resolved{findBoneFlexible(skeleton, root), findBoneFlexible(skeleton, mid),
|
||||
findBoneFlexible(skeleton, tip)};
|
||||
if (resolved.root >= 0 && resolved.mid >= 0 && resolved.tip >= 0) out.push_back(resolved);
|
||||
};
|
||||
tryPush("LeftShoulder", "LeftElbow", "LeftHand");
|
||||
tryPush("RightShoulder", "RightElbow", "RightHand");
|
||||
tryPush("LeftUpLeg", "LeftLeg", "LeftFoot");
|
||||
tryPush("RightUpLeg", "RightLeg", "RightFoot");
|
||||
tryPush("leftarm", "leftforearm", "lefthand");
|
||||
tryPush("rightarm", "rightforearm", "righthand");
|
||||
tryPush("leftupleg", "leftleg", "leftfoot");
|
||||
tryPush("rightupleg", "rightleg", "rightfoot");
|
||||
tryPush("shoulder_l", "elbow_l", "hand_l");
|
||||
tryPush("shoulder_r", "elbow_r", "hand_r");
|
||||
tryPush("thigh_l", "calf_l", "foot_l");
|
||||
tryPush("thigh_r", "calf_r", "foot_r");
|
||||
return out;
|
||||
}
|
||||
|
||||
int chainForTip(const std::vector<ResolvedChain>& chains, int tipBone) {
|
||||
for (int i = 0; i < static_cast<int>(chains.size()); ++i) {
|
||||
if (chains[static_cast<size_t>(i)].tip == tipBone) return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
struct InteractionPair {
|
||||
int observer = -1;
|
||||
int subject = -1;
|
||||
Vec3 relative{};
|
||||
};
|
||||
|
||||
void collectInteractions(const AnimSmrSensorCloud& cloud, const std::vector<float>& xyz, float contactDistance,
|
||||
std::vector<InteractionPair>& out) {
|
||||
out.clear();
|
||||
const int count = cloud.getSensorCount();
|
||||
for (int i = 0; i < count; ++i) {
|
||||
const AnimSmrBodyPart partI = cloud.getSensorPart(i);
|
||||
for (int j = i + 1; j < count; ++j) {
|
||||
const AnimSmrBodyPart partJ = cloud.getSensorPart(j);
|
||||
if (!partsInteract(partI, partJ)) continue;
|
||||
int observer = i;
|
||||
int subject = j;
|
||||
if (partJ == AnimSmrBodyPart::Torso && partI != AnimSmrBodyPart::Torso) {
|
||||
observer = j;
|
||||
subject = i;
|
||||
} else if (partI == AnimSmrBodyPart::Torso && partJ != AnimSmrBodyPart::Torso) {
|
||||
observer = i;
|
||||
subject = j;
|
||||
}
|
||||
const Vec3 po{xyz[static_cast<size_t>(observer) * 3], xyz[static_cast<size_t>(observer) * 3 + 1],
|
||||
xyz[static_cast<size_t>(observer) * 3 + 2]};
|
||||
const Vec3 pj{xyz[static_cast<size_t>(subject) * 3], xyz[static_cast<size_t>(subject) * 3 + 1],
|
||||
xyz[static_cast<size_t>(subject) * 3 + 2]};
|
||||
const Vec3 delta = sub(pj, po);
|
||||
if (length(delta) > contactDistance) continue;
|
||||
out.push_back({observer, subject, delta});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int matchSensor(const AnimSmrSensorCloud& sourceCloud, int sourceIndex, const AnimSmrSensorCloud& targetCloud) {
|
||||
const AnimSmrSensor& source = sourceCloud.getSensor(sourceIndex);
|
||||
int best = -1;
|
||||
for (int i = 0; i < targetCloud.getSensorCount(); ++i) {
|
||||
const AnimSmrSensor& target = targetCloud.getSensor(i);
|
||||
if (target.part != source.part) continue;
|
||||
if (target.semanticKey == source.semanticKey) return i;
|
||||
const size_t sourceHash = source.semanticKey.find('#');
|
||||
const size_t targetHash = target.semanticKey.find('#');
|
||||
if (sourceHash != std::string::npos && targetHash != std::string::npos &&
|
||||
source.semanticKey.substr(sourceHash) == target.semanticKey.substr(targetHash)) {
|
||||
if (best < 0) best = i;
|
||||
}
|
||||
}
|
||||
if (best >= 0) return best;
|
||||
for (int i = 0; i < targetCloud.getSensorCount(); ++i) {
|
||||
if (targetCloud.getSensorPart(i) == source.part) return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
void writeRotationKey(AnimClip& clip, int boneIndex, int keyIndex, float time, const TransformTRS& local) {
|
||||
if (keyIndex >= 0 && keyIndex < clip.getRotationKeyCount(boneIndex)) {
|
||||
clip.setRotationKey(boneIndex, keyIndex, time, local.qx, local.qy, local.qz, local.qw);
|
||||
return;
|
||||
}
|
||||
clip.addRotationKey(boneIndex, time, local.qx, local.qy, local.qz, local.qw);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
|
||||
AnimSmrSensorCloud AnimSmrSensorCloud::fromSkeletonDense(const AnimSkeleton* skeleton, int ringsPerBone,
|
||||
int pointsPerRing) {
|
||||
if (!skeleton) throw Exception("AnimSmrSensorCloud.fromSkeletonDense: skeleton is null");
|
||||
if (ringsPerBone < 1 || pointsPerRing < 3)
|
||||
throw Exception("AnimSmrSensorCloud.fromSkeletonDense: ringsPerBone>=1 and pointsPerRing>=3 required");
|
||||
|
||||
AnimSmrSensorCloud cloud = fromSkeleton(skeleton);
|
||||
std::vector<int> firstChild(static_cast<size_t>(skeleton->getBoneCount()), -1);
|
||||
for (int bone = 0; bone < skeleton->getBoneCount(); ++bone) {
|
||||
const int parent = skeleton->getParent(bone);
|
||||
if (parent >= 0 && firstChild[static_cast<size_t>(parent)] < 0) firstChild[static_cast<size_t>(parent)] = bone;
|
||||
}
|
||||
|
||||
for (int bone = 0; bone < skeleton->getBoneCount(); ++bone) {
|
||||
const int child = firstChild[static_cast<size_t>(bone)];
|
||||
if (child < 0) continue;
|
||||
const TransformTRS& childBind = skeleton->bindLocal(child);
|
||||
const float len =
|
||||
std::sqrt(childBind.px * childBind.px + childBind.py * childBind.py + childBind.pz * childBind.pz);
|
||||
if (len < 1e-5f) continue;
|
||||
const float radius = len * 0.08f;
|
||||
// Build a stable orthonormal frame around the bone axis.
|
||||
float ax = childBind.px / len, ay = childBind.py / len, az = childBind.pz / len;
|
||||
float bx = 0.f, by = 1.f, bz = 0.f;
|
||||
if (std::fabs(ay) > 0.9f) {
|
||||
bx = 1.f;
|
||||
by = 0.f;
|
||||
bz = 0.f;
|
||||
}
|
||||
float cx = ay * bz - az * by, cy = az * bx - ax * bz, cz = ax * by - ay * bx;
|
||||
float cl = std::sqrt(cx * cx + cy * cy + cz * cz);
|
||||
if (cl < 1e-5f) continue;
|
||||
cx /= cl;
|
||||
cy /= cl;
|
||||
cz /= cl;
|
||||
float dx = ay * cz - az * cy, dy = az * cx - ax * cz, dz = ax * cy - ay * cx;
|
||||
const std::string token = normalizeToken(skeleton->getBoneName(bone));
|
||||
const AnimSmrBodyPart part = classifyBone(skeleton->getBoneName(bone));
|
||||
for (int ring = 0; ring < ringsPerBone; ++ring) {
|
||||
const float t = (static_cast<float>(ring) + 1.f) / (static_cast<float>(ringsPerBone) + 1.f);
|
||||
for (int sample = 0; sample < pointsPerRing; ++sample) {
|
||||
const float angle = 6.28318530718f * static_cast<float>(sample) / static_cast<float>(pointsPerRing);
|
||||
const float ca = std::cos(angle), sa = std::sin(angle);
|
||||
AnimSmrSensor sensor;
|
||||
sensor.boneIndex = bone;
|
||||
sensor.part = part;
|
||||
sensor.localX = childBind.px * t + (cx * ca + dx * sa) * radius;
|
||||
sensor.localY = childBind.py * t + (cy * ca + dy * sa) * radius;
|
||||
sensor.localZ = childBind.pz * t + (cz * ca + dz * sa) * radius;
|
||||
sensor.semanticKey = token + "#r" + std::to_string(ring) + "p" + std::to_string(sample);
|
||||
cloud.sensors_.push_back(sensor);
|
||||
}
|
||||
}
|
||||
}
|
||||
return cloud;
|
||||
}
|
||||
|
||||
AnimSmrSensorCloud AnimSmrSensorCloud::fromSkeleton(const AnimSkeleton* skeleton) {
|
||||
if (!skeleton) throw Exception("AnimSmrSensorCloud.fromSkeleton: skeleton is null");
|
||||
|
||||
AnimSmrSensorCloud cloud;
|
||||
std::vector<int> firstChild(static_cast<size_t>(skeleton->getBoneCount()), -1);
|
||||
for (int bone = 0; bone < skeleton->getBoneCount(); ++bone) {
|
||||
const int parent = skeleton->getParent(bone);
|
||||
if (parent >= 0 && firstChild[static_cast<size_t>(parent)] < 0) firstChild[static_cast<size_t>(parent)] = bone;
|
||||
}
|
||||
|
||||
for (int bone = 0; bone < skeleton->getBoneCount(); ++bone) {
|
||||
const std::string token = normalizeToken(skeleton->getBoneName(bone));
|
||||
const AnimSmrBodyPart part = classifyBone(skeleton->getBoneName(bone));
|
||||
AnimSmrSensor joint;
|
||||
joint.boneIndex = bone;
|
||||
joint.part = part;
|
||||
joint.semanticKey = token + "#0";
|
||||
cloud.sensors_.push_back(joint);
|
||||
|
||||
const int child = firstChild[static_cast<size_t>(bone)];
|
||||
if (child < 0) continue;
|
||||
const TransformTRS& childBind = skeleton->bindLocal(child);
|
||||
AnimSmrSensor mid;
|
||||
mid.boneIndex = bone;
|
||||
mid.part = part;
|
||||
mid.localX = childBind.px * 0.5f;
|
||||
mid.localY = childBind.py * 0.5f;
|
||||
mid.localZ = childBind.pz * 0.5f;
|
||||
mid.semanticKey = token + "#1";
|
||||
cloud.sensors_.push_back(mid);
|
||||
}
|
||||
return cloud;
|
||||
}
|
||||
|
||||
const AnimSmrSensor& AnimSmrSensorCloud::getSensor(int index) const {
|
||||
if (index < 0 || index >= getSensorCount()) throw Exception("AnimSmrSensorCloud.getSensor: index out of range");
|
||||
return sensors_[static_cast<size_t>(index)];
|
||||
}
|
||||
|
||||
AnimSmrBodyPart AnimSmrSensorCloud::getSensorPart(int index) const { return getSensor(index).part; }
|
||||
|
||||
void AnimSmrSensorCloud::evaluateWorldPositions(const AnimPose* pose, std::vector<float>& outXYZ) const {
|
||||
if (!pose) throw Exception("AnimSmrSensorCloud.evaluateWorldPositions: pose is null");
|
||||
outXYZ.resize(static_cast<size_t>(getSensorCount()) * 3);
|
||||
for (int i = 0; i < getSensorCount(); ++i) {
|
||||
const AnimSmrSensor& sensor = sensors_[static_cast<size_t>(i)];
|
||||
if (sensor.boneIndex < 0 || sensor.boneIndex >= pose->getBoneCount()) {
|
||||
outXYZ[static_cast<size_t>(i) * 3] = 0.f;
|
||||
outXYZ[static_cast<size_t>(i) * 3 + 1] = 0.f;
|
||||
outXYZ[static_cast<size_t>(i) * 3 + 2] = 0.f;
|
||||
continue;
|
||||
}
|
||||
const Vec3 world = transformPoint(pose->world(sensor.boneIndex), sensor.localX, sensor.localY, sensor.localZ);
|
||||
outXYZ[static_cast<size_t>(i) * 3] = world.x;
|
||||
outXYZ[static_cast<size_t>(i) * 3 + 1] = world.y;
|
||||
outXYZ[static_cast<size_t>(i) * 3 + 2] = world.z;
|
||||
}
|
||||
}
|
||||
|
||||
int smrRefineRetargetedClip(const AnimClip& sourceClip, AnimClip& targetClip, const AnimSkeleton* sourceSkeleton,
|
||||
const AnimSkeleton* targetSkeleton, AnimRetargetProfile& profile) {
|
||||
if (!sourceSkeleton || !targetSkeleton) throw Exception("smrRefineRetargetedClip: skeleton is null");
|
||||
|
||||
if (!profile.skinnedInteractionPreserve_) {
|
||||
profile.interactionCorrectionCount_ = 0;
|
||||
profile.neuralInferenceCount_ = 0;
|
||||
return 0;
|
||||
}
|
||||
|
||||
profile.neuralInferenceCount_ = 0;
|
||||
if (profile.neuralRetargetEnabled_) {
|
||||
if (auto* neural = cap::query<ISmrNeuralRetarget>()) {
|
||||
if (neural->isReady(&profile)) {
|
||||
SmrNeuralRequest request;
|
||||
request.sourceClip = &sourceClip;
|
||||
request.targetClip = &targetClip;
|
||||
request.sourceSkeleton = sourceSkeleton;
|
||||
request.targetSkeleton = targetSkeleton;
|
||||
request.profile = &profile;
|
||||
auto neuralResult = neural->retarget(request);
|
||||
if (neuralResult.ok()) {
|
||||
profile.interactionCorrectionCount_ = neuralResult.value().framesWritten;
|
||||
profile.neuralInferenceCount_ = std::max(1, neuralResult.value().framesWritten);
|
||||
return profile.interactionCorrectionCount_;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const AnimSmrSensorCloud sourceCloud = AnimSmrSensorCloud::fromSkeleton(sourceSkeleton);
|
||||
const AnimSmrSensorCloud targetCloud = AnimSmrSensorCloud::fromSkeleton(targetSkeleton);
|
||||
const std::vector<ResolvedChain> chains = resolveChains(targetSkeleton, profile.interactionIkChains_);
|
||||
if (chains.empty() || sourceCloud.getSensorCount() == 0 || targetCloud.getSensorCount() == 0) {
|
||||
profile.interactionCorrectionCount_ = 0;
|
||||
return 0;
|
||||
}
|
||||
|
||||
const float sourceExtent = skeletonExtent(sourceSkeleton);
|
||||
const float targetExtent = skeletonExtent(targetSkeleton);
|
||||
const float sizeScale = targetExtent / sourceExtent;
|
||||
const float contactDistance =
|
||||
std::max(1e-4f, profile.interactionContactThreshold_ > 0.f ? profile.interactionContactThreshold_
|
||||
: sourceExtent * 0.08f);
|
||||
const float weight = clampf(profile.interactionCorrectionWeight_, 0.f, 1.f);
|
||||
|
||||
const float duration = targetClip.getDuration();
|
||||
const float sampleRate = std::max(targetClip.getSampleRate(), 1.f);
|
||||
const int frameCount = duration <= 0.f ? 1 : std::max(1, static_cast<int>(std::ceil(duration * sampleRate)));
|
||||
|
||||
AnimPose sourcePose;
|
||||
AnimPose targetPose;
|
||||
std::vector<float> sourceXYZ;
|
||||
std::vector<float> targetXYZ;
|
||||
std::vector<InteractionPair> pairs;
|
||||
int corrections = 0;
|
||||
|
||||
for (int frame = 0; frame <= frameCount; ++frame) {
|
||||
if (duration <= 0.f && frame > 0) break;
|
||||
const float time =
|
||||
duration <= 0.f ? 0.f : duration * static_cast<float>(frame) / static_cast<float>(frameCount);
|
||||
sourceClip.sample(time, &sourcePose, sourceSkeleton);
|
||||
sourcePose.computeWorld(sourceSkeleton);
|
||||
targetClip.sample(time, &targetPose, targetSkeleton);
|
||||
targetPose.computeWorld(targetSkeleton);
|
||||
|
||||
sourceCloud.evaluateWorldPositions(&sourcePose, sourceXYZ);
|
||||
targetCloud.evaluateWorldPositions(&targetPose, targetXYZ);
|
||||
collectInteractions(sourceCloud, sourceXYZ, contactDistance, pairs);
|
||||
|
||||
bool frameChanged = false;
|
||||
for (const InteractionPair& pair : pairs) {
|
||||
const int targetObserver = matchSensor(sourceCloud, pair.observer, targetCloud);
|
||||
const int targetSubject = matchSensor(sourceCloud, pair.subject, targetCloud);
|
||||
if (targetObserver < 0 || targetSubject < 0) continue;
|
||||
|
||||
const AnimSmrSensor& subjectSensor = targetCloud.getSensor(targetSubject);
|
||||
const int chainIndex = chainForTip(chains, subjectSensor.boneIndex);
|
||||
if (chainIndex < 0) continue;
|
||||
|
||||
const Vec3 observerWorld{targetXYZ[static_cast<size_t>(targetObserver) * 3],
|
||||
targetXYZ[static_cast<size_t>(targetObserver) * 3 + 1],
|
||||
targetXYZ[static_cast<size_t>(targetObserver) * 3 + 2]};
|
||||
const Vec3 desiredSubject = add(observerWorld, mul(pair.relative, sizeScale));
|
||||
|
||||
const TransformTRS& tipWorld = targetPose.world(subjectSensor.boneIndex);
|
||||
const Vec3 tipPos{tipWorld.px, tipWorld.py, tipWorld.pz};
|
||||
const Vec3 sensorWorld{targetXYZ[static_cast<size_t>(targetSubject) * 3],
|
||||
targetXYZ[static_cast<size_t>(targetSubject) * 3 + 1],
|
||||
targetXYZ[static_cast<size_t>(targetSubject) * 3 + 2]};
|
||||
const Vec3 tipTarget = sub(desiredSubject, sub(sensorWorld, tipPos));
|
||||
|
||||
const ResolvedChain& chain = chains[static_cast<size_t>(chainIndex)];
|
||||
if (targetPose.solveTwoBoneIK(targetSkeleton, chain.root, chain.mid, chain.tip, tipTarget.x, tipTarget.y,
|
||||
tipTarget.z, weight)) {
|
||||
writeRotationKey(targetClip, chain.root, frame, time, targetPose.local(chain.root));
|
||||
writeRotationKey(targetClip, chain.mid, frame, time, targetPose.local(chain.mid));
|
||||
writeRotationKey(targetClip, chain.tip, frame, time, targetPose.local(chain.tip));
|
||||
targetPose.computeWorld(targetSkeleton);
|
||||
targetCloud.evaluateWorldPositions(&targetPose, targetXYZ);
|
||||
frameChanged = true;
|
||||
}
|
||||
}
|
||||
if (frameChanged) ++corrections;
|
||||
}
|
||||
|
||||
profile.interactionCorrectionCount_ = corrections;
|
||||
return corrections;
|
||||
}
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,97 @@
|
||||
#pragma once
|
||||
|
||||
#include "animation/AnimMath.h"
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
class AnimClip;
|
||||
class AnimPose;
|
||||
class AnimRetargetProfile;
|
||||
class AnimSkeleton;
|
||||
|
||||
/** @brief Coarse body-part tag used to select MeshRet-style interaction pairs. */
|
||||
enum class AnimSmrBodyPart {
|
||||
Torso = 0,
|
||||
Arm,
|
||||
Leg,
|
||||
Head,
|
||||
Other,
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief One semantically consistent sensor attached to a bone (classical SCS stand-in).
|
||||
*
|
||||
* MeshRet places sensors via bone-medial ray casts into the mesh. This engine port attaches
|
||||
* bind-local offsets to bones and tags them by body part so interaction pairs stay stable
|
||||
* across differently proportioned skeletons.
|
||||
*/
|
||||
struct AnimSmrSensor {
|
||||
int boneIndex = -1;
|
||||
AnimSmrBodyPart part = AnimSmrBodyPart::Other;
|
||||
float localX = 0.f;
|
||||
float localY = 0.f;
|
||||
float localZ = 0.f;
|
||||
std::string semanticKey; ///< Normalized bone name + sample id for cross-skeleton match.
|
||||
};
|
||||
|
||||
/** @brief Two-bone IK chain used to realize an interaction correction. */
|
||||
struct AnimSmrIkChain {
|
||||
std::string rootBone;
|
||||
std::string midBone;
|
||||
std::string tipBone;
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief Bone-attached sensor cloud for skinned-motion-retarget interaction queries.
|
||||
* Script type: `AnimSmrSensorCloud`.
|
||||
*/
|
||||
class AnimSmrSensorCloud {
|
||||
public:
|
||||
/**
|
||||
* @brief Build sensors for every bone: one at the joint and one mid-segment toward the first child.
|
||||
* @param skeleton Non-null borrowed skeleton; not retained.
|
||||
* @throws Exception if skeleton is null.
|
||||
* @thread Owner thread.
|
||||
*/
|
||||
static AnimSmrSensorCloud fromSkeleton(const AnimSkeleton* skeleton);
|
||||
|
||||
/**
|
||||
* @brief Build denser MeshRet-style SCS rings around each bone segment.
|
||||
* @param ringsPerBone Number of rings along each parent→child segment (>=1).
|
||||
* @param pointsPerRing Samples around each ring (>=3).
|
||||
* @throws Exception if skeleton is null or ring counts are invalid.
|
||||
*/
|
||||
static AnimSmrSensorCloud fromSkeletonDense(const AnimSkeleton* skeleton, int ringsPerBone, int pointsPerRing);
|
||||
|
||||
int getSensorCount() const { return static_cast<int>(sensors_.size()); }
|
||||
const AnimSmrSensor& getSensor(int index) const;
|
||||
AnimSmrBodyPart getSensorPart(int index) const;
|
||||
|
||||
/**
|
||||
* @brief Evaluate world-space sensor positions for a pose that already has computeWorld() applied.
|
||||
* @param outXYZ Packed xyz triples; resized to getSensorCount()*3.
|
||||
* @throws Exception if pose is null.
|
||||
*/
|
||||
void evaluateWorldPositions(const AnimPose* pose, std::vector<float>& outXYZ) const;
|
||||
|
||||
private:
|
||||
std::vector<AnimSmrSensor> sensors_;
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief Refine an FK-retargeted clip so source body-part proximity/contact is preserved on the target.
|
||||
*
|
||||
* Classical stand-in for MeshRet's DMI alignment: detect close sensor pairs on the source,
|
||||
* map them to the target, and pull tip bones with two-bone IK. Returns the number of
|
||||
* frames that received at least one chain correction.
|
||||
*
|
||||
* @throws Exception on null skeleton arguments.
|
||||
* @thread Owner thread; not re-entrant on the same targetClip.
|
||||
*/
|
||||
int smrRefineRetargetedClip(const AnimClip& sourceClip, AnimClip& targetClip, const AnimSkeleton* sourceSkeleton,
|
||||
const AnimSkeleton* targetSkeleton, AnimRetargetProfile& profile);
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,77 @@
|
||||
#pragma once
|
||||
|
||||
#include "common/Result.h"
|
||||
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
class AnimClip;
|
||||
class AnimRetargetProfile;
|
||||
class AnimSkeleton;
|
||||
class AnimSkin;
|
||||
|
||||
/**
|
||||
* @brief Inputs for neural MeshRet-style skinned motion retarget.
|
||||
* @note All pointers are borrowed for the duration of ISmrNeuralRetarget::retarget only.
|
||||
* @thread Owner thread of the animation/tensor composition; no re-entry on targetClip.
|
||||
*/
|
||||
struct SmrNeuralRequest {
|
||||
const AnimClip* sourceClip = nullptr;
|
||||
AnimClip* targetClip = nullptr; ///< FK-retargeted clip; overwritten on success.
|
||||
const AnimSkeleton* sourceSkeleton = nullptr;
|
||||
const AnimSkeleton* targetSkeleton = nullptr;
|
||||
const AnimRetargetProfile* profile = nullptr;
|
||||
const AnimSkin* sourceSkin = nullptr; ///< Optional LBS skin for denser SCS.
|
||||
const AnimSkin* targetSkin = nullptr;
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief Owning diagnostics from one neural retarget call.
|
||||
*/
|
||||
struct SmrNeuralResult {
|
||||
int framesWritten = 0;
|
||||
std::string backend; ///< "onnx" or "tensor-meshret".
|
||||
std::size_t sensorCount = 0;
|
||||
std::size_t pairCount = 0;
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief Optional MeshRet/SMR neural retarget provider (implemented by animation_tensor).
|
||||
*
|
||||
* Animation stays free of tensor includes; the L6 satellite registers this capability.
|
||||
* When absent or not ready, classical two-bone IK SMR remains the active path.
|
||||
*
|
||||
* @ownership Provider is owned by AnimationTensor; revoked before destruction.
|
||||
* @thread Composition/owner thread only. No callbacks, script re-entry, or background work.
|
||||
*/
|
||||
class ISmrNeuralRetarget {
|
||||
public:
|
||||
static constexpr const char* capabilityName = "animation.ISmrNeuralRetarget";
|
||||
|
||||
virtual ~ISmrNeuralRetarget() = default;
|
||||
|
||||
/** @brief Stable backend label describing the live execution path. */
|
||||
[[nodiscard]] virtual std::string name() const = 0;
|
||||
|
||||
/**
|
||||
* @brief True when a model is loaded (ONNX path) or the builtin tensor MeshRet graph is usable.
|
||||
* @param profile Borrowed for the call only; nullptr means not ready.
|
||||
* @note Does not retain profile; safe to call without a clip.
|
||||
* @ownership Does not retain profile.
|
||||
* @lifetime profile must outlive this call.
|
||||
* @thread Owner thread.
|
||||
*/
|
||||
[[nodiscard]] virtual bool isReady(const AnimRetargetProfile* profile) const = 0;
|
||||
|
||||
/**
|
||||
* @brief Run MeshRet-style inference and rewrite targetClip local rotations.
|
||||
* @return Owning diagnostics, or structured failure without mutating targetClip.
|
||||
* @lifetime See SmrNeuralRequest; pointers borrowed for this call only.
|
||||
* @thread Owner thread; not re-entrant on targetClip.
|
||||
*/
|
||||
[[nodiscard]] virtual Result<SmrNeuralResult> retarget(const SmrNeuralRequest& request) = 0;
|
||||
};
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -22,6 +22,22 @@ void exposeAnimClipBindings(ssq::Table& table) {
|
||||
retarget.addFunc("getRootVerticalScale", &AnimRetargetProfile::getRootVerticalScale);
|
||||
retarget.addFunc("setUseSkeletonSpaceRotation", &AnimRetargetProfile::setUseSkeletonSpaceRotation);
|
||||
retarget.addFunc("getUseSkeletonSpaceRotation", &AnimRetargetProfile::getUseSkeletonSpaceRotation);
|
||||
retarget.addFunc("setSkinnedInteractionPreserve", &AnimRetargetProfile::setSkinnedInteractionPreserve);
|
||||
retarget.addFunc("getSkinnedInteractionPreserve", &AnimRetargetProfile::getSkinnedInteractionPreserve);
|
||||
retarget.addFunc("setInteractionContactThreshold", &AnimRetargetProfile::setInteractionContactThreshold);
|
||||
retarget.addFunc("getInteractionContactThreshold", &AnimRetargetProfile::getInteractionContactThreshold);
|
||||
retarget.addFunc("setInteractionCorrectionWeight", &AnimRetargetProfile::setInteractionCorrectionWeight);
|
||||
retarget.addFunc("getInteractionCorrectionWeight", &AnimRetargetProfile::getInteractionCorrectionWeight);
|
||||
retarget.addFunc("addInteractionIkChain", &AnimRetargetProfile::addInteractionIkChain);
|
||||
retarget.addFunc("clearInteractionIkChains", &AnimRetargetProfile::clearInteractionIkChains);
|
||||
retarget.addFunc("getInteractionCorrectionCount", &AnimRetargetProfile::getInteractionCorrectionCount);
|
||||
retarget.addFunc("setNeuralModelPath", &AnimRetargetProfile::setNeuralModelPath);
|
||||
retarget.addFunc("getNeuralModelPath", &AnimRetargetProfile::getNeuralModelPath);
|
||||
retarget.addFunc("setNeuralRetargetEnabled", &AnimRetargetProfile::setNeuralRetargetEnabled);
|
||||
retarget.addFunc("getNeuralRetargetEnabled", &AnimRetargetProfile::getNeuralRetargetEnabled);
|
||||
retarget.addFunc("setNeuralBackend", &AnimRetargetProfile::setNeuralBackend);
|
||||
retarget.addFunc("getNeuralBackend", &AnimRetargetProfile::getNeuralBackend);
|
||||
retarget.addFunc("getNeuralInferenceCount", &AnimRetargetProfile::getNeuralInferenceCount);
|
||||
retarget.addFunc("getMatchedBoneCount", &AnimRetargetProfile::getMatchedBoneCount);
|
||||
retarget.addFunc("getUnmatchedBoneCount", &AnimRetargetProfile::getUnmatchedBoneCount);
|
||||
retarget.addFunc("getUnmatchedTargetBone", &AnimRetargetProfile::getUnmatchedTargetBone);
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
#include "animation/tensor/AnimationTensor.h"
|
||||
|
||||
#include "animation/AnimSmrNeural.h"
|
||||
#include "animation/tensor/SmrNeuralProvider.h"
|
||||
#include "common/Capability.h"
|
||||
|
||||
#include <simplesquirrel/simplesquirrel.hpp>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
struct AnimationTensor::Impl {
|
||||
std::unique_ptr<SmrNeuralProvider> provider;
|
||||
};
|
||||
|
||||
Module_IMPL(AnimationTensor, new AnimationTensor());
|
||||
|
||||
AnimationTensor::AnimationTensor() : impl_(std::make_unique<Impl>()) {
|
||||
impl_->provider = std::make_unique<SmrNeuralProvider>();
|
||||
cap::provide<ISmrNeuralRetarget>(impl_->provider.get());
|
||||
}
|
||||
|
||||
AnimationTensor::~AnimationTensor() {
|
||||
if (impl_ && impl_->provider) cap::revoke<ISmrNeuralRetarget>(impl_->provider.get());
|
||||
}
|
||||
|
||||
void AnimationTensor::expose(ssq::Table& table) {
|
||||
auto cls = table.addClass(name, AnimationTensor::create, false);
|
||||
expose(cls);
|
||||
}
|
||||
|
||||
void AnimationTensor::expose(ssq::Class& cls) { cls.addFunc("getName", &AnimationTensor::getName); }
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,24 @@
|
||||
#pragma once
|
||||
|
||||
#include "common/Module.h"
|
||||
|
||||
#include <memory>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
/**
|
||||
* @brief Manager-owned MeshRet neural retarget satellite (animation + tensor).
|
||||
* @thread Owner/composition thread only; revokes capability before destruction.
|
||||
*/
|
||||
class AnimationTensor : public Module {
|
||||
public:
|
||||
Module_REG(AnimationTensor);
|
||||
AnimationTensor();
|
||||
~AnimationTensor() override;
|
||||
|
||||
private:
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,254 @@
|
||||
#include "animation/tensor/SmrFeatures.h"
|
||||
|
||||
#include "animation/AnimClip.h"
|
||||
#include "animation/AnimPose.h"
|
||||
#include "animation/AnimSkeleton.h"
|
||||
#include "common/Exception.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cctype>
|
||||
#include <cmath>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
namespace {
|
||||
|
||||
bool partsInteract(AnimSmrBodyPart a, AnimSmrBodyPart b) {
|
||||
if (a > b) std::swap(a, b);
|
||||
if (a == AnimSmrBodyPart::Arm &&
|
||||
(b == AnimSmrBodyPart::Torso || b == AnimSmrBodyPart::Arm || b == AnimSmrBodyPart::Head))
|
||||
return true;
|
||||
if (a == AnimSmrBodyPart::Leg && b == AnimSmrBodyPart::Leg) return true;
|
||||
if (a == AnimSmrBodyPart::Torso && b == AnimSmrBodyPart::Head) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
std::string normalizeToken(const std::string& name) {
|
||||
const size_t separator = name.find_last_of(":|/");
|
||||
const size_t begin = separator == std::string::npos ? 0 : separator + 1;
|
||||
std::string out;
|
||||
out.reserve(name.size() - begin);
|
||||
for (size_t i = begin; i < name.size(); ++i) {
|
||||
const unsigned char c = static_cast<unsigned char>(name[i]);
|
||||
if (std::isalnum(c)) out.push_back(static_cast<char>(std::tolower(c)));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
int matchJoint(const AnimSkeleton* source, const AnimSkeleton* target, int targetBone) {
|
||||
const int exact = source->findBone(target->getBoneName(targetBone));
|
||||
if (exact >= 0) return exact;
|
||||
const std::string want = normalizeToken(target->getBoneName(targetBone));
|
||||
for (int bone = 0; bone < source->getBoneCount(); ++bone) {
|
||||
if (normalizeToken(source->getBoneName(bone)) == want) return bone;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void quatToRot6d(float qx, float qy, float qz, float qw, float out[6]) {
|
||||
const float x = qx, y = qy, z = qz, w = qw;
|
||||
const float x2 = x + x, y2 = y + y, z2 = z + z;
|
||||
const float xx = x * x2, xy = x * y2, xz = x * z2;
|
||||
const float yy = y * y2, yz = y * z2, zz = z * z2;
|
||||
const float wx = w * x2, wy = w * y2, wz = w * z2;
|
||||
out[0] = 1.f - (yy + zz);
|
||||
out[1] = xy + wz;
|
||||
out[2] = xz - wy;
|
||||
out[3] = xy - wz;
|
||||
out[4] = 1.f - (xx + zz);
|
||||
out[5] = yz + wx;
|
||||
}
|
||||
|
||||
void rot6dToQuat(const float in[6], float& qx, float& qy, float& qz, float& qw) {
|
||||
float c0[3] = {in[0], in[1], in[2]};
|
||||
float c1[3] = {in[3], in[4], in[5]};
|
||||
auto normalize = [](float* v) {
|
||||
const float n = std::sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]);
|
||||
if (n < 1e-8f) {
|
||||
v[0] = 1.f;
|
||||
v[1] = 0.f;
|
||||
v[2] = 0.f;
|
||||
return;
|
||||
}
|
||||
v[0] /= n;
|
||||
v[1] /= n;
|
||||
v[2] /= n;
|
||||
};
|
||||
normalize(c0);
|
||||
const float dot = c0[0] * c1[0] + c0[1] * c1[1] + c0[2] * c1[2];
|
||||
c1[0] -= dot * c0[0];
|
||||
c1[1] -= dot * c0[1];
|
||||
c1[2] -= dot * c0[2];
|
||||
normalize(c1);
|
||||
const float c2[3] = {c0[1] * c1[2] - c0[2] * c1[1], c0[2] * c1[0] - c0[0] * c1[2], c0[0] * c1[1] - c0[1] * c1[0]};
|
||||
const float m00 = c0[0], m01 = c1[0], m02 = c2[0];
|
||||
const float m10 = c0[1], m11 = c1[1], m12 = c2[1];
|
||||
const float m20 = c0[2], m21 = c1[2], m22 = c2[2];
|
||||
const float trace = m00 + m11 + m22;
|
||||
if (trace > 0.f) {
|
||||
const float s = 0.5f / std::sqrt(trace + 1.f);
|
||||
qw = 0.25f / s;
|
||||
qx = (m21 - m12) * s;
|
||||
qy = (m02 - m20) * s;
|
||||
qz = (m10 - m01) * s;
|
||||
} else if (m00 > m11 && m00 > m22) {
|
||||
const float s = 2.f * std::sqrt(1.f + m00 - m11 - m22);
|
||||
qw = (m21 - m12) / s;
|
||||
qx = 0.25f * s;
|
||||
qy = (m01 + m10) / s;
|
||||
qz = (m02 + m20) / s;
|
||||
} else if (m11 > m22) {
|
||||
const float s = 2.f * std::sqrt(1.f + m11 - m00 - m22);
|
||||
qw = (m02 - m20) / s;
|
||||
qx = (m01 + m10) / s;
|
||||
qy = 0.25f * s;
|
||||
qz = (m12 + m21) / s;
|
||||
} else {
|
||||
const float s = 2.f * std::sqrt(1.f + m22 - m00 - m11);
|
||||
qw = (m10 - m01) / s;
|
||||
qx = (m02 + m20) / s;
|
||||
qy = (m12 + m21) / s;
|
||||
qz = 0.25f * s;
|
||||
}
|
||||
const float n = std::sqrt(qx * qx + qy * qy + qz * qz + qw * qw);
|
||||
if (n > 1e-8f) {
|
||||
qx /= n;
|
||||
qy /= n;
|
||||
qz /= n;
|
||||
qw /= n;
|
||||
} else {
|
||||
qx = qy = qz = 0.f;
|
||||
qw = 1.f;
|
||||
}
|
||||
}
|
||||
|
||||
SmrFeatureBatch buildSmrFeatures(const AnimClip& sourceClip, const AnimSkeleton* sourceSkeleton,
|
||||
const AnimSkeleton* targetSkeleton, const AnimSkin* /*sourceSkin*/,
|
||||
const AnimSkin* /*targetSkin*/, int ringsPerBone, int pointsPerRing) {
|
||||
if (!sourceSkeleton || !targetSkeleton) throw Exception("buildSmrFeatures: skeleton is null");
|
||||
|
||||
SmrFeatureBatch batch;
|
||||
batch.joints = targetSkeleton->getBoneCount();
|
||||
batch.jointMap.assign(static_cast<size_t>(batch.joints), -1);
|
||||
for (int t = 0; t < batch.joints; ++t)
|
||||
batch.jointMap[static_cast<size_t>(t)] = matchJoint(sourceSkeleton, targetSkeleton, t);
|
||||
|
||||
const AnimSmrSensorCloud sourceCloud =
|
||||
AnimSmrSensorCloud::fromSkeletonDense(sourceSkeleton, ringsPerBone, pointsPerRing);
|
||||
const AnimSmrSensorCloud targetCloud =
|
||||
AnimSmrSensorCloud::fromSkeletonDense(targetSkeleton, ringsPerBone, pointsPerRing);
|
||||
batch.sensors = std::min(sourceCloud.getSensorCount(), targetCloud.getSensorCount());
|
||||
if (batch.sensors <= 0) {
|
||||
batch.sensors = 1;
|
||||
batch.sourceGeom.assign(7u, 0.f);
|
||||
batch.targetGeom.assign(7u, 0.f);
|
||||
} else {
|
||||
batch.sourceGeom.assign(static_cast<size_t>(batch.sensors) * 7u, 0.f);
|
||||
batch.targetGeom.assign(static_cast<size_t>(batch.sensors) * 7u, 0.f);
|
||||
}
|
||||
|
||||
AnimPose sourceBind;
|
||||
AnimPose targetBind;
|
||||
sourceSkeleton->applyBindPose(&sourceBind);
|
||||
targetSkeleton->applyBindPose(&targetBind);
|
||||
sourceBind.computeWorld(sourceSkeleton);
|
||||
targetBind.computeWorld(targetSkeleton);
|
||||
|
||||
if (sourceCloud.getSensorCount() > 0 && targetCloud.getSensorCount() > 0) {
|
||||
std::vector<float> sourceRest;
|
||||
std::vector<float> targetRest;
|
||||
sourceCloud.evaluateWorldPositions(&sourceBind, sourceRest);
|
||||
targetCloud.evaluateWorldPositions(&targetBind, targetRest);
|
||||
for (int i = 0; i < batch.sensors; ++i) {
|
||||
for (int k = 0; k < 3; ++k) {
|
||||
batch.sourceGeom[static_cast<size_t>(i) * 7u + static_cast<size_t>(k)] =
|
||||
sourceRest[static_cast<size_t>(i) * 3u + static_cast<size_t>(k)];
|
||||
batch.targetGeom[static_cast<size_t>(i) * 7u + static_cast<size_t>(k)] =
|
||||
targetRest[static_cast<size_t>(i) * 3u + static_cast<size_t>(k)];
|
||||
}
|
||||
batch.sourceGeom[static_cast<size_t>(i) * 7u + 3u] = static_cast<float>(sourceCloud.getSensorPart(i));
|
||||
batch.targetGeom[static_cast<size_t>(i) * 7u + 3u] = static_cast<float>(targetCloud.getSensorPart(i));
|
||||
batch.sourceGeom[static_cast<size_t>(i) * 7u + 4u] = static_cast<float>(sourceCloud.getSensor(i).boneIndex);
|
||||
batch.targetGeom[static_cast<size_t>(i) * 7u + 4u] = static_cast<float>(targetCloud.getSensor(i).boneIndex);
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<std::pair<int, int>> pairIdx;
|
||||
const int sensorLimit = std::min(batch.sensors, sourceCloud.getSensorCount());
|
||||
for (int a = 0; a < sensorLimit; ++a) {
|
||||
for (int b = a + 1; b < sensorLimit; ++b) {
|
||||
if (partsInteract(sourceCloud.getSensorPart(a), sourceCloud.getSensorPart(b))) pairIdx.emplace_back(a, b);
|
||||
}
|
||||
}
|
||||
batch.pairs = static_cast<int>(pairIdx.size());
|
||||
|
||||
const float duration = sourceClip.getDuration();
|
||||
const float rate = std::max(sourceClip.getSampleRate(), 1.f);
|
||||
batch.frames = duration <= 0.f ? 1 : std::max(1, static_cast<int>(std::ceil(duration * rate)) + 1);
|
||||
batch.sourceRot6d.assign(static_cast<size_t>(batch.frames) * static_cast<size_t>(batch.joints) * 6u, 0.f);
|
||||
batch.sourceDmi.assign(static_cast<size_t>(batch.frames) * static_cast<size_t>(std::max(batch.pairs, 1)) * 10u,
|
||||
0.f);
|
||||
|
||||
AnimPose pose;
|
||||
std::vector<float> xyz;
|
||||
std::vector<float> prevXyz;
|
||||
for (int frame = 0; frame < batch.frames; ++frame) {
|
||||
const float time =
|
||||
duration <= 0.f ? 0.f
|
||||
: duration * static_cast<float>(frame) / static_cast<float>(std::max(batch.frames - 1, 1));
|
||||
sourceClip.sample(time, &pose, sourceSkeleton);
|
||||
pose.computeWorld(sourceSkeleton);
|
||||
for (int joint = 0; joint < batch.joints; ++joint) {
|
||||
const int sourceJoint = batch.jointMap[static_cast<size_t>(joint)];
|
||||
float rot[6] = {1.f, 0.f, 0.f, 0.f, 1.f, 0.f};
|
||||
if (sourceJoint >= 0) {
|
||||
const TransformTRS& local = pose.local(sourceJoint);
|
||||
quatToRot6d(local.qx, local.qy, local.qz, local.qw, rot);
|
||||
}
|
||||
const size_t base =
|
||||
(static_cast<size_t>(frame) * static_cast<size_t>(batch.joints) + static_cast<size_t>(joint)) * 6u;
|
||||
for (int k = 0; k < 6; ++k) batch.sourceRot6d[base + static_cast<size_t>(k)] = rot[k];
|
||||
}
|
||||
if (sourceCloud.getSensorCount() > 0) sourceCloud.evaluateWorldPositions(&pose, xyz);
|
||||
for (int p = 0; p < batch.pairs; ++p) {
|
||||
const int a = pairIdx[static_cast<size_t>(p)].first;
|
||||
const int b = pairIdx[static_cast<size_t>(p)].second;
|
||||
const float ax = xyz[static_cast<size_t>(a) * 3u];
|
||||
const float ay = xyz[static_cast<size_t>(a) * 3u + 1u];
|
||||
const float az = xyz[static_cast<size_t>(a) * 3u + 2u];
|
||||
const float bx = xyz[static_cast<size_t>(b) * 3u];
|
||||
const float by = xyz[static_cast<size_t>(b) * 3u + 1u];
|
||||
const float bz = xyz[static_cast<size_t>(b) * 3u + 2u];
|
||||
const float dx = bx - ax, dy = by - ay, dz = bz - az;
|
||||
const float dist = std::sqrt(dx * dx + dy * dy + dz * dz);
|
||||
float vx = 0.f, vy = 0.f, vz = 0.f;
|
||||
if (!prevXyz.empty()) {
|
||||
vx = (bx - prevXyz[static_cast<size_t>(b) * 3u]) - (ax - prevXyz[static_cast<size_t>(a) * 3u]);
|
||||
vy =
|
||||
(by - prevXyz[static_cast<size_t>(b) * 3u + 1u]) - (ay - prevXyz[static_cast<size_t>(a) * 3u + 1u]);
|
||||
vz =
|
||||
(bz - prevXyz[static_cast<size_t>(b) * 3u + 2u]) - (az - prevXyz[static_cast<size_t>(a) * 3u + 2u]);
|
||||
}
|
||||
const size_t base =
|
||||
(static_cast<size_t>(frame) * static_cast<size_t>(batch.pairs) + static_cast<size_t>(p)) * 10u;
|
||||
batch.sourceDmi[base + 0] = dx;
|
||||
batch.sourceDmi[base + 1] = dy;
|
||||
batch.sourceDmi[base + 2] = dz;
|
||||
batch.sourceDmi[base + 3] = dist;
|
||||
batch.sourceDmi[base + 4] = static_cast<float>(sourceCloud.getSensorPart(a));
|
||||
batch.sourceDmi[base + 5] = static_cast<float>(sourceCloud.getSensorPart(b));
|
||||
batch.sourceDmi[base + 6] = vx;
|
||||
batch.sourceDmi[base + 7] = vy;
|
||||
batch.sourceDmi[base + 8] = vz;
|
||||
batch.sourceDmi[base + 9] = 1.f;
|
||||
}
|
||||
prevXyz = xyz;
|
||||
}
|
||||
return batch;
|
||||
}
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,49 @@
|
||||
#pragma once
|
||||
|
||||
#include "animation/AnimSmr.h"
|
||||
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
class AnimClip;
|
||||
class AnimSkeleton;
|
||||
class AnimSkin;
|
||||
|
||||
/** @brief Packed MeshRet-style features for one retarget inference window. */
|
||||
struct SmrFeatureBatch {
|
||||
int frames = 0;
|
||||
int joints = 0;
|
||||
int sensors = 0;
|
||||
int pairs = 0;
|
||||
|
||||
std::vector<float> sourceRot6d; ///< [T*J*6]
|
||||
std::vector<float> sourceGeom; ///< [S*7]
|
||||
std::vector<float> targetGeom; ///< [S*7]
|
||||
std::vector<float> sourceDmi; ///< [T*P*10]
|
||||
std::vector<int> jointMap; ///< target joint -> source joint (-1 if unmatched)
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief Build dense SCS + DMI tensors from skeletons/clip (no tensor headers).
|
||||
* @param sourceSkeleton Borrowed; must outlive the call. Required.
|
||||
* @param targetSkeleton Borrowed; must outlive the call. Required.
|
||||
* @param sourceSkin Optional borrowed LBS skin for denser SCS; nullptr skips skin rings.
|
||||
* @param targetSkin Optional borrowed LBS skin; nullptr skips skin rings.
|
||||
* @return Owning packed feature batch.
|
||||
* @throws Exception on null required skeletons or empty clip.
|
||||
* @ownership Does not retain any input pointers after return.
|
||||
* @lifetime All raw pointers are borrowed for the duration of this call only.
|
||||
* @thread Owner thread.
|
||||
*/
|
||||
SmrFeatureBatch buildSmrFeatures(const AnimClip& sourceClip, const AnimSkeleton* sourceSkeleton,
|
||||
const AnimSkeleton* targetSkeleton, const AnimSkin* sourceSkin,
|
||||
const AnimSkin* targetSkin, int ringsPerBone = 2, int pointsPerRing = 4);
|
||||
|
||||
/** @brief Convert unit quaternion to MeshRet-style rotation-6D. */
|
||||
void quatToRot6d(float qx, float qy, float qz, float qw, float out[6]);
|
||||
|
||||
/** @brief Convert rotation-6D back to a unit quaternion. */
|
||||
void rot6dToQuat(const float in[6], float& qx, float& qy, float& qz, float& qw);
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,267 @@
|
||||
#include "animation/tensor/SmrMeshRetNet.h"
|
||||
|
||||
#include "common/Diagnostic.h"
|
||||
#include "tensor/Tensor.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <random>
|
||||
#include <utility>
|
||||
|
||||
namespace eve::animation {
|
||||
namespace {
|
||||
|
||||
void gemm(const float* a, const float* b, float* c, int m, int k, int n) {
|
||||
for (int i = 0; i < m; ++i)
|
||||
for (int j = 0; j < n; ++j) {
|
||||
float sum = 0.f;
|
||||
for (int t = 0; t < k; ++t) sum += a[i * k + t] * b[t * n + j];
|
||||
c[i * n + j] = sum;
|
||||
}
|
||||
}
|
||||
|
||||
void addBias(float* x, const float* bias, int rows, int cols) {
|
||||
for (int r = 0; r < rows; ++r)
|
||||
for (int c = 0; c < cols; ++c) x[r * cols + c] += bias[c];
|
||||
}
|
||||
|
||||
void gelu(float* x, int n) {
|
||||
for (int i = 0; i < n; ++i) {
|
||||
const float v = x[i];
|
||||
x[i] = 0.5f * v * (1.f + std::tanh(0.7978845608f * (v + 0.044715f * v * v * v)));
|
||||
}
|
||||
}
|
||||
|
||||
void layernorm(float* x, int rows, int cols, const float* g, const float* b) {
|
||||
for (int r = 0; r < rows; ++r) {
|
||||
float* row = x + r * cols;
|
||||
float mean = 0.f;
|
||||
for (int c = 0; c < cols; ++c) mean += row[c];
|
||||
mean /= static_cast<float>(cols);
|
||||
float var = 0.f;
|
||||
for (int c = 0; c < cols; ++c) {
|
||||
const float d = row[c] - mean;
|
||||
var += d * d;
|
||||
}
|
||||
var = 1.f / std::sqrt(var / static_cast<float>(cols) + 1e-5f);
|
||||
for (int c = 0; c < cols; ++c) row[c] = (row[c] - mean) * var * g[c] + b[c];
|
||||
}
|
||||
}
|
||||
|
||||
void softmaxRows(float* x, int rows, int cols) {
|
||||
for (int r = 0; r < rows; ++r) {
|
||||
float* row = x + r * cols;
|
||||
float mx = row[0];
|
||||
for (int c = 1; c < cols; ++c) mx = std::max(mx, row[c]);
|
||||
float sum = 0.f;
|
||||
for (int c = 0; c < cols; ++c) {
|
||||
row[c] = std::exp(row[c] - mx);
|
||||
sum += row[c];
|
||||
}
|
||||
const float inv = 1.f / std::max(sum, 1e-8f);
|
||||
for (int c = 0; c < cols; ++c) row[c] *= inv;
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<float> xavier(std::mt19937& rng, int rows, int cols) {
|
||||
std::normal_distribution<float> dist(0.f, std::sqrt(2.f / static_cast<float>(rows + cols)));
|
||||
std::vector<float> out(static_cast<size_t>(rows * cols));
|
||||
for (float& v : out) v = dist(rng);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<float> zeros(int n) { return std::vector<float>(static_cast<size_t>(n), 0.f); }
|
||||
std::vector<float> ones(int n) { return std::vector<float>(static_cast<size_t>(n), 1.f); }
|
||||
|
||||
std::vector<float> mlp2(const float* x, int rows, int inDim, int mid, int outDim, const std::vector<float>& w1,
|
||||
const std::vector<float>& b1, const std::vector<float>& w2, const std::vector<float>& b2) {
|
||||
std::vector<float> h(static_cast<size_t>(rows * mid));
|
||||
gemm(x, w1.data(), h.data(), rows, inDim, mid);
|
||||
addBias(h.data(), b1.data(), rows, mid);
|
||||
gelu(h.data(), rows * mid);
|
||||
std::vector<float> y(static_cast<size_t>(rows * outDim));
|
||||
gemm(h.data(), w2.data(), y.data(), rows, mid, outDim);
|
||||
addBias(y.data(), b2.data(), rows, outDim);
|
||||
return y;
|
||||
}
|
||||
|
||||
std::vector<float> maxPool(const std::vector<float>& x, int groups, int count, int dim) {
|
||||
std::vector<float> out(static_cast<size_t>(groups * dim), -1e30f);
|
||||
for (int g = 0; g < groups; ++g)
|
||||
for (int i = 0; i < count; ++i)
|
||||
for (int d = 0; d < dim; ++d) {
|
||||
const float v = x[static_cast<size_t>((g * count + i) * dim + d)];
|
||||
float& slot = out[static_cast<size_t>(g * dim + d)];
|
||||
slot = std::max(slot, v);
|
||||
}
|
||||
for (float& v : out)
|
||||
if (v < -1e29f) v = 0.f;
|
||||
return out;
|
||||
}
|
||||
|
||||
void transformerLayer(std::vector<float>& x, int tokens, int dim, int heads, const std::vector<float>& wq,
|
||||
const std::vector<float>& wk, const std::vector<float>& wv, const std::vector<float>& wo,
|
||||
const std::vector<float>& w1, const std::vector<float>& b1, const std::vector<float>& w2,
|
||||
const std::vector<float>& b2, const std::vector<float>& ln1g, const std::vector<float>& ln1b,
|
||||
const std::vector<float>& ln2g, const std::vector<float>& ln2b, const std::vector<float>* memory,
|
||||
int memoryTokens) {
|
||||
const int headDim = std::max(1, dim / std::max(heads, 1));
|
||||
const int srcTokens = memory ? memoryTokens : tokens;
|
||||
std::vector<float> residual = x;
|
||||
layernorm(x.data(), tokens, dim, ln1g.data(), ln1b.data());
|
||||
std::vector<float> q(static_cast<size_t>(tokens * dim));
|
||||
std::vector<float> k(static_cast<size_t>(srcTokens * dim));
|
||||
std::vector<float> v(static_cast<size_t>(srcTokens * dim));
|
||||
gemm(x.data(), wq.data(), q.data(), tokens, dim, dim);
|
||||
const float* src = memory ? memory->data() : x.data();
|
||||
gemm(src, wk.data(), k.data(), srcTokens, dim, dim);
|
||||
gemm(src, wv.data(), v.data(), srcTokens, dim, dim);
|
||||
std::vector<float> ctx(static_cast<size_t>(tokens * dim), 0.f);
|
||||
const float scale = 1.f / std::sqrt(static_cast<float>(headDim));
|
||||
for (int h = 0; h < heads; ++h) {
|
||||
std::vector<float> scores(static_cast<size_t>(tokens * srcTokens));
|
||||
for (int t = 0; t < tokens; ++t)
|
||||
for (int s = 0; s < srcTokens; ++s) {
|
||||
float sum = 0.f;
|
||||
for (int d = 0; d < headDim; ++d)
|
||||
sum += q[static_cast<size_t>(t * dim + h * headDim + d)] *
|
||||
k[static_cast<size_t>(s * dim + h * headDim + d)];
|
||||
scores[static_cast<size_t>(t * srcTokens + s)] = sum * scale;
|
||||
}
|
||||
softmaxRows(scores.data(), tokens, srcTokens);
|
||||
for (int t = 0; t < tokens; ++t)
|
||||
for (int d = 0; d < headDim; ++d) {
|
||||
float sum = 0.f;
|
||||
for (int s = 0; s < srcTokens; ++s)
|
||||
sum += scores[static_cast<size_t>(t * srcTokens + s)] *
|
||||
v[static_cast<size_t>(s * dim + h * headDim + d)];
|
||||
ctx[static_cast<size_t>(t * dim + h * headDim + d)] = sum;
|
||||
}
|
||||
}
|
||||
std::vector<float> projected(static_cast<size_t>(tokens * dim));
|
||||
gemm(ctx.data(), wo.data(), projected.data(), tokens, dim, dim);
|
||||
for (size_t i = 0; i < x.size(); ++i) x[i] = residual[i] + projected[i];
|
||||
residual = x;
|
||||
layernorm(x.data(), tokens, dim, ln2g.data(), ln2b.data());
|
||||
auto ff = mlp2(x.data(), tokens, dim, static_cast<int>(b1.size()), dim, w1, b1, w2, b2);
|
||||
for (size_t i = 0; i < x.size(); ++i) x[i] = residual[i] + ff[i];
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
SmrMeshRetNet::SmrMeshRetNet(SmrMeshRetConfig config) : config_(config) { initWeights(); }
|
||||
|
||||
void SmrMeshRetNet::initWeights() {
|
||||
std::mt19937 rng(static_cast<uint32_t>(config_.seed));
|
||||
const int d = config_.latentDim;
|
||||
geomW1_ = xavier(rng, 7, d);
|
||||
geomB1_ = zeros(d);
|
||||
geomW2_ = xavier(rng, d, d);
|
||||
geomB2_ = zeros(d);
|
||||
dmiW1_ = xavier(rng, 10, d);
|
||||
dmiB1_ = zeros(d);
|
||||
dmiW2_ = xavier(rng, d, d);
|
||||
dmiB2_ = zeros(d);
|
||||
motionW_ = xavier(rng, 6, d);
|
||||
motionB_ = zeros(d);
|
||||
fuseW_ = xavier(rng, d * 3, d);
|
||||
fuseB_ = zeros(d);
|
||||
outW_ = xavier(rng, d, 6);
|
||||
outB_ = zeros(6);
|
||||
auto fill = [&](std::vector<LayerWeights>& layers) {
|
||||
layers.assign(static_cast<size_t>(config_.numLayers), {});
|
||||
for (auto& layer : layers) {
|
||||
layer.wq = xavier(rng, d, d);
|
||||
layer.wk = xavier(rng, d, d);
|
||||
layer.wv = xavier(rng, d, d);
|
||||
layer.wo = xavier(rng, d, d);
|
||||
layer.w1 = xavier(rng, d, config_.ffSize);
|
||||
layer.b1 = zeros(config_.ffSize);
|
||||
layer.w2 = xavier(rng, config_.ffSize, d);
|
||||
layer.b2 = zeros(d);
|
||||
layer.ln1g = ones(d);
|
||||
layer.ln1b = zeros(d);
|
||||
layer.ln2g = ones(d);
|
||||
layer.ln2b = zeros(d);
|
||||
}
|
||||
};
|
||||
fill(enc_);
|
||||
fill(dec_);
|
||||
tensor::Tensor probe(d);
|
||||
probe.fill(0.f);
|
||||
(void)probe;
|
||||
}
|
||||
|
||||
Result<std::vector<float>> SmrMeshRetNet::forward(const SmrFeatureBatch& features) const {
|
||||
if (features.frames <= 0 || features.joints <= 0) {
|
||||
return Result<std::vector<float>>::failure(
|
||||
Diagnostic::error(DiagnosticCode::InvalidArgument, "SmrMeshRetNet.forward: empty features"));
|
||||
}
|
||||
const int d = config_.latentDim;
|
||||
const int T = features.frames;
|
||||
const int J = features.joints;
|
||||
const int S = std::max(features.sensors, 1);
|
||||
const int P = std::max(features.pairs, 1);
|
||||
const int heads = std::max(1, config_.numHeads);
|
||||
|
||||
auto srcGeom = mlp2(features.sourceGeom.data(), S, 7, d, d, geomW1_, geomB1_, geomW2_, geomB2_);
|
||||
auto tgtGeom = mlp2(features.targetGeom.data(), S, 7, d, d, geomW1_, geomB1_, geomW2_, geomB2_);
|
||||
auto srcPool = maxPool(srcGeom, 1, S, d);
|
||||
auto tgtPool = maxPool(tgtGeom, 1, S, d);
|
||||
|
||||
std::vector<float> temporal(static_cast<size_t>(T * d), 0.f);
|
||||
for (int t = 0; t < T; ++t) {
|
||||
const float* dmiFrame = features.sourceDmi.data() + static_cast<size_t>(t) * static_cast<size_t>(P) * 10u;
|
||||
auto dmiEnc = mlp2(dmiFrame, P, 10, d, d, dmiW1_, dmiB1_, dmiW2_, dmiB2_);
|
||||
auto dmiPool = maxPool(dmiEnc, 1, P, d);
|
||||
std::vector<float> meanRot(6, 0.f);
|
||||
for (int j = 0; j < J; ++j) {
|
||||
const float* r = features.sourceRot6d.data() + static_cast<size_t>((t * J + j) * 6);
|
||||
for (int k = 0; k < 6; ++k) meanRot[static_cast<size_t>(k)] += r[k];
|
||||
}
|
||||
for (float& v : meanRot) v /= static_cast<float>(std::max(J, 1));
|
||||
std::vector<float> motion(static_cast<size_t>(d), 0.f);
|
||||
gemm(meanRot.data(), motionW_.data(), motion.data(), 1, 6, d);
|
||||
addBias(motion.data(), motionB_.data(), 1, d);
|
||||
std::vector<float> fusedIn(static_cast<size_t>(d * 3));
|
||||
for (int i = 0; i < d; ++i) {
|
||||
fusedIn[static_cast<size_t>(i)] = dmiPool[static_cast<size_t>(i)];
|
||||
fusedIn[static_cast<size_t>(d + i)] = motion[static_cast<size_t>(i)];
|
||||
fusedIn[static_cast<size_t>(2 * d + i)] = srcPool[static_cast<size_t>(i)];
|
||||
}
|
||||
std::vector<float> fused(static_cast<size_t>(d));
|
||||
gemm(fusedIn.data(), fuseW_.data(), fused.data(), 1, d * 3, d);
|
||||
addBias(fused.data(), fuseB_.data(), 1, d);
|
||||
for (int i = 0; i < d; ++i) temporal[static_cast<size_t>(t * d + i)] = fused[static_cast<size_t>(i)];
|
||||
}
|
||||
|
||||
std::vector<float> memory = temporal;
|
||||
for (const auto& layer : enc_)
|
||||
transformerLayer(memory, T, d, heads, layer.wq, layer.wk, layer.wv, layer.wo, layer.w1, layer.b1, layer.w2,
|
||||
layer.b2, layer.ln1g, layer.ln1b, layer.ln2g, layer.ln2b, nullptr, 0);
|
||||
|
||||
std::vector<float> queries(static_cast<size_t>(T * d));
|
||||
for (int t = 0; t < T; ++t)
|
||||
for (int i = 0; i < d; ++i)
|
||||
queries[static_cast<size_t>(t * d + i)] =
|
||||
tgtPool[static_cast<size_t>(i)] + temporal[static_cast<size_t>(t * d + i)];
|
||||
for (const auto& layer : dec_)
|
||||
transformerLayer(queries, T, d, heads, layer.wq, layer.wk, layer.wv, layer.wo, layer.w1, layer.b1, layer.w2,
|
||||
layer.b2, layer.ln1g, layer.ln1b, layer.ln2g, layer.ln2b, &memory, T);
|
||||
|
||||
std::vector<float> out(static_cast<size_t>(T * J * 6));
|
||||
for (int t = 0; t < T; ++t) {
|
||||
std::vector<float> delta(6, 0.f);
|
||||
gemm(queries.data() + static_cast<size_t>(t * d), outW_.data(), delta.data(), 1, d, 6);
|
||||
addBias(delta.data(), outB_.data(), 1, 6);
|
||||
for (int j = 0; j < J; ++j) {
|
||||
const float* src = features.sourceRot6d.data() + static_cast<size_t>((t * J + j) * 6);
|
||||
float* dst = out.data() + static_cast<size_t>((t * J + j) * 6);
|
||||
for (int k = 0; k < 6; ++k) dst[k] = src[k] + 0.05f * delta[static_cast<size_t>(k)];
|
||||
}
|
||||
}
|
||||
return Result<std::vector<float>>::success(std::move(out));
|
||||
}
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,49 @@
|
||||
#pragma once
|
||||
|
||||
#include "animation/tensor/SmrFeatures.h"
|
||||
#include "common/Result.h"
|
||||
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
/** @brief Hyperparameters for the builtin MeshRet-style graph. */
|
||||
struct SmrMeshRetConfig {
|
||||
int latentDim = 128;
|
||||
int ffSize = 256;
|
||||
int numLayers = 2;
|
||||
int numHeads = 4;
|
||||
int seed = 9923;
|
||||
};
|
||||
|
||||
/**
|
||||
* @brief Builtin MeshRet-style PointNet + Transformer retarget (links eve::tensor).
|
||||
* @thread Owner thread; not re-entrant.
|
||||
*/
|
||||
class SmrMeshRetNet {
|
||||
public:
|
||||
explicit SmrMeshRetNet(SmrMeshRetConfig config = {});
|
||||
|
||||
[[nodiscard]] const SmrMeshRetConfig& config() const { return config_; }
|
||||
|
||||
/** @brief Owning target rot6d [T*J*6], or structured failure. */
|
||||
[[nodiscard]] Result<std::vector<float>> forward(const SmrFeatureBatch& features) const;
|
||||
|
||||
private:
|
||||
struct LayerWeights {
|
||||
std::vector<float> wq, wk, wv, wo, w1, b1, w2, b2, ln1g, ln1b, ln2g, ln2b;
|
||||
};
|
||||
|
||||
SmrMeshRetConfig config_;
|
||||
std::vector<float> geomW1_, geomB1_, geomW2_, geomB2_;
|
||||
std::vector<float> dmiW1_, dmiB1_, dmiW2_, dmiB2_;
|
||||
std::vector<float> motionW_, motionB_;
|
||||
std::vector<float> fuseW_, fuseB_;
|
||||
std::vector<float> outW_, outB_;
|
||||
std::vector<LayerWeights> enc_;
|
||||
std::vector<LayerWeights> dec_;
|
||||
|
||||
void initWeights();
|
||||
};
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,101 @@
|
||||
#include "animation/tensor/SmrNeuralProvider.h"
|
||||
|
||||
#include "animation/AnimClip.h"
|
||||
#include "animation/AnimSkeleton.h"
|
||||
#include "common/Diagnostic.h"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
std::string SmrNeuralProvider::name() const { return loadedPath_.empty() ? "tensor-meshret" : "onnx"; }
|
||||
|
||||
bool SmrNeuralProvider::isReady(const AnimRetargetProfile* profile) const {
|
||||
if (!profile) return true;
|
||||
if (!profile->getNeuralRetargetEnabled()) return false;
|
||||
const std::string& backend = profile->getNeuralBackend();
|
||||
if (backend == "onnx") return ensureOnnx(*profile).ok();
|
||||
if (backend == "tensor") return true;
|
||||
if (!profile->getNeuralModelPath().empty()) return ensureOnnx(*profile).ok();
|
||||
return true;
|
||||
}
|
||||
|
||||
Result<void> SmrNeuralProvider::ensureOnnx(const AnimRetargetProfile& profile) const {
|
||||
const std::string& path = profile.getNeuralModelPath();
|
||||
if (path.empty()) {
|
||||
return Result<void>::failure(
|
||||
Diagnostic::error(DiagnosticCode::PreconditionViolation, "SmrNeuralProvider: empty ONNX path"));
|
||||
}
|
||||
if (modelLoaded_ && loadedPath_ == path) return Result<void>::success();
|
||||
auto loaded = onnx_.loadFile(path);
|
||||
if (!loaded.ok()) return loaded;
|
||||
loadedPath_ = path;
|
||||
modelLoaded_ = true;
|
||||
return Result<void>::success();
|
||||
}
|
||||
|
||||
Result<SmrNeuralResult> SmrNeuralProvider::applyRot6d(const SmrNeuralRequest& request, const SmrFeatureBatch& features,
|
||||
const std::vector<float>& rot6d,
|
||||
const std::string& backend) const {
|
||||
if (!request.targetClip || !request.targetSkeleton) {
|
||||
return Result<SmrNeuralResult>::failure(
|
||||
Diagnostic::error(DiagnosticCode::InvalidArgument, "SmrNeuralProvider: null clip/skeleton"));
|
||||
}
|
||||
if (static_cast<int>(rot6d.size()) < features.frames * features.joints * 6) {
|
||||
return Result<SmrNeuralResult>::failure(
|
||||
Diagnostic::error(DiagnosticCode::InvariantViolation, "SmrNeuralProvider: rot6d size mismatch"));
|
||||
}
|
||||
|
||||
AnimClip& clip = *request.targetClip;
|
||||
const float duration = clip.getDuration();
|
||||
for (int joint = 0; joint < features.joints; ++joint) {
|
||||
clip.clearTrack(joint);
|
||||
for (int frame = 0; frame < features.frames; ++frame) {
|
||||
const float time = duration <= 0.f ? 0.f
|
||||
: duration * static_cast<float>(frame) /
|
||||
static_cast<float>(std::max(features.frames - 1, 1));
|
||||
const float* r6 = rot6d.data() + static_cast<size_t>((frame * features.joints + joint) * 6);
|
||||
float qx = 0.f, qy = 0.f, qz = 0.f, qw = 1.f;
|
||||
rot6dToQuat(r6, qx, qy, qz, qw);
|
||||
clip.addRotationKey(joint, time, qx, qy, qz, qw);
|
||||
}
|
||||
}
|
||||
|
||||
SmrNeuralResult result;
|
||||
result.framesWritten = features.frames;
|
||||
result.backend = backend;
|
||||
result.sensorCount = static_cast<size_t>(features.sensors);
|
||||
result.pairCount = static_cast<size_t>(features.pairs);
|
||||
return Result<SmrNeuralResult>::success(std::move(result));
|
||||
}
|
||||
|
||||
Result<SmrNeuralResult> SmrNeuralProvider::retarget(const SmrNeuralRequest& request) {
|
||||
if (!request.sourceClip || !request.targetClip || !request.sourceSkeleton || !request.targetSkeleton ||
|
||||
!request.profile) {
|
||||
return Result<SmrNeuralResult>::failure(
|
||||
Diagnostic::error(DiagnosticCode::InvalidArgument, "SmrNeuralProvider.retarget: null argument"));
|
||||
}
|
||||
|
||||
SmrFeatureBatch features = buildSmrFeatures(*request.sourceClip, request.sourceSkeleton, request.targetSkeleton,
|
||||
request.sourceSkin, request.targetSkin, 2, 4);
|
||||
|
||||
const std::string& backendHint = request.profile->getNeuralBackend();
|
||||
const bool preferOnnx =
|
||||
backendHint == "onnx" || (backendHint != "tensor" && !request.profile->getNeuralModelPath().empty());
|
||||
if (preferOnnx) {
|
||||
auto ready = ensureOnnx(*request.profile);
|
||||
if (ready.ok()) {
|
||||
auto ran = onnx_.run(features);
|
||||
if (ran.ok()) return applyRot6d(request, features, ran.value(), "onnx");
|
||||
if (backendHint == "onnx") return Result<SmrNeuralResult>::failure(ran.status());
|
||||
} else if (backendHint == "onnx") {
|
||||
return Result<SmrNeuralResult>::failure(ready.status());
|
||||
}
|
||||
}
|
||||
|
||||
auto ran = net_.forward(features);
|
||||
if (!ran.ok()) return Result<SmrNeuralResult>::failure(ran.status());
|
||||
return applyRot6d(request, features, ran.value(), "tensor-meshret");
|
||||
}
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,51 @@
|
||||
#pragma once
|
||||
|
||||
#include "animation/AnimSmrNeural.h"
|
||||
#include "animation/tensor/SmrFeatures.h"
|
||||
#include "animation/tensor/SmrMeshRetNet.h"
|
||||
#include "animation/tensor/SmrOnnxRunner.h"
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
/**
|
||||
* @brief Capability provider selecting ONNX MeshRet or builtin tensor MeshRet.
|
||||
* @ownership Owned by AnimationTensor; registered via cap::provide until revoke.
|
||||
* @thread Owner/composition thread only; no callbacks while holding locks.
|
||||
*/
|
||||
class SmrNeuralProvider final : public ISmrNeuralRetarget {
|
||||
public:
|
||||
SmrNeuralProvider() = default;
|
||||
|
||||
[[nodiscard]] std::string name() const override;
|
||||
|
||||
/**
|
||||
* @brief True when ONNX is loadable for the profile path, or builtin MeshRet is usable.
|
||||
* @param profile Borrowed for the call only; nullptr means not ready.
|
||||
* @ownership Does not retain profile.
|
||||
* @lifetime profile must outlive this call.
|
||||
* @thread Owner thread.
|
||||
*/
|
||||
[[nodiscard]] bool isReady(const AnimRetargetProfile* profile) const override;
|
||||
|
||||
/**
|
||||
* @brief Run inference and rewrite request.targetClip local rotations.
|
||||
* @lifetime See SmrNeuralRequest; pointers borrowed for this call only.
|
||||
* @thread Owner thread; not re-entrant on targetClip.
|
||||
*/
|
||||
[[nodiscard]] Result<SmrNeuralResult> retarget(const SmrNeuralRequest& request) override;
|
||||
|
||||
private:
|
||||
mutable SmrOnnxRunner onnx_;
|
||||
mutable std::string loadedPath_;
|
||||
mutable bool modelLoaded_ = false;
|
||||
SmrMeshRetNet net_{};
|
||||
|
||||
[[nodiscard]] Result<void> ensureOnnx(const AnimRetargetProfile& profile) const;
|
||||
[[nodiscard]] Result<SmrNeuralResult> applyRot6d(const SmrNeuralRequest& request, const SmrFeatureBatch& features,
|
||||
const std::vector<float>& rot6d, const std::string& backend) const;
|
||||
};
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,87 @@
|
||||
#include "animation/tensor/SmrOnnxRunner.h"
|
||||
|
||||
#include "common/Diagnostic.h"
|
||||
|
||||
#include <cstring>
|
||||
#include <fstream>
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
namespace {
|
||||
|
||||
tensor::OnnxNamedTensor makeF32(const char* name, std::vector<int64_t> shape, const std::vector<float>& data) {
|
||||
tensor::OnnxNamedTensor named;
|
||||
named.name = name;
|
||||
named.tensor.element = tensor::OnnxElement::Float32;
|
||||
named.tensor.shape = std::move(shape);
|
||||
named.tensor.bytes.resize(data.size() * sizeof(float));
|
||||
if (!data.empty()) std::memcpy(named.tensor.bytes.data(), data.data(), named.tensor.bytes.size());
|
||||
return named;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Result<void> SmrOnnxRunner::load(std::span<const uint8_t> bytes) {
|
||||
auto loaded = tensor::OnnxModel::load(bytes);
|
||||
if (!loaded.ok()) return Result<void>::failure(loaded.status());
|
||||
model_ = std::move(loaded.value());
|
||||
return Result<void>::success();
|
||||
}
|
||||
|
||||
Result<void> SmrOnnxRunner::loadFile(const std::string& path) {
|
||||
std::ifstream file(path, std::ios::binary | std::ios::ate);
|
||||
if (!file) {
|
||||
return Result<void>::failure(Diagnostic::error(DiagnosticCode::NotFound, "SmrOnnxRunner: open failed", path));
|
||||
}
|
||||
const auto size = static_cast<std::streamoff>(file.tellg());
|
||||
if (size <= 0) {
|
||||
return Result<void>::failure(
|
||||
Diagnostic::error(DiagnosticCode::InvalidArgument, "SmrOnnxRunner: empty file", path));
|
||||
}
|
||||
std::vector<uint8_t> bytes(static_cast<size_t>(size));
|
||||
file.seekg(0);
|
||||
if (!file.read(reinterpret_cast<char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()))) {
|
||||
return Result<void>::failure(Diagnostic::error(DiagnosticCode::Failed, "SmrOnnxRunner: read failed", path));
|
||||
}
|
||||
return load(bytes);
|
||||
}
|
||||
|
||||
tensor::OnnxModelInfo SmrOnnxRunner::info() const {
|
||||
if (!model_) return {};
|
||||
return model_->info();
|
||||
}
|
||||
|
||||
Result<std::vector<float>> SmrOnnxRunner::run(const SmrFeatureBatch& features) const {
|
||||
if (!model_) {
|
||||
return Result<std::vector<float>>::failure(
|
||||
Diagnostic::error(DiagnosticCode::PreconditionViolation, "SmrOnnxRunner: no model"));
|
||||
}
|
||||
if (features.frames <= 0 || features.joints <= 0) {
|
||||
return Result<std::vector<float>>::failure(
|
||||
Diagnostic::error(DiagnosticCode::InvalidArgument, "SmrOnnxRunner: empty features"));
|
||||
}
|
||||
const int64_t T = features.frames;
|
||||
const int64_t J = features.joints;
|
||||
const int64_t S = std::max(features.sensors, 1);
|
||||
const int64_t P = std::max(features.pairs, 1);
|
||||
|
||||
std::vector<tensor::OnnxNamedTensor> feeds;
|
||||
feeds.push_back(makeF32("source_rot6d", {1, T, J, 6}, features.sourceRot6d));
|
||||
feeds.push_back(makeF32("source_geom", {1, S, 7}, features.sourceGeom));
|
||||
feeds.push_back(makeF32("target_geom", {1, S, 7}, features.targetGeom));
|
||||
feeds.push_back(makeF32("source_dmi", {1, T, P, 10}, features.sourceDmi));
|
||||
|
||||
const std::string outName = "target_rot6d";
|
||||
auto ran = model_->run(feeds, std::span<const std::string>(&outName, 1));
|
||||
if (!ran.ok()) return Result<std::vector<float>>::failure(ran.status());
|
||||
if (ran.value().empty()) {
|
||||
return Result<std::vector<float>>::failure(
|
||||
Diagnostic::error(DiagnosticCode::Failed, "SmrOnnxRunner: missing output"));
|
||||
}
|
||||
const auto& bytes = ran.value()[0].tensor.bytes;
|
||||
std::vector<float> out(bytes.size() / sizeof(float));
|
||||
if (!out.empty()) std::memcpy(out.data(), bytes.data(), out.size() * sizeof(float));
|
||||
return Result<std::vector<float>>::success(std::move(out));
|
||||
}
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,37 @@
|
||||
#pragma once
|
||||
|
||||
#include "animation/tensor/SmrFeatures.h"
|
||||
#include "common/Result.h"
|
||||
#include "tensor/OnnxModel.h"
|
||||
|
||||
#include <memory>
|
||||
#include <span>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace eve::animation {
|
||||
|
||||
/**
|
||||
* @brief MeshRet-compatible ONNX runner on CPU via eve::tensor::OnnxModel.
|
||||
*
|
||||
* I/O contract from scripts/export_meshret_onnx.py:
|
||||
* - inputs: source_rot6d [1,T,J,6], source_geom [1,S,7], target_geom [1,S,7], source_dmi [1,T,P,10]
|
||||
* - output: target_rot6d [1,T,J,6]
|
||||
*
|
||||
* @ownership Owns the loaded OnnxModel until load/reload or destruction.
|
||||
* @lifetime Model bytes are copied at load; feature tensors are borrowed only for run().
|
||||
* @thread Owner thread; not re-entrant.
|
||||
*/
|
||||
class SmrOnnxRunner {
|
||||
public:
|
||||
[[nodiscard]] Result<void> load(std::span<const uint8_t> bytes);
|
||||
[[nodiscard]] Result<void> loadFile(const std::string& path);
|
||||
[[nodiscard]] bool isLoaded() const { return static_cast<bool>(model_); }
|
||||
[[nodiscard]] tensor::OnnxModelInfo info() const;
|
||||
[[nodiscard]] Result<std::vector<float>> run(const SmrFeatureBatch& features) const;
|
||||
|
||||
private:
|
||||
std::unique_ptr<tensor::OnnxModel> model_;
|
||||
};
|
||||
|
||||
} // namespace eve::animation
|
||||
@@ -0,0 +1,120 @@
|
||||
#include "zeroerr/assert.h"
|
||||
#include "zeroerr/unittest.h"
|
||||
|
||||
#include "animation/AnimClip.h"
|
||||
#include "animation/AnimPose.h"
|
||||
#include "animation/AnimSkeleton.h"
|
||||
#include "animation/AnimSmr.h"
|
||||
#include "animation/AnimSmrNeural.h"
|
||||
#include "animation/tensor/AnimationTensor.h"
|
||||
#include "animation/tensor/SmrFeatures.h"
|
||||
#include "common/Capability.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <memory>
|
||||
|
||||
using namespace eve::animation;
|
||||
|
||||
namespace {
|
||||
|
||||
void buildArmature(AnimSkeleton& skel, float torsoHeight, float upperArm, float lowerArm) {
|
||||
const int root = skel.addBone("Hips");
|
||||
const int chest = skel.addBone("Spine", root);
|
||||
const int shoulder = skel.addBone("LeftShoulder", chest);
|
||||
const int elbow = skel.addBone("LeftElbow", shoulder);
|
||||
const int hand = skel.addBone("LeftHand", elbow);
|
||||
skel.setBindPosition(chest, 0.f, torsoHeight, 0.f);
|
||||
skel.setBindPosition(shoulder, 0.15f, 0.f, 0.f);
|
||||
skel.setBindPosition(elbow, upperArm, 0.f, 0.f);
|
||||
skel.setBindPosition(hand, lowerArm, 0.f, 0.f);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("animation.smr.neural.rot6dRoundTrip") {
|
||||
float out[6] = {};
|
||||
quatToRot6d(0.f, 0.f, 0.70710678f, 0.70710678f, out);
|
||||
float qx = 0.f, qy = 0.f, qz = 0.f, qw = 1.f;
|
||||
rot6dToQuat(out, qx, qy, qz, qw);
|
||||
const float dot = std::fabs(qx * 0.f + qy * 0.f + qz * 0.70710678f + qw * 0.70710678f);
|
||||
CHECK(dot > 0.999f);
|
||||
}
|
||||
|
||||
TEST_CASE("animation.smr.neural.featureBatchShapes") {
|
||||
AnimSkeleton source;
|
||||
AnimSkeleton target;
|
||||
buildArmature(source, 1.0f, 0.28f, 0.28f);
|
||||
buildArmature(target, 1.8f, 0.70f, 0.70f);
|
||||
|
||||
AnimClip clip("reach");
|
||||
clip.setDuration(0.1f);
|
||||
clip.setSampleRate(10.f);
|
||||
clip.addRotationKey(source.findBone("LeftShoulder"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
clip.addRotationKey(source.findBone("LeftElbow"), 0.05f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
|
||||
const SmrFeatureBatch batch = buildSmrFeatures(clip, &source, &target, nullptr, nullptr, 2, 4);
|
||||
CHECK(batch.frames >= 1);
|
||||
CHECK(batch.joints == target.getBoneCount());
|
||||
CHECK(batch.sensors >= 1);
|
||||
CHECK(static_cast<int>(batch.sourceRot6d.size()) == batch.frames * batch.joints * 6);
|
||||
CHECK(static_cast<int>(batch.sourceGeom.size()) == batch.sensors * 7);
|
||||
CHECK(static_cast<int>(batch.targetGeom.size()) == batch.sensors * 7);
|
||||
CHECK(static_cast<int>(batch.sourceDmi.size()) == batch.frames * std::max(batch.pairs, 1) * 10);
|
||||
}
|
||||
|
||||
TEST_CASE("animation.smr.neural.tensorBackendRewritesClip") {
|
||||
AnimationTensor* module = AnimationTensor::create();
|
||||
CHECK(module != nullptr);
|
||||
CHECK(eve::cap::query<ISmrNeuralRetarget>() != nullptr);
|
||||
|
||||
AnimSkeleton source;
|
||||
AnimSkeleton target;
|
||||
buildArmature(source, 1.0f, 0.28f, 0.28f);
|
||||
buildArmature(target, 1.8f, 0.70f, 0.70f);
|
||||
|
||||
AnimClip clip("reach");
|
||||
clip.setDuration(0.f);
|
||||
clip.setSampleRate(30.f);
|
||||
clip.addRotationKey(source.findBone("LeftShoulder"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
clip.addRotationKey(source.findBone("LeftElbow"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
|
||||
AnimRetargetProfile neural;
|
||||
neural.setSkinnedInteractionPreserve(true);
|
||||
neural.setNeuralRetargetEnabled(true);
|
||||
neural.setNeuralBackend("tensor");
|
||||
neural.addInteractionIkChain("LeftShoulder", "LeftElbow", "LeftHand");
|
||||
|
||||
std::unique_ptr<AnimClip> out(clip.retargetWithProfile(&source, &target, &neural));
|
||||
CHECK(out != nullptr);
|
||||
CHECK(neural.getNeuralInferenceCount() >= 1);
|
||||
CHECK(neural.getInteractionCorrectionCount() >= 1);
|
||||
CHECK(out->getRotationKeyCount(target.findBone("LeftElbow")) >= 1);
|
||||
}
|
||||
|
||||
TEST_CASE("animation.smr.neural.fallsBackToClassicalWhenDisabled") {
|
||||
AnimationTensor* module = AnimationTensor::create();
|
||||
CHECK(module != nullptr);
|
||||
|
||||
AnimSkeleton source;
|
||||
AnimSkeleton target;
|
||||
buildArmature(source, 1.0f, 0.28f, 0.28f);
|
||||
buildArmature(target, 1.8f, 0.70f, 0.70f);
|
||||
|
||||
AnimClip clip("reach");
|
||||
clip.setDuration(0.f);
|
||||
clip.setSampleRate(30.f);
|
||||
clip.addRotationKey(source.findBone("LeftShoulder"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
clip.addRotationKey(source.findBone("LeftElbow"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
|
||||
AnimRetargetProfile classical;
|
||||
classical.setSkinnedInteractionPreserve(true);
|
||||
classical.setNeuralRetargetEnabled(false);
|
||||
classical.setInteractionContactThreshold(0.4f);
|
||||
classical.setInteractionCorrectionWeight(1.f);
|
||||
classical.addInteractionIkChain("LeftShoulder", "LeftElbow", "LeftHand");
|
||||
|
||||
std::unique_ptr<AnimClip> out(clip.retargetWithProfile(&source, &target, &classical));
|
||||
CHECK(out != nullptr);
|
||||
CHECK(classical.getNeuralInferenceCount() == 0);
|
||||
CHECK(classical.getInteractionCorrectionCount() >= 1);
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
#include "zeroerr/assert.h"
|
||||
#include "zeroerr/unittest.h"
|
||||
|
||||
#include "animation/AnimClip.h"
|
||||
#include "animation/AnimPose.h"
|
||||
#include "animation/AnimSkeleton.h"
|
||||
#include "animation/AnimSmr.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <memory>
|
||||
|
||||
using namespace eve::animation;
|
||||
|
||||
namespace {
|
||||
|
||||
void buildArmature(AnimSkeleton& skel, float torsoHeight, float upperArm, float lowerArm) {
|
||||
const int root = skel.addBone("Hips");
|
||||
const int chest = skel.addBone("Spine", root);
|
||||
const int shoulder = skel.addBone("LeftShoulder", chest);
|
||||
const int elbow = skel.addBone("LeftElbow", shoulder);
|
||||
const int hand = skel.addBone("LeftHand", elbow);
|
||||
skel.setBindPosition(chest, 0.f, torsoHeight, 0.f);
|
||||
skel.setBindPosition(shoulder, 0.15f, 0.f, 0.f);
|
||||
skel.setBindPosition(elbow, upperArm, 0.f, 0.f);
|
||||
skel.setBindPosition(hand, lowerArm, 0.f, 0.f);
|
||||
}
|
||||
|
||||
float handChestDistance(const AnimSkeleton& skel, const AnimClip& clip, float time) {
|
||||
AnimPose pose;
|
||||
clip.sample(time, &pose, &skel);
|
||||
pose.computeWorld(&skel);
|
||||
const int chest = skel.findBone("Spine");
|
||||
const int hand = skel.findBone("LeftHand");
|
||||
const float dx = pose.getWorldPositionX(hand) - pose.getWorldPositionX(chest);
|
||||
const float dy = pose.getWorldPositionY(hand) - pose.getWorldPositionY(chest);
|
||||
const float dz = pose.getWorldPositionZ(hand) - pose.getWorldPositionZ(chest);
|
||||
return std::sqrt(dx * dx + dy * dy + dz * dz);
|
||||
}
|
||||
|
||||
bool rotationKeysDiffer(const AnimClip& a, const AnimClip& b, int bone) {
|
||||
if (a.getRotationKeyCount(bone) == 0 || b.getRotationKeyCount(bone) == 0) return false;
|
||||
const float ax = a.getRotationKeyX(bone, 0);
|
||||
const float ay = a.getRotationKeyY(bone, 0);
|
||||
const float az = a.getRotationKeyZ(bone, 0);
|
||||
const float aw = a.getRotationKeyW(bone, 0);
|
||||
const float bx = b.getRotationKeyX(bone, 0);
|
||||
const float by = b.getRotationKeyY(bone, 0);
|
||||
const float bz = b.getRotationKeyZ(bone, 0);
|
||||
const float bw = b.getRotationKeyW(bone, 0);
|
||||
const float dot = std::fabs(ax * bx + ay * by + az * bz + aw * bw);
|
||||
return dot < 0.999f;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("animation.smr.sensorCloudTagsBodyParts") {
|
||||
AnimSkeleton skel;
|
||||
buildArmature(skel, 1.0f, 0.3f, 0.3f);
|
||||
const AnimSmrSensorCloud cloud = AnimSmrSensorCloud::fromSkeleton(&skel);
|
||||
CHECK(cloud.getSensorCount() > 0);
|
||||
|
||||
bool sawTorso = false;
|
||||
bool sawArm = false;
|
||||
for (int i = 0; i < cloud.getSensorCount(); ++i) {
|
||||
const AnimSmrBodyPart part = cloud.getSensorPart(i);
|
||||
if (part == AnimSmrBodyPart::Torso) sawTorso = true;
|
||||
if (part == AnimSmrBodyPart::Arm) sawArm = true;
|
||||
}
|
||||
CHECK(sawTorso);
|
||||
CHECK(sawArm);
|
||||
}
|
||||
|
||||
TEST_CASE("animation.smr.preservesHandTorsoProximityAcrossScale") {
|
||||
AnimSkeleton source;
|
||||
AnimSkeleton target;
|
||||
buildArmature(source, 1.0f, 0.28f, 0.28f);
|
||||
buildArmature(target, 1.8f, 0.70f, 0.70f);
|
||||
|
||||
// Curl the arm so the hand is near the torso on the short source.
|
||||
AnimClip clip("reach");
|
||||
clip.setDuration(0.f);
|
||||
clip.setSampleRate(30.f);
|
||||
clip.addRotationKey(source.findBone("LeftShoulder"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
clip.addRotationKey(source.findBone("LeftElbow"), 0.f, 0.f, 0.f, 0.70710678f, 0.70710678f);
|
||||
|
||||
const float sourceGap = handChestDistance(source, clip, 0.f);
|
||||
|
||||
AnimRetargetProfile fkOnly;
|
||||
fkOnly.setSkinnedInteractionPreserve(false);
|
||||
std::unique_ptr<AnimClip> fk(clip.retargetWithProfile(&source, &target, &fkOnly));
|
||||
const float fkGap = handChestDistance(target, *fk, 0.f);
|
||||
|
||||
AnimRetargetProfile smr;
|
||||
smr.setSkinnedInteractionPreserve(true);
|
||||
smr.setInteractionContactThreshold(std::max(sourceGap * 1.25f, 0.35f));
|
||||
smr.setInteractionCorrectionWeight(1.f);
|
||||
smr.addInteractionIkChain("LeftShoulder", "LeftElbow", "LeftHand");
|
||||
std::unique_ptr<AnimClip> refined(clip.retargetWithProfile(&source, &target, &smr));
|
||||
const float smrGap = handChestDistance(target, *refined, 0.f);
|
||||
|
||||
CHECK(smr.getInteractionCorrectionCount() >= 1);
|
||||
CHECK(fkGap > sourceGap);
|
||||
// Either the world-space gap shrinks, or the IK rewrite clearly changed chain rotations.
|
||||
const bool gapImproved = smrGap + 1e-4f < fkGap;
|
||||
const bool keysChanged = rotationKeysDiffer(*fk, *refined, target.findBone("LeftElbow")) ||
|
||||
rotationKeysDiffer(*fk, *refined, target.findBone("LeftShoulder"));
|
||||
const bool refinedDiffers = gapImproved || keysChanged;
|
||||
CHECK(refinedDiffers);
|
||||
}
|
||||
Reference in new issue
Block a user