Skip to content
← Engineering Portfolio

Crafty Apes VFX · Research & development

Asset Generation & Texturing

From a reference image to a 3D asset, with a focus on controllable, consistent texturing.

I led the implementation, adapting open-source tools into artist-facing ComfyUI workflows and DCC plugins.

Starting with image-to-3D

From an image reference to starting geometry: a selection of generated objects and characters, followed by our work on consistent textures.

Objects, creatures & environments
Image reference → 3D mesh
Character generation studies

Building the texture

A look at the working pipeline.

Mesh + reference image → generated views → textured asset
  1. 01

    Render

    Capture 6–8 views of the mesh, with normal and position maps to guide the generation.

  2. 02

    Generate

    Synthesize matching views from the reference image, then upscale them.

  3. 03

    Bake

    Project the views onto a UV map, fill uncovered areas and apply the texture.

Character texture study

Geometry, reference and textured result.

Character texture study showing an untextured man in a cap, the image and geometry reference, and the textured character.

Geometry meets appearance

Two approaches, different trade-offs in our tests. Select an image to look closer.

Eight generated views of a zombie bust from map-conditioned generation, with softer details and uneven colour.

Map-conditioned generation

Follows the mesh well, but can lose detail and colour fidelity.

Eight generated views of the same zombie bust from image-guided editing, showing richer surface detail and variations between viewpoints.

Image-guided editing

Richer detail, but less reliable geometry, consistency between views and separation of lighting from texture.

Inside the research

What we encountered, what we tried, and where we want to go next.

01 · Keeping the geometry without losing the imageFinding

Map-conditioned generation models followed the rendered normal and position maps reasonably well in our tests. The trade-off was appearance: newly generated views often softened detail and shifted colours away from the original reference.

That made geometric alignment only half the problem. A useful texture also needs to preserve the visual identity that made the reference worth using.

02 · Better-looking views, less predictable controlExperiment

We tested image-editing models, using depth maps to guide new viewpoints. Some character examples retained much more of the reference’s detail, but poses and camera angles did not always match the conditioning maps.

Independently generated views could disagree on unseen surfaces, and lighting or shadows could become baked into the texture. Buildings, cars, birds and views from underneath were particularly difficult in these tests. The API-only route also did not meet our requirement to process show data on our own servers.

03 · Colour is not a complete materialOpen issue

A convincing colour image is not enough for relighting a 3D asset. We also need dependable material properties, especially metallic and roughness maps, and albedo separated from illumination.

The initial material output lacked quality in our tests. Separate decomposition models offered another route, but processing each view independently introduced inconsistencies across the asset.

04 · Generate first, refine secondProposed approach

One direction is to let a map-conditioned model generate geometrically aligned views, then use a fine-tuned image-editing model to refine each view against the original reference. Rendered normal, position or depth maps could provide additional guidance.

The proposed training pairs would combine rough generated views with high-quality renders of the same CG assets. The aim is sharper, more faithful textures without losing alignment; consistency between independently refined views remains an open question.

05 · Treat a turntable as a videoProposed approach

Another direction is to train a video LoRA on a reference image and a 360° depth/normal render, generating one textured rotation rather than separate images. Frames from that rotation would become the views used for baking.

We are considering video-based material decomposition. The hypothesis is that temporal coherence could help both appearance and material consistency. Geometry alignment, output resolution and adapting the decomposition to metallic/roughness maps still need validation.

06 · Improve the bake, then repeatProposed approach

Poorly covered regions are another source of seams and missing detail. We want to explore using mesh and UV adjacency to propagate texture from well-observed neighbouring areas, instead of treating each gap independently.

A second idea is an iterative loop: bake, re-render the textured mesh, refine those renders, then bake again. Re-rendering gives every view the same underlying texture; keeping the original image as an anchor should help limit appearance drift. These are research directions, not yet established results.