[REL] ReScalePA: Converter to create smaller units

A picture for the feels first:


This is about changing the scale of units. For that purpose I have written a program that allows you with a single command to create a mod that scales units of the game. You can configure which units and which factor. My default setting is everything but commanders and orbital.
The program works by modifing the model/animation/unit definitions directly and copying them together into a mod.
Currently there is a bug that results in the need to repackage textures with the mod. This means the mod for everything but commanders and orbitals has a size of ~90mb. That means 90s uploading on hosting the game on my rather fast upload of 1mb/s. To get around this you can split the mod (the version on pamm works like that, the textures client mod has all current textures, so it can work with any scale mod together).

To create your own mod with this, read the readme of the project on github:
https://github.com/pamods/ReScalePa

ReScale 0.4x is on pamm as a server mod that is depending on a client mod.
Every player in the game will need to have the client mod as well or they will see white textures. The client mod contains all textures renamed for the changed models from the server mod, this pushes the size of the server mod to ~12mb. But again: Every player in the game will need the client textures mod as well. Download the server mod on pamm, it should download the client mod as well. If you host such a game warn every player in it: they _need_ the textures mod installed and active.

The following warning is not true if you split up the mods like the version on pamm. Just notice that everyone will need the textures client mod.
And just a little red warning again: These mods currently get BIG. As a server mod the host has to upload it on creation of a lobby and all clients download it. Until that upload/download process is finished the game won't even load the planets. This happens very game hosted. There are potential bugfixes that Uber can make to drastically reduce the size to less than 10mb though.
BIG means ~90mb for the version without commanders and orbital.

Technical considerations:
The mod currently scales the following things, I am ofc not 100% sure it does so correctly and there most likely are a few quirks, I've not played a real game with it:

- All papa meshes
- All papa animations

From the unit json files:
- mesh size
- placement size
- event-effect scale (i.e. explosions)
- foundations of buildings
- physics radius
- fx offsets
- headlights offsets
- lamps offsets
- offset on units that determines where to attach to a transport unit

Unit movement speeds as well as vision ranges are unchanged.

If you find weird things please report them.

Bugs affecting this currently:

- units refuse to walk into the teleporters at scale 0.3, so they do not work. Dunno why so far. For this reason I have added the teleporter to the "ignore" list in the conf file of the converter, so it is unscaled. Not sure how interaction with upscaled units would look like, with 0.3x it looks fine: Small unit walks into giant stargate.
- the pgen lights look somehow wrong.
- replays of these games seem to crash the server

The size of the mod is a result of a bug that forces me to rename all models to make them work as a server mod. PA doesn't allow to replace models via mods like this for some reasons. This causes me to repackage all textures as well. So the the big part of the mod is renamed texture files.
There are 2 things that Uber can do to make this mod much smaller:
a) fix the bug, so I can shadow models directly.
b) use gzip to transmit the mod files. the mod zip file is ~30mb instead of the unpacked 90mb.


Screenshots
So here's a few screenshots at a factor of 0.3x. I know 0.3x pushes it into a rather "too small" scenario, but hey I am just showing off that it works. You can use whatever scale you want
Notice I've not changed the size of commanders due to the filesize of the mod. So they are a nice scale on pics like this one:










The unchanged orbital units in the launcher look gigantic, in orbit I like them though:





Teleporters are on the list of not scaled units for the default 0.3x settings as well, as they do not work with 0.3 or 0.5 and I don't know why.

It also works the other way, if you like big things

Looks interesting.

In pa\effects\specs you can find all the decals. They all have a scale, for example "scale":[16,16,1]. They should be easy to fix.

I have ran into this issue before. The only solution that I know of is to just rename the model/unit.

Some effects like the nuke explosion can't be easily scaled. However for most units it's as simple as adding "effect_scale".

Code:

    "died": {
      "audio_cue": "/SE/Death/Sea_large",
      "effect_scale": 2.0,
      "effect_spec": "/pa/effects/specs/default_building_explosion.pfx"
    }

You just want a big fat slobbery hug from me dontcha?!??

No thanks.

Uff. So the first thing to try would be to rename the model and make all unit jsons use the changed models?
Do I understand correctly I can do that via the property model.filename in the json definition of the file?

Also about scaling bones for the animation:
I am have no idea what am I doing at all for these. Currently I just read the 101 editor template from @raevn and decided to scale down the TranslationX/Y/Z of PapaSkeletonSegment. Doesn't look all that good so far. Maybe I am missunderstanding this somewhere?
The template for the papa files that raevn gave me looks like this:

Code:

struct PapaFile;
struct PapaModelInformation;
struct PapaTextureInformation;
struct PapaMeshBinding;
struct PapaBoneMapping;
struct PapaMeshInformation;
struct PapaMaterialGroup;
struct PapaMaterialInformation;
struct PapaMaterialVectorParameter;
struct PapaMatrix3;
struct PapaMatrix4;
struct PapaMatrixRow3;
struct PapaMatrixRow4;
struct PapaVerticesInformation;
struct PapaVertex7;
struct PapaVertex8;
struct PapaVertex10;
struct PapaIndicesInformation;
struct PapaPrimitive0;
struct PapaBonesHeader;
struct PapaSkeletonInformation;
struct PapaSkeletonSegment;
struct PapaBone;
struct PapaRGBA;

struct PapaMatrix4;

struct PapaFile
{
    char Identification[4];
    short Unknown1[2];
    short NumberOfBones;
    short NumberOfTextures;
    short NumberOfVertexBuffers;
    short NumberOfIndexBuffers;
    short NumberOfMaterials;
    short NumberOfMeshes;
    short NumberOfSkeletons;
    short NumberOfModels;
    short Unknown2[4];
    int64 OffsetBonesHeader;
    int64 OffsetTextureInformation;
    int64 OffsetVerticesInformation;
    int64 OffsetIndicesInformation;
    int64 OffsetMaterialInformation;
    int64 OffsetMeshInformation;
    int64 OffsetSkeletonInformation;
    local int64 _OffsetSkeletonHeader = OffsetSkeletonInformation;
    int64 OffsetModelInformation;
    int64 OffsetAnimationInformation;

    if(OffsetModelInformation > -1)
    {
        FSeek(OffsetModelInformation);
        PapaModelInformation Model[NumberOfModels] <optimize=false>;
    };

    if(OffsetTextureInformation > -1)
    {
        FSeek(OffsetTextureInformation);
        PapaTextureInformation TextureInformation[NumberOfTextures] <optimize=false>;
    };

    if(OffsetMeshInformation > -1)
    {
        FSeek(OffsetMeshInformation);
        PapaMeshInformation MeshInformation[NumberOfMeshes] <optimize=false>;
    };

    if(OffsetMaterialInformation > -1)
    {
        FSeek(OffsetMaterialInformation);
        PapaMaterialInformation MaterialInformation[NumberOfMaterials] <optimize=false>;
    };

    if(OffsetVerticesInformation > -1)
    {
        FSeek(OffsetVerticesInformation);
        PapaVerticesInformation Vertices;
    }

    if(OffsetIndicesInformation > -1)
    {
        FSeek(OffsetIndicesInformation);
        PapaIndicesInformation Indices;
    }

    if (OffsetSkeletonInformation > -1)
    {
        FSeek(OffsetSkeletonInformation);
        PapaSkeletonInformation SkeletonInformation;
    };

    if(OffsetBonesHeader > -1)
    {
        FSeek(OffsetBonesHeader);
        PapaBonesHeader Bones(NumberOfBones);
    };
};

struct PapaRGBA
{
    unsigned char r;
    unsigned char g;
    unsigned char b;
    unsigned char a;
};

struct PapaTextureInformation()
{
    char Unknown1[2];
    enum <char> {TF_DXT1 = 4, TF_DXT5 = 6} TextureFormat;
    char MIPS:4; //(TF_DXT1 only)
    char Unknown2:3;
    char SRGB:1 ;
    short Width;
    short Height;
    int64 Length;
    int64 Unknown3; //Always 128
    PapaRGBA RGBAData[Length / 4];
};

struct PapaModelInformation()
{
    int SkeletonData;
    int NumberOfMeshBindings;
    PapaMatrix4 Model2Scene;
    int64 OffsetMeshBinding;

    FSeek(OffsetMeshBinding);
    PapaMeshBinding MeshBinding[NumberOfMeshBindings] <optimize=false>;
};

struct PapaMeshBinding()
{
    int Unknown1;
    int SkeletonSegments;
    PapaMatrix4 Mesh2Model;
    int64 OffsetBoneMappings;

    //Counts from 0 to SkeletonSegments
    if (OffsetBoneMappings > -1) {
        FSeek(OffsetBoneMappings);
        PapaBoneMapping BoneMapping[SkeletonSegments];
    };
};

struct PapaBoneMapping
{
    short SegmentID;
};

struct PapaMeshInformation
{
    int Unknown1; //possibly 2x shorts
    int NumberOfMaterialGroups;
    int64 OffsetMaterialGroup;

    FSeek(OffsetMaterialGroup);
    PapaMaterialGroup MaterialGroup[NumberOfMaterialGroups] <optimize=false>;
};

struct PapaMaterialGroup
{
    short Unknown1;
    short MaterialIndex;
    short FirstIndex;
    short Unknown4;
    int NumberOfPrimitives;
    enum <int> {PRIM_Triangles = 2} PrimitiveType;
};

struct PapaMaterialInformation
{
    short Unknown1;
    short VectorParameters;
    short TexturedParameters;
    short MatrixParameters;
    int64 OffsetMaterialVectorParameter;
    int64 OffsetMaterialTextureParameters;
    int64 OffsetMaterialMatrixParameters;

    local int _Restore = FTell();
    if (OffsetMaterialVectorParameter > 0) {
        FSeek(OffsetMaterialVectorParameter);
        PapaMaterialVectorParameter MaterialVectorParameter[VectorParameters] <optimize=false>;
    };
    FSeek(_Restore);
};

struct PapaMaterialVectorParameter
{
    int Unknown1;
    float Vector[4];
    float Unknown6;
};

struct PapaMatrix3
{
    PapaMatrixRow3 Row1;
    PapaMatrixRow3 Row2;
    PapaMatrixRow3 Row3;
};

struct PapaMatrix4
{
    PapaMatrixRow4 Row1;
    PapaMatrixRow4 Row2;
    PapaMatrixRow4 Row3;
    PapaMatrixRow4 Row4;
};

struct PapaMatrixRow3
{
    float Column1;
    float Column2;
    float Column3;
};

struct PapaMatrixRow4
{
    float Column1;
    float Column2;
    float Column3;
    float Column4;
};

struct PapaVerticesInformation
{
    enum <int> {Position3Normal3Color4TexCoord4 = 7, Position3Weights4bBones4bNormal3TexCoord2 = 8, Position3Normal3Tan3Bin3TexCoord4 = 10} VertexFormat;
    int NumberOfVertices;
    int64 SizeVerticesBlock;
    int64 OffsetVerticesBlock;

    FSeek(OffsetVerticesBlock);

    //Position3Normal3Color4TexCoord4
    if (VertexFormat == 7) {
        PapaVertex7 Vertices[NumberOfVertices];
    }

    //Position3Weights4bBones4bNormal3TexCoord2
    if (VertexFormat == 8) {
        PapaVertex8 Vertices[NumberOfVertices];
    }

    //Position3Normal3Tan3Bin3TexCoord4
    if (VertexFormat == 10) {
        PapaVertex10 Vertices[NumberOfVertices];
    }
};

//For VertexFormat 7
//Position3Normal3Color4TexCoord4
struct PapaVertex7
{
    float PositionX;
    float PositionY;
    float PositionZ;
    float NormalX;
    float NormalY;
    float NormalZ;
    byte Colour[4];
    float U;
    float V;
    float X;
    float Y;
};

//For VertexFormat 8
//Position3Weights4bBones4bNormal3TexCoord2
struct PapaVertex8
{
    float PositionX;
    float PositionY;
    float PositionZ;
    byte Weights[4];
    int BoneSegmentIndex; //should be 4 x bytes?
    float NormalX;
    float NormalY;
    float NormalZ;
    float TexCoord1;
    float TexCoord2;
};

//For VertexFormat 10
//Position3Normal3Tan3Bin3TexCoord4
struct PapaVertex10
{
    float PositionX;
    float PositionY;
    float PositionZ;
    float NormalX;
    float NormalY;
    float NormalZ;
    float TanX;
    float TanY;
    float TanZ;
    float BinX;
    float BinY;
    float BinZ;
    float X;
    float Y;
    float U;
    float V;
};

struct PapaIndicesInformation
{
    enum <int> {IF_UInt16 = 0} IndexFormat;
    int NumberOfIndices;
    int64 SizeIndicesBlock;
    int64 OffsetIndicesBlock;

    //IF_UInt16
    if (IndexFormat == 0) {
        FSeek(OffsetIndicesBlock);
        PapaPrimitive0 Primitives[NumberOfIndices/3];
    };
};

struct PapaPrimitive0
{
    short VertexA;
    short VertexB;
    short VertexC;
};

struct PapaSkeletonInformation
{
    short NumberOfBones;
    short Unknown[3];
    int64 Offset;
    PapaSkeletonSegment SkeletonSegment[NumberOfBones];
};

struct PapaSkeletonSegment
{
    short BoneIndex;
    short ParentSegmentIndex;
    float TranslationX;
    float TranslationY;
    float TranslationZ;
    PapaMatrix3 ShearScale;
    PapaMatrix4 Bind2Bone;
    float BoneOffsetPositionX;  //?
    float BoneOffsetPositionY;  //?
    float BoneOffsetPositionZ;  //?
    float Unknown;              //always 1
};

struct PapaBonesHeader(int NumberOfBones)
{
    PapaBone Bones[NumberOfBones] <optimize=false>;
};

struct PapaBone
{
    int64 LengthOfBoneName;
    int64 OffsetBoneName;

    local int _Restore = FTell();
    FSeek(OffsetBoneName);
    char BoneName[LengthOfBoneName];

    FSeek(_Restore);
};

PapaFile file;

Also does somebody know the meaning of the property physics.radius in the unit jsons? Do I need to care for those as well?

Looking at the json file other potentially interesting things, maybe somebody can comment on them:

attachable.offsets.head and similar
guard_radius
walk_speed?
to actually scale down everything I might also want to scale down the vision radius which is recoon.observer.items.radius ?

Yeah, that's what I would try first. However that does mean you will have to bundle in renamed textures (and say goodbye to uploading it to the server).

I have no idea what this does. Every unit added to RCBM had no need for this.

The radius at which a unit will move to attack an enemy.

That's to do with animations. You tweak this number to match up the movement speed with the movement of the legs. Or to put it simply, to stop bots from moonwalking.

I don't know enough about animations to help you on that front.

Why would I need to rename the textures? The whole point is to make it actually upload to the server

EDIT:

Who needs textures anyway.....
Does it just use the texture by the name of the model? This means I cannot change the name of the model? Can I change the name of the unit without reuploading basically half of PA?

The issue is that PA doesn't like to shadow a model that is in vanilla PA. To get around this you need to rename the model. Do you understand this part?

Lets say you are loading example.papa, PA will then go on to look for example_diffuse.papa, example_mask.papa and example_material.papa. It wouldn't load example2_diffuse.papa even if it's in the same folder.

I didn't know PA would determine the name of the textures based on the name of the model. I would've guessed the model has the name of the texture backed into it.
I understand just fine and I understand the issue is that I can't solve it this way, as the textures are way to big to be all part of the mod.
Renaming the textures and the model works fine though for a single unit:



So any ideas how to solve this without changing the model+texture name?
If there are none then my next best idea is to continue working based on "if you want to use this you have to copy and replace these models into your PA to make it actually look correct."

As well as do this:
@bgolus @chargrove @varrak
Can any of you guys maybe take a quick look at the issue "server mods do not allow to shadow existing models"?
Maybe it's an easy fix, if so I'd appreciate a fix

Or up the server mod size limit to 60mb, that would be enough if I cut out commanders, orbital and unused units.

EDIT: oh there is no upload limit. Well then a fix for this bug would reduce the mod size by like 70mb, but that's all

This is about transportation, and where the attach point is - note the following in the transporter blueprint:

Code:

"transportable_attach_offset": "head", 

Hmm PA is crashing due to some errors in how I rename the models, but after roughly 60s of waiting at a constant upload of 1mb/s the mod is shown as loaded in PA.
Maybe there actually is no upload limit?
Has anybody ever actually hit that limit?

There is no upload limit.

Statera was 25+ meg before we went through and deleted some unneeded files.

That's good because I just hit ~85mb.

EDIT:
The 85mb version contains everything but orbital units and commanders and works

There's no hard limit, but Uber have asked for them to be relatively small:
https://forums.uberent.com/threads/coming-up-soon-server-mods.59738/#post-931451

1mb? yeah.... no way. I am not a magician. Not to mention that the server browser loads that much down within like 2 seconds I don't think I am hurting anyone, I have no randoms joining the game anyway.

@raevn:
There seems to be trouble with the bot_factory.papa model. The 101 template fails on it and the model stays unchanged after I try to modify it. A quick fix to have the 101 template at least successfully parse is to remove this block in line 60:

Code:

    if(OffsetMeshInformation > -1)
    {
        FSeek(OffsetMeshInformation);
        PapaMeshInformation MeshInformation[NumberOfMeshes] <optimize=false>;
    };

Not sure yet how that results in my code not doing anything visible to the model.

EDIT 2:
Weird my code correctly hits the vertex x/y/z and scales them down.

I had a quick look at the bot factory in blender to see if there was anything odd about it. For some reason the bot_factory and bot_factory_adv models don't load correctly. It's very odd.


If you have an outdated copy of PA you may want to give it a go with that. Previously I haven't had any issues with importing the bot factories.

Using a version from before last patch seems to work fine.

EDIT:
Whatever I do with the sekelton, it is not what I need to do to fix animations.
It seems to have no visible effect. I.e. the build animation of factories is pretty out of place and hovers above the smaller model.


Maybe I'll have a look at loading a model into blender. I guess that can display the "bones" so I get an idea of what they actually are? So far all I know of them is that they are numbers that are important for animations.

Found the problem. I wasn't saving/restoring the pointer for MeshInformation or ModelInformation when they read in the MaterialGroup/MeshBinding respectively. It wasn't a problem before, because previously there weren't any models that had more than 1 of these.

The blender importer won't handle this correctly either, presumably. I'll take a look a bit later.

Here's the updated 010 template.

Err. Wrong thread for this attachement. I'll leave it here as well though

I've had the same problem in the past. Not sure why...

But but but... This is exactly what I said Uber should do yet you said it wouldn't help, I'm confused tats.

Something is still not right with the 101 template in relation to the bot template:
It only has 72 verticies. That's far too few.
The file only states that it has 3 models in it. Maybe the 72 verticies are for the simple model you also get in blender? There is more in the file than the 101 template finds. After the 72 vertices nothing is mapped at all to the template for quite a while.
I am trying to figure it out, but I am slow. I have no idea how to reverse such a file format, apart from guessing.