commit c3e126cabb575aefdce9e487c4a7104593e38d83 Author: liam Date: Sat Sep 16 16:13:06 2023 -0400 Initial commit diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..add57be --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +bin/ +obj/ +/packages/ +riderModule.iml +/_ReSharper.Caches/ \ No newline at end of file diff --git a/EXDSchema.sln b/EXDSchema.sln new file mode 100644 index 0000000..9dbb61d --- /dev/null +++ b/EXDSchema.sln @@ -0,0 +1,22 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SchemaConverter", "SchemaConverter\SchemaConverter.csproj", "{9AC0C7BB-939B-496A-A882-44327DAF7C3F}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SchemaValidator", "SchemaValidator\SchemaValidator.csproj", "{22B92DA2-D46B-4219-A0E7-CC55F9213BAD}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {9AC0C7BB-939B-496A-A882-44327DAF7C3F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {9AC0C7BB-939B-496A-A882-44327DAF7C3F}.Debug|Any CPU.Build.0 = Debug|Any CPU + {9AC0C7BB-939B-496A-A882-44327DAF7C3F}.Release|Any CPU.ActiveCfg = Release|Any CPU + {9AC0C7BB-939B-496A-A882-44327DAF7C3F}.Release|Any CPU.Build.0 = Release|Any CPU + {22B92DA2-D46B-4219-A0E7-CC55F9213BAD}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {22B92DA2-D46B-4219-A0E7-CC55F9213BAD}.Debug|Any CPU.Build.0 = Debug|Any CPU + {22B92DA2-D46B-4219-A0E7-CC55F9213BAD}.Release|Any CPU.ActiveCfg = Release|Any CPU + {22B92DA2-D46B-4219-A0E7-CC55F9213BAD}.Release|Any CPU.Build.0 = Release|Any CPU + EndGlobalSection +EndGlobal diff --git a/README.md b/README.md new file mode 100644 index 0000000..c4b73cf --- /dev/null +++ b/README.md @@ -0,0 +1,41 @@ +# EXDSchema +## Introduction +This is the schema repository for SqPack [Excel data](https://xiv.dev/game-data/file-formats/excel). + +## Sheets +Inside SqPack, category 0A consists of Excel sheets serialized into a proprietary binary format read by the game. +The development cycle generates header files for each sheet, which are then compiled into the game, thus, all structure information +is lost on the client side when the game is compiled. This repository is an attempt to consolidate efforts into a +language agnostic schema, easily parsed into any language that wishes to consume it, that accurately describes the structure +of the EXH files as they are provided to the client. + +## Schema +Schemas are written in YML to define the fields of the structure in an EXH file and the links between different fields. +The schema provides a number of features, all of which is enforced by the provided JSON schema for the schema. When applied +against an EXD schema file, it will provide IDE completion and error-checking to improve the manual editing experience. + +## Features +Since EXH files define the data types for each column, the schema does not care or define any data types in the standard sense. +Instead, it focuses on declaratively defining the structure of the compiled EXH data structures and the relationships between fields. + +The schema includes the following: +- Full declaration of fields is required, nothing can be omitted +- Support for a few common types across sheets, such as `modelId`, `color`, and `icon` + - While these do not affect the overall parsing, they are + useful for research as they provide an important hint for the purpose of the data +- Field names +- Arrays +- Links between fields +- Multi-targeting another sheet +- Complex linking between fields based on a `switch` conditional +- Comment support on any schema object +- Maps out-of-the-box to a very simple object mapping +- JSON schema for the schema itself, providing IDE completion and error-checking +- Example code + - A C# project that perform the conversion from the + common JSON schemas ("SC Schema") to the EXDSchema format + - A C# project that will validate EXDSchemas against a given game installation + - A tool for editing schemas and viewing the results of parsing on-the-fly + +## Usage +See Usage.md. \ No newline at end of file diff --git a/SchemaConverter/Program.cs b/SchemaConverter/Program.cs new file mode 100644 index 0000000..e5dff12 --- /dev/null +++ b/SchemaConverter/Program.cs @@ -0,0 +1,3 @@ +// See https://aka.ms/new-console-template for more information + +Console.WriteLine("Hello, World!"); \ No newline at end of file diff --git a/SchemaConverter/SchemaConverter.csproj b/SchemaConverter/SchemaConverter.csproj new file mode 100644 index 0000000..2b14c81 --- /dev/null +++ b/SchemaConverter/SchemaConverter.csproj @@ -0,0 +1,10 @@ + + + + Exe + net7.0 + enable + enable + + + diff --git a/SchemaValidator/Program.cs b/SchemaValidator/Program.cs new file mode 100644 index 0000000..e5dff12 --- /dev/null +++ b/SchemaValidator/Program.cs @@ -0,0 +1,3 @@ +// See https://aka.ms/new-console-template for more information + +Console.WriteLine("Hello, World!"); \ No newline at end of file diff --git a/SchemaValidator/SchemaValidator.csproj b/SchemaValidator/SchemaValidator.csproj new file mode 100644 index 0000000..2b14c81 --- /dev/null +++ b/SchemaValidator/SchemaValidator.csproj @@ -0,0 +1,10 @@ + + + + Exe + net7.0 + enable + enable + + + diff --git a/Schemas/.gitkeep b/Schemas/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/Usage.md b/Usage.md new file mode 100644 index 0000000..0b33de3 --- /dev/null +++ b/Usage.md @@ -0,0 +1,262 @@ +# Usage + +## Initial Creation +To define a schema, you should create a file with the same name as the sheet it is defining. +The name field must contain the name of the sheet as well. If we were to write a schema for `AozActionTransient`, we would do the following: + +```yml +name: AozActionTransient +fields: + - name: Field1 + - name: Field2 + - name: Field3 +# etc ... +``` + +#### DisplayField +The `displayField` key is provided for consumers that wish to resolve a sheet reference within a single cell. It provides a hint +of what a user will *most likely* want to see when the current sheet is targeted by a link. For example, when linking to `BNpcName`, +the most likely column to reference would be `Name`. For `Item`, the most likely column might be `Name` or `Singular`. + +## Defining Fields +All sheets must have a number of field entries that corresponds to the number of columns in that sheet. +If not, parsing should fail. + +We can define fields like this: +```yml +type: sheet +fields: + - name: Stats + - name: Description + - name: Icon + - name: RequiredForQuest + - name: PreviousQuest + - name: Location + - name: Number + - name: LocationKey + - name: CauseStun + - name: CauseBlind + - name: CauseInterrupt + - name: CauseParalysis + - name: TargetsSelfOrAlly + - name: CauseSlow + - name: TargetsEnemy + - name: CausePetrify + - name: CauseHeavy + - name: CauseSleepy + - name: CauseBind + - name: CauseDeath +``` +This schema is valid because it is accurate - not in name, but in structure. It defines a field for each column in the EXH file as of 6.48. + +### Types +Valid types for fields in a schema are `scalar`, `array`, `icon`, `modelId`, and `color`. + +#### scalar +The default type. If the `type` is omitted from a field, it will be assumed to be a scalar. Effectively does nothing except tell consumers that +"this field is not an array". + +#### icon : uint32 +In the above AozActionTransient example, +```yml + - name: Icon +``` +can become +```yml + - name: Icon + type: icon +``` +While this may seem redundant, there are many fields in column that refer to an icon within the `06`, or the `ui/` category, +but the field itself is just a uint32. This is a hint for any consumer that attempts to display this field that the data in this column +can be used to format an icon path, like generating `ui/icon/132000/132122_hr1.tex` when the field contains `132122`, without the consumer having +to manually determine which columns contain icons. + +#### modelId : uint32, uint64 +Model IDs in the game are packed into either a uint32 or a uint64. + +uint32 packing is like so: +``` +uint16 modelId +uint8 variantId +uint8 stain +``` +uint64 packing is like so: +``` +uint16 skeletonId +uint16 modelId +uint16 variantId +uint16 stainId +``` +To anyone *viewing* the data for research, the packed values are useless, so consumers that provide a view into sheet data can opt +to unpack these values and display them as their unpacked counterparts. Many tools utilize these values individually rather than packed, +so it's important to have the ability to define a field this way. + +#### color : uint32 +Some fields contain an RGB value for color in the ARGB format with no alpha. This is simply a hint if a consumer opts to display these +columns' fields as actual colors rather than the raw value. + +#### array +Array fields provide the ability to group and repeat nested structures. For example, the notorious SpecialShop sheet: +```yml +name: SpecialShop +fields: + - name: Name + - name: Item + type: array + count: 60 + fields: + - name: ReceiveCount + type: array + count: 2 + - name: CurrencyCost + type: array + count: 3 + - name: Item + type: array + count: 2 + - name: Category + type: array + count: 2 + - name: ItemCost + type: array + count: 3 + - name: Quest + type: array + count: 2 + - name: Unknown + - name: AchievementUnlock + - name: CollectabilityCost + type: array + count: 3 + - name: PatchNumber + - name: HqCost + type: array + count: 3 + - type: array + count: 3 + - name: ReceiveHq + type: array + count: 3 + - name: Quest + - type: scalar + - type: scalar + - name: CompleteText + - name: NotCompleteText + - type: scalar + - name: UseCurrencyType + - type: scalar + - type: scalar +``` +As you can see, we have nested arrays in this structure. This means that the in-memory structure follows like so: +```C +struct SpecialShop +{ + struct + { + example_type ReceiveCount[2]; + example_type CurrencyCost[3]; + example_type Item[2]; + example_type Category[2]; + example_type ItemCost[3]; + example_type Quest[2]; + example_type Unknown; + example_type AchievementUnlock; + example_type CollectabilityCost[3]; + example_type PatchNumber; + example_type HqCost[3]; + example_type Unknown2[2]; + example_type ReceiveHq[3]; + } Items[60]; + example_type Quest; + example_type Unknown; + example_type Unknown2; + example_type CompleteText; + example_type NotCompleteText; + example_type Unknown3; + example_type UseCurrencyType; + example_type Unknown4; + example_type Unknown5; +}; +``` +As you can see, the overall schema is similar to defining structures in YML but omitting the actual data type. +This nested capability allows you to define complex structures. However, to cut down on overall parsing complexity, +based on existing knowledge of the EXH data, **you may only nest twice.** + +### Linking +The sheets that power the game are relational in nature, so the schema supports a few different kinds of linking. + +#### Single Link +To define a link, simply add a link object: +```yml + - name: Quest + link: + target: [Quest] +``` +Note that the link target is an array of strings. They must be sheet names, and there must be at least one sheet. To link to one sheet, leave a single sheet in the array. + +#### Multi Link +A sheet's single column can link to multiple columns: +```yml + - name: Requirement + link: + target: [Quest, GrandCompany] +``` +In this case, disparate sheet key ranges will provide the ability to determine which sheet a link should resolve to. +For example, if a row's `Requirement` is `2`, it will resolve to `GrandCompany`, because row `2` exists in `GrandCompany` and not in `Quest.` +The same thing happens in the other direction: if `Requirement` is `69208`, it will link to `Quest` and not `GrandCompany` for the same reason. + +#### Conditional Link +A sheet's single column can link to multiple columns depending on another field in the sheet: +```yml + - name: Location + comment: PlaceName when LocationKey is 1, ContentFinderCondition when LocationKey is 4 + link: + target: [PlaceName, ContentFinderCondition] + condition: + switch: LocationKey + cases: + 1: [0] + 4: [1] +``` +The targets array must contain all possible sheets that this field can link to. +When defining the link, add a `condition` object with a `switch` key that defines the field to switch on the value of. +The `cases` dictionary contains arrays of *the indexes of* sheets to resolve to when the case matches. + +Yes, the `case` dictionary may contain an *array*. This means that each case can be a [multi link](#multi-link) as well. Take `Item` for example: +```yml + - name: AdditionalData + link: + target: + - Stain + - TreasureHuntRank + - GardeningSeed + - AetherialWheel + - CompanyAction + - TripleTriadCard + - AirshipExplorationPart + - Orchestrion + - SubmarinePart + - HousingExterior + - HousingInterior + - HousingYardObject + - HousingFurniture + - HousingPreset + - HousingUnitedExterior + condition: + switch: FilterGroup + cases: + 14: [9, 10, 11, 12, 13, 14] + 15: [0] + 18: [1] + 20: [2] + 25: [3] + 26: [4] + 27: [5] + 28: [6] + 32: [7] + 36: [8] +``` +The `AdditionalData` column does a lot of heavy lifting. We can assume during game execution that the use of the field is heavily based on context, +but for research and data exploration, having the ability to define the exact sheet is very useful. Here, we can see that when `FilterGroup` is `14`, +we can link to any of `HousingExterior`, `HousingInterior`, `HousingYardObject`, `HousingFurniture`, `HousingPreset`, or finally `HousingUnitedExterior`. +This works because the value for `AdditionalData` are distinct ranges, even when `FilterGroup` is 14, thus allowing the definition here to behave like a multi link. \ No newline at end of file