redstrate/novus
1
Fork 0
mirror of https://github.com/redstrate/Novus.git synced 2025-04-24 04:57:45 +00:00
Code Activity
novus/apps/sagasu/src/indexer.cpp

262 lines
13 KiB
C++
Raw Normal View History

Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
// SPDX-FileCopyrightText: 2023 Joshua Goins <josh@redstrate.com>
// SPDX-License-Identifier: GPL-3.0-or-later
#include "hashdatabase.h"
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
#include "physis.hpp"
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
#include "settings.h"
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
#include <QCommandLineParser>
#include <QCoreApplication>
#include <QHttpServer>
#include <QJsonObject>
#include <QJsonParseError>
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
sagasu: Add support for exlpart, exdpart and mdlpart for more previews
2023-10-12 21:33:32 -04:00
const std::array known_folders{"common",
"common/font",
"common/graphics",
"common/graphics/texture",
"common/softwarecursor",
"exd",
"chara/equipment/e0000/material/v0001",
"chara/equipment/e0000/texture",
sagasu: Add some more hardcoded files to indexer
2023-10-13 15:00:30 -04:00
"chara/human/c0101/obj/face/f0001/model",
"shader/sm5/shpk",
Make armoury's CMP editor a general-purpose part, add it to sagasu
2023-10-13 15:20:01 -04:00
"chara/xls/bonedeformer",
Make armoury's bone editor a general-purpose part, add it to sagasu
2023-10-13 15:36:36 -04:00
"chara/xls/charamake",
"chara/human/c0101/skeleton/base/b0001"};
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
Add more known common folders and ilfes
2023-10-12 19:29:27 -04:00
const std::array common_font{"common/VulgarWordsFilter.dic",
"common/VulgarWordsFilter_party.dic",
"common/font/fontIcon_Xinput.tex",
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
"common/font/gfdata.gfd",
"common/font/fontIcon_Ps3.tex",
"common/font/gfdata.gfd",
"common/font/fontIcon_Ps4.tex",
"common/font/gfdata.gfd",
"common/font/fontIcon_Ps5.tex",
"common/font/gfdata.gfd",
"common/font/MiedingerMid_36.fdt",
"common/font/MiedingerMid_18.fdt",
"common/font/MiedingerMid_14.fdt",
"common/font/MiedingerMid_12.fdt",
"common/font/MiedingerMid_10.fdt",
"common/font/Meidinger_40.fdt",
"common/font/Meidinger_20.fdt",
"common/font/Meidinger_16.fdt",
"common/font/=TrumpGothic_68.fdt",
"common/font/TrumpGothic_34.fdt",
"common/font/TrumpGothic_23.fdt",
"common/font/TrumpGothic_184.fdt",
"common/font/Jupiter_46.fdt",
"common/font/Jupiter_23.fdt",
"common/font/Jupiter_20.fdt",
"common/font/Jupiter_16.fdt",
"common/font/Jupiter_90.fdt",
"common/font/Jupiter_45.fdt",
"common/font/MiedingerMid_36.fdt",
"common/font/MiedingerMid_18.fdt",
"common/font/MiedingerMid_14.fdt",
"common/font/MiedingerMid_12.fdt",
"common/font/MiedingerMid_10.fdt",
"common/font/Meidinger_40.fdt",
"common/font/Meidinger_20.fdt",
"common/font/Meidinger_16.fdt",
"common/font/TrumpGothic_68.fdt",
"common/font/TrumpGothic_34.fdt",
"common/font/TrumpGothic_23.fdt",
"common/font/TrumpGothic_184.fdt",
"common/font/Jupiter_46.fdt",
"common/font/Jupiter_23.fdt",
"common/font/Jupiter_20.fdt",
"common/font/Jupiter_16.fdt",
"common/font/Jupiter_90.fdt",
"common/font/Jupiter_45.fdt",
"common/font/AXIS_18_lobby.fdt",
"common/font/AXIS_14_lobby.fdt",
"common/font/AXIS_12_lobby.fdt",
"common/font/AXIS_12_lobby.fdt",
"common/font/>MiedingerMid_18_lobby.fdt",
"common/font/L=MiedingerMid_18_lobby.fdt",
"common/font/>MiedingerMid_14_lobby.fdt",
"common/font/MiedingerMid_12_lobby.fdt",
"common/font/MiedingerMid_10_lobby.fdt",
"common/font/Meidinger_20_lobby.fdt",
"common/font/Meidinger_20_lobby.fdt",
"common/font/Meidinger_16_lobby.fdt",
"common/font/TrumpGothic_34_lobby.fdt",
"common/font/TrumpGothic_34_lobby.fdt",
"common/font/TrumpGothic_23_lobby.fdt",
"common/font/TrumpGothic_184_lobby.fdt",
"common/font/Jupiter_23_lobby.fdt",
"common/font/Jupiter_23_lobby.fdt",
"common/font/Jupiter_20_lobby.fdt",
"common/font/Jupiter_16_lobby.fdt",
"common/font/Jupiter_45_lobby.fdt",
"common/font/Jupiter_45_lobby.fdt",
"common/font/AXIS_18_lobby.fdt",
"common/font/MiedingerMid_18_lobby.fdt",
"common/font/MiedingerMid_18_lobby.fdt",
"common/font/MiedingerMid_14_lobby.fdt",
"common/font/MiedingerMid_12_lobby.fdt",
"common/font/MiedingerMid_10_lobby.fdt",
"common/font/Meidinger_20_lobby.fdt",
"common/font/Meidinger_20_lobby.fdt",
"common/font/Meidinger_16_lobby.fdt",
"common/font/TrumpGothic_34_lobby.fdt",
"common/font/TrumpGothic_34_lobby.fdt",
"common/font/TrumpGothic_23_lobby.fdt",
"common/font/TrumpGothic_184_lobby.fdt",
"common/font/Jupiter_23_lobby.fdt",
"common/font/Jupiter_23_lobby.fdt",
"common/font/Jupiter_20_lobby.fdt",
"common/font/Jupiter_16_lobby.fdt",
"common/font/Jupiter_45_lobby.fdt",
"common/font/Jupiter_45_lobby.fdt",
"common/font/AXIS_18_lobby.fdt",
"common/font/AXIS_14_lobby.fdt",
"common/font/AXIS_12_lobby.fdt",
"common/font/AXIS_12_lobby.fdt",
"common/font/MiedingerMid_36_lobby.fdt",
"common/font/MiedingerMid_18_lobby.fdt",
"common/font/MiedingerMid_14_lobby.fdt",
"common/font/MiedingerMid_12_lobby.fdt",
"common/font/MiedingerMid_10_lobby.fdt",
"common/font/Meidinger_40_lobby.fdt",
"common/font/Meidinger_20_lobby.fdt",
"common/font/Meidinger_16_lobby.fdt",
"common/font/TrumpGothic_68_lobby.fdt",
"common/font/TrumpGothic_34_lobby.fdt",
"common/font/TrumpGothic_23_lobby.fdt",
"common/font/TrumpGothic_184_lobby.fdt",
"common/font/Jupiter_46_lobby.fdt",
"common/font/Jupiter_23_lobby.fdt",
"common/font/Jupiter_20_lobby.fdt",
"common/font/Jupiter_16_lobby.fdt",
"common/font/Jupiter_90_lobby.fdt",
"common/font/Jupiter_45_lobby.fdt",
"common/font/AXIS_36_lobby.fdt",
"common/font/MiedingerMid_36_lobby.fdt",
"common/font/MiedingerMid_18_lobby.fdt",
"common/font/MiedingerMid_14_lobby.fdt",
"common/font/MiedingerMid_12_lobby.fdt",
"common/font/MiedingerMid_10_lobby.fdt",
"common/font/Meidinger_40_lobby.fdt",
"common/font/Meidinger_20_lobby.fdt",
"common/font/Meidinger_16_lobby.fdt",
"common/font/TrumpGothic_68_lobby.fdt",
"common/font/TrumpGothic_34_lobby.fdt",
"common/font/TrumpGothic_23_lobby.fdt",
"common/font/TrumpGothic_184_lobby.fdt",
"common/font/Jupiter_46_lobby.fdt",
"common/font/Jupiter_23_lobby.fdt",
"common/font/Jupiter_20_lobby.fdt",
"common/font/Jupiter_16_lobby.fdt",
"common/font/Jupiter_90_lobby.fdt",
"common/font/Jupiter_45_lobby.fdt"};
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
static std::optional<QJsonObject> byteArrayToJsonObject(const QByteArray &arr)
{
QJsonParseError err;
const auto json = QJsonDocument::fromJson(arr, &err);
if (err.error || !json.isObject())
return std::nullopt;
return json.object();
}
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
int main(int argc, char *argv[])
{
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
QCoreApplication app(argc, argv);
QCommandLineParser parser;
QCommandLineOption indexExcelOption(QStringLiteral("excel"), QStringLiteral("Enable the indexing of known Excel files, will take a very long time."));
parser.addOption(indexExcelOption);
QCommandLineOption listenOption(QStringLiteral("listen"), QStringLiteral("Listen for incoming filename requests, for the SqPackIndexer Dalamud plugin."));
parser.addOption(listenOption);
parser.process(app);
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
HashDatabase database;
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
if (parser.isSet(indexExcelOption)) {
const QString gameDir{getGameDirectory()};
const std::string gameDirStd{gameDir.toStdString()};
auto data = physis_gamedata_initialize(gameDirStd.c_str());
auto sheetNames = physis_gamedata_get_all_sheet_names(data);
Fix most warnings
2023-12-09 22:35:59 -05:00
for (uint32_t i = 0; i < sheetNames.name_count; i++) {
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
auto sheetName = sheetNames.names[i];
auto nameLowercase = QString::fromStdString(sheetName).toLower().toStdString();
QString headerName = QStringLiteral("exd/") + QLatin1String(nameLowercase.c_str()) + QStringLiteral(".exh");
database.addFile(headerName);
std::string headerNameStd = headerName.toStdString();
auto exh = physis_parse_excel_sheet_header(physis_gamedata_extract_file(data, headerNameStd.c_str()));
Fix most warnings
2023-12-09 22:35:59 -05:00
for (uint32_t j = 0; j < exh->page_count; j++) {
for (uint32_t z = 0; z < exh->language_count; z++) {
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
std::string path = physis_gamedata_get_exd_filename(nameLowercase.c_str(), exh, exh->languages[z], j);
database.addFile(QStringLiteral("exd/") + QString::fromStdString(path));
}
}
}
return 0;
}
if (parser.isSet(listenOption)) {
QHttpServer server;
server.route(QStringLiteral("/add_hash"), QHttpServerRequest::Method::Post, [&database](const QHttpServerRequest &request) {
const auto json = byteArrayToJsonObject(request.body());
if (!json)
return QHttpServerResponse(QHttpServerResponder::StatusCode::BadRequest);
const QString filename = (*json)[QLatin1String("filename")].toString();
int lastSlashIndex = filename.lastIndexOf(QStringLiteral("/"));
QString folder = filename.left(lastSlashIndex);
qInfo() << "Adding hash for file" << filename;
qInfo() << "Adding hash for folder" << folder;
database.addFile(filename);
database.addFolder(folder);
return QHttpServerResponse(QHttpServerResponder::StatusCode::Ok);
});
server.listen(QHostAddress::Any, 3500);
qInfo() << "Now listening on port 3500.";
return QCoreApplication::exec();
}
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
for (auto &folder : known_folders) {
database.addFolder(QLatin1String(folder));
}
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
qInfo() << "No special options set, adding all known static files.";
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
for (auto &file : common_font) {
database.addFile(QLatin1String(file));
}
sagasu: Index all ffxiv index files, stop hardcoding a path you dunce
2023-10-12 20:30:17 -04:00
database.addFile(QStringLiteral("exd/root.exl"));
sagasu: Add support for exlpart, exdpart and mdlpart for more previews
2023-10-12 21:33:32 -04:00
database.addFile(QStringLiteral("exd/achievement.exh"));
database.addFile(QStringLiteral("exd/Achievement_0_en.exd"));
database.addFile(QStringLiteral("chara/equipment/e0000/material/v0001/mt_c0101e0000_sho_a.mtrl"));
database.addFile(QStringLiteral("chara/equipment/e0000/texture/v01_c0101e0000_sho_d.tex"));
database.addFile(QStringLiteral("chara/human/c0101/obj/face/f0001/model/c0101f0001_fac.mdl"));
sagasu: Add some more hardcoded files to indexer
2023-10-13 15:00:30 -04:00
database.addFile(QStringLiteral("shader/sm5/shpk/character.shpk"));
database.addFile(QStringLiteral("chara/xls/bonedeformer/human.pbd"));
Make armoury's CMP editor a general-purpose part, add it to sagasu
2023-10-13 15:20:01 -04:00
database.addFile(QStringLiteral("chara/xls/charamake/human.cmp"));
Make armoury's bone editor a general-purpose part, add it to sagasu
2023-10-13 15:36:36 -04:00
database.addFile(QStringLiteral("chara/human/c0101/skeleton/base/b0001/skl_c0101b0001.sklb"));
Create reusable material part This takes the existing properties widget in the new material editor and makes it reusable across multiple applications. It's now added to the Armoury which shows the gear's used materials. It's also added to the Data Explorer which now supports viewing material files. I fixed the render viewport crashing when hiding it again, and made it even more resilient.
2024-04-27 13:45:08 -04:00
database.addFile(QStringLiteral("chara/equipment/e0028/material/v0001/mt_c0101e0028_top_a.mtrl"));
sagasu: Index all ffxiv index files, stop hardcoding a path you dunce
2023-10-12 20:30:17 -04:00
Add support for adding filenames via HTTP Meant to be used by the new Dalamud plugin
2023-10-14 20:23:27 -04:00
return 0;
Improve Sagasu design & indexing Now the hashes are collected in a central database (location to be improved) similar to FFXIV Explorer. This database needs to be generated once and doesn't have to be regen every time Sagasu is opened like before. This indexer currently is a separate program. Also adds a feature to extract files from the file tree window.
2023-10-12 19:04:19 -04:00
}
Copy permalink