commit 7dcc3c3eeb3bab57e9594fde8c8d2a2c767bbfe1 parent 91eda88e767529a60f021389246a28ed3acd4da2 Author: JohnMcPMS <johnmcp@microsoft.com> Date: Thu, 18 Apr 2024 16:31:01 -0700 Index v2 (#4387) ## Change This change introduces schema 2.0 for the `SQLiteIndex`. This new major version takes the learnings on how we actually used the index to shift to a package centralized store. It also moves some of the data that is not directly needed for search and correlation to intermediate manifests per-package. The goal is that `search` and `list` functionality (including `update` probing) should not need the intermediate files; only operations that change the system state (write operations) should. Using a recent index as a starting point for comparison, the 2.0 index reduced the size of a ZIP archive (containing the 1 file and produced by Windows Explorer) by **79%**. This change only implements the index creation. A future change will implement the consumption, including a correct implementation of the 2.0 search functionality. The current search is just copied/commented from a previous schema version for now. ### Implementation The 1.* index schema allows easy updating from individual manifest changes, without the need to inspect any other manifests for the same package. The final 2.* schema will not allow that. Instead, 2.0 actually uses a 1.7 schema internally, with an additional table to track changes so that we can produce the intermediate manifests. When `PrepareForPackaging` is called, the schema 2.0 tables are created and the data migrated to them. The intermediate manifest files that have changed are also written to disk. ### Intermediate Manifests The intermediate manifest files (`PackageVersionDataManifest`) are YAML with shortened key values. This saves some bytes since humans are neither authoring them nor reading them (except to debug). They are also stored in a compressed stream, using the MSZIP compression algorithm. The compression brings the average size down from 1236 bytes to 463 bytes, and the median down from 338 bytes to 206 bytes. The future consumption change will cache these intermediate manifests (and likely the version manifests as well), allowing their reuse as long as they have not changed. ### Additional Functionality To support our use in creating the index in our services, some additional functionality was added to the `SQLiteIndex`. #### Migration A schema version migration function is added, allowing the target schema to migrate from the existing one as it sees fit. If the migration is not supported, it can simply return a value indicating that. Only 1.7 => 2.0 migration is implemented. #### Properties Properties can be set on the index object, some of which are for that object only and some of which are persisted into the database itself. An implicit property of the database file name is stored when appropriate. The caller can set the directory path to output intermediate files to. The caller can also set the time (in Unix epoch) to use as the baseline for which intermediate files should be output [in practice, only an empty string (for "now") and "0" (to output everything) are likely to be used]. Diffstat:
73 files changed, 5084 insertions(+), 371 deletions(-)
diff --git a/.github/actions/spelling/expect.txt b/.github/actions/spelling/expect.txt @@ -72,6 +72,7 @@ CODEOWNERS COINIT COMGLB commandline +compressapi contactsupport contentfiles contoso @@ -273,6 +274,8 @@ msftrubengu MSIHASH MSIXHASH msstore +MSZIP +mszyml Mta Mugiwara Multideclaration diff --git a/src/AppInstallerCLITests/AppInstallerCLITests.vcxproj b/src/AppInstallerCLITests/AppInstallerCLITests.vcxproj @@ -232,6 +232,7 @@ <ClCompile Include="PackageCollection.cpp" /> <ClCompile Include="PackageDependenciesValidationUtil.cpp" /> <ClCompile Include="PackageTrackingCatalog.cpp" /> + <ClCompile Include="PackageVersionDataManifest.cpp" /> <ClCompile Include="PathVariable.cpp" /> <ClCompile Include="PinFlow.cpp" /> <ClCompile Include="PinningIndex.cpp" /> diff --git a/src/AppInstallerCLITests/AppInstallerCLITests.vcxproj.filters b/src/AppInstallerCLITests/AppInstallerCLITests.vcxproj.filters @@ -344,6 +344,9 @@ <ClCompile Include="RestInterface_1_7.cpp"> <Filter>Source Files\Repository</Filter> </ClCompile> + <ClCompile Include="PackageVersionDataManifest.cpp"> + <Filter>Source Files\Common</Filter> + </ClCompile> </ItemGroup> <ItemGroup> <None Include="PropertySheet.props" /> diff --git a/src/AppInstallerCLITests/PackageVersionDataManifest.cpp b/src/AppInstallerCLITests/PackageVersionDataManifest.cpp @@ -0,0 +1,103 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "TestCommon.h" +#include <winget/PackageVersionDataManifest.h> + +using namespace TestCommon; +using namespace AppInstaller; +using namespace AppInstaller::Manifest; +using namespace AppInstaller::Utility; + +void RequireVersionDataEqual(const PackageVersionDataManifest::VersionData& first, const PackageVersionDataManifest::VersionData& second) +{ + REQUIRE(first.Version == second.Version); + REQUIRE(first.ArpMinVersion == second.ArpMinVersion); + REQUIRE(first.ArpMaxVersion == second.ArpMaxVersion); + REQUIRE(first.ManifestRelativePath == second.ManifestRelativePath); + REQUIRE(first.ManifestHash == second.ManifestHash); +} + +TEST_CASE("PackageVersionDataManifest_Empty", "[PackageVersionDataManifest]") +{ + PackageVersionDataManifest original; + + PackageVersionDataManifest copy; + copy.Deserialize(original.Serialize()); + + REQUIRE(original.Versions().empty()); + REQUIRE(copy.Versions().empty()); +} + +TEST_CASE("PackageVersionDataManifest_Single_Simple", "[PackageVersionDataManifest]") +{ + PackageVersionDataManifest original; + original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, {}, {}, "path", "hash" }); + + PackageVersionDataManifest copy; + copy.Deserialize(original.Serialize()); + + REQUIRE(original.Versions().size() == 1); + REQUIRE(copy.Versions().size() == 1); + + RequireVersionDataEqual(copy.Versions()[0], original.Versions()[0]); +} + +TEST_CASE("PackageVersionDataManifest_Single_Complete", "[PackageVersionDataManifest]") +{ + PackageVersionDataManifest original; + original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, ".99", "1.01", "path", "hash"}); + + PackageVersionDataManifest copy; + copy.Deserialize(original.Serialize()); + + REQUIRE(original.Versions().size() == 1); + REQUIRE(copy.Versions().size() == 1); + + RequireVersionDataEqual(copy.Versions()[0], original.Versions()[0]); +} + +TEST_CASE("PackageVersionDataManifest_Multiple", "[PackageVersionDataManifest]") +{ + PackageVersionDataManifest original; + original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, ".99", "1.01", "path", "hash" }); + original.AddVersion({ VersionAndChannel{ Version{ "1.1" }, Channel{} }, "1.99", "2.01", "path2", "hash2" }); + original.AddVersion({ VersionAndChannel{ Version{ "1.2" }, Channel{} }, {}, {}, "path2", "hash2" }); + original.AddVersion({ VersionAndChannel{ Version{ "2.0" }, Channel{} }, "3.99", "15.01", "path4", "hash4" }); + + PackageVersionDataManifest copy; + copy.Deserialize(original.Serialize()); + + REQUIRE(original.Versions().size() == copy.Versions().size()); + + for (size_t i = 0; i < original.Versions().size(); ++i) + { + INFO(i); + RequireVersionDataEqual(copy.Versions()[i], original.Versions()[i]); + } +} + +TEST_CASE("PackageVersionDataManifest_CompressionRoundTrip", "[PackageVersionDataManifest]") +{ + PackageVersionDataManifest original; + original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, ".99", "1.01", "path", "hash" }); + original.AddVersion({ VersionAndChannel{ Version{ "1.1" }, Channel{} }, "1.99", "2.01", "path2", "hash2" }); + original.AddVersion({ VersionAndChannel{ Version{ "1.2" }, Channel{} }, {}, {}, "path2", "hash2" }); + original.AddVersion({ VersionAndChannel{ Version{ "2.0" }, Channel{} }, "3.99", "15.01", "path4", "hash4" }); + + std::string serialized = original.Serialize(); + auto compressed = PackageVersionDataManifest::CreateCompressor().Compress(serialized); + + auto decompressed = PackageVersionDataManifest::CreateDecompressor().Decompress(compressed); + + PackageVersionDataManifest copy; + copy.Deserialize(decompressed); + + REQUIRE(original.Versions().size() == copy.Versions().size()); + + for (size_t i = 0; i < original.Versions().size(); ++i) + { + INFO(i); + RequireVersionDataEqual(copy.Versions()[i], original.Versions()[i]); + } +} diff --git a/src/AppInstallerCLITests/SQLiteIndex.cpp b/src/AppInstallerCLITests/SQLiteIndex.cpp @@ -8,6 +8,8 @@ #include <Microsoft/SQLiteIndex.h> #include <winget/Manifest.h> #include <AppInstallerStrings.h> +#include <winget/SQLiteMetadataTable.h> +#include <winget/PackageVersionDataManifest.h> #include <Microsoft/Schema/1_0/IdTable.h> #include <Microsoft/Schema/1_0/NameTable.h> @@ -20,6 +22,8 @@ #include <Microsoft/Schema/1_0/CommandsTable.h> #include <Microsoft/Schema/1_0/SearchResultsTable.h> #include <Microsoft/Schema/1_4/DependenciesTable.h> +#include <Microsoft/Schema/2_0/Interface.h> +#include <Microsoft/Schema/2_0/PackageUpdateTrackingTable.h> using namespace std::string_literals; using namespace std::string_view_literals; @@ -38,15 +42,9 @@ SQLiteIndex CreateTestIndex(const std::string& filePath, std::optional<SQLiteVer // If no specific version requested, then use generator to run against the last 3 versions. if (!version) { - SQLiteVersion latestVersion = SQLiteIndex::GetLatestVersion(); - if (latestVersion.MajorVersion != 1) - { - throw std::exception("You added major version 2, figure out how to deal with these tests that do back compat coverage!"); - } - - // Relies on the fact that min version is already >= 2 - SQLiteVersion versionMinus1 = SQLiteVersion{ 1, latestVersion.MinorVersion - 1 }; - SQLiteVersion versionMinus2 = SQLiteVersion{ 1, latestVersion.MinorVersion - 2 }; + SQLiteVersion latestVersion{ 2, 0 }; + SQLiteVersion versionMinus1 = SQLiteVersion{ 1, 7 }; + SQLiteVersion versionMinus2 = SQLiteVersion{ 1, 6 }; version = GENERATE_COPY(SQLiteVersion{ versionMinus2 }, SQLiteVersion{ versionMinus1 }, SQLiteVersion{ latestVersion }); } @@ -56,15 +54,9 @@ SQLiteIndex CreateTestIndex(const std::string& filePath, std::optional<SQLiteVer SQLiteVersion TestPrepareForRead(SQLiteIndex& index) { - SQLiteVersion latestVersion = SQLiteIndex::GetLatestVersion(); - if (latestVersion.MajorVersion != 1) - { - throw std::exception("You added major version 2, figure out how to deal with these tests that do back compat coverage!"); - } - - // Relies on the fact that min version is already >= 2 - SQLiteVersion versionMinus1 = SQLiteVersion{ 1, latestVersion.MinorVersion - 1 }; - SQLiteVersion versionMinus2 = SQLiteVersion{ 1, latestVersion.MinorVersion - 2 }; + SQLiteVersion latestVersion{ 2, 0 }; + SQLiteVersion versionMinus1 = SQLiteVersion{ 1, 7 }; + SQLiteVersion versionMinus2 = SQLiteVersion{ 1, 6 }; if (index.GetVersion() == versionMinus2) { @@ -1027,7 +1019,7 @@ TEST_CASE("SQLiteIndex_RemoveManifest_EnsureConsistentRowId", "[sqliteindex]") manifest2.DefaultLocalization.Add<Localization::Tags>({}); manifest2.Installers[0].Commands = { "test1", "test2", "test3" }; - SQLiteIndex index = CreateTestIndex(tempFile); + SQLiteIndex index = CreateTestIndex(tempFile, SQLiteVersion{ 1, 7 }); index.AddManifest(manifest1, manifest1Path); index.AddManifest(manifest2, manifest2Path); @@ -1062,6 +1054,7 @@ TEST_CASE("SQLiteIndex_RemoveManifest_EnsureConsistentRowId", "[sqliteindex]") REQUIRE(manifest2.Id == index.GetPropertyByManifestId(manifest2RowId, PackageVersionProperty::Id)); REQUIRE(manifest2.DefaultLocalization.Get<Localization::PackageName>() == index.GetPropertyByManifestId(manifest2RowId, PackageVersionProperty::Name)); + REQUIRE(manifest2.Moniker == index.GetPropertyByManifestId(manifest2RowId, PackageVersionProperty::Moniker)); REQUIRE(manifest2.Version == index.GetPropertyByManifestId(manifest2RowId, PackageVersionProperty::Version)); REQUIRE(manifest2.Channel == index.GetPropertyByManifestId(manifest2RowId, PackageVersionProperty::Channel)); REQUIRE(manifest2Path == index.GetPropertyByManifestId(manifest2RowId, PackageVersionProperty::RelativePath)); @@ -2739,7 +2732,52 @@ TEST_CASE("SQLiteIndex_GetMultiProperty_ProductCode", "[sqliteindex]") } } -TEST_CASE("SQLiteIndex_ManifestMetadata", "[sqliteindex]") +TEST_CASE("SQLiteIndex_GetMultiProperty_Tag", "[sqliteindex]") +{ + TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; + INFO("Using temporary file named: " << tempFile.GetPath()); + + SQLiteIndex index = SearchTestSetup(tempFile, { + { "Id1", "Name1", "Moniker", "Version", "Channel", { "Tag1", "Tag2" }, { "Command" }, "Path1", {}, { "PC1", "PC2" } }, + }); + + SQLiteVersion testVersion = TestPrepareForRead(index); + + SearchRequest request; + + auto results = index.Search(request); + REQUIRE(results.Matches.size() == 1); + + auto props = index.GetMultiPropertyByManifestId(results.Matches[0].first, PackageVersionMultiProperty::Tag); + + REQUIRE(props.size() == 2); + REQUIRE(std::find(props.begin(), props.end(), "Tag1") != props.end()); + REQUIRE(std::find(props.begin(), props.end(), "Tag2") != props.end()); +} + +TEST_CASE("SQLiteIndex_GetMultiProperty_Command", "[sqliteindex]") +{ + TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; + INFO("Using temporary file named: " << tempFile.GetPath()); + + SQLiteIndex index = SearchTestSetup(tempFile, { + { "Id1", "Name1", "Moniker", "Version", "Channel", { "Tag1", "Tag2" }, { "Command" }, "Path1", {}, { "PC1", "PC2" } }, + }); + + SQLiteVersion testVersion = TestPrepareForRead(index); + + SearchRequest request; + + auto results = index.Search(request); + REQUIRE(results.Matches.size() == 1); + + auto props = index.GetMultiPropertyByManifestId(results.Matches[0].first, PackageVersionMultiProperty::Command); + + REQUIRE(props.size() == 1); + REQUIRE(props[0] == "Command"); +} + +TEST_CASE("SQLiteIndex_ManifestMetadata", "[sqliteindex][V1_7]") { TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; INFO("Using temporary file named: " << tempFile.GetPath()); @@ -2747,7 +2785,7 @@ TEST_CASE("SQLiteIndex_ManifestMetadata", "[sqliteindex]") SQLiteIndex index = SearchTestSetup(tempFile, { { "Id1", "Name1", "Moniker", "Version", "Channel", { "Tag" }, { "Command" }, "Path1", {}, { "PC1", "PC2" } }, { "Id2", "Name2", "Moniker", "Version", "Channel", { "Tag" }, { "Command" }, "Path2", { "PFN1", "PFN2" }, {} }, - }); + }, SQLiteVersion{ 1, 7 }); SQLiteVersion testVersion = TestPrepareForRead(index); @@ -3213,7 +3251,7 @@ TEST_CASE("SQLiteIndex_CheckConsistency_FindEmbeddedNull", "[sqliteindex]") REQUIRE(!index.CheckConsistency(true)); } -TEST_CASE("SQLiteIndex_MapDataFolding_Tags", "[sqliteindex][mapdatafolding]") +TEST_CASE("SQLiteIndex_MapDataFolding_Tags", "[sqliteindex][mapdatafolding][V1_7]") { TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; INFO("Using temporary file named: " << tempFile.GetPath()); @@ -3224,7 +3262,7 @@ TEST_CASE("SQLiteIndex_MapDataFolding_Tags", "[sqliteindex][mapdatafolding]") SQLiteIndex index = SearchTestSetup(tempFile, { { "Id", "Name", "Publisher", "Moniker", "Version1", "", { tag1 }, { "Command" }, "Path1", {}, { "PC1" } }, { "Id", "Name", "Publisher", "Moniker", "Version2", "", { tag2 }, { "Command" }, "Path2", {}, { "PC2" } }, - }); + }, SQLiteVersion{ 1, 7 }); // Apply the map data folding if it is present in the created test index. index.PrepareForPackaging(); @@ -3244,7 +3282,7 @@ TEST_CASE("SQLiteIndex_MapDataFolding_Tags", "[sqliteindex][mapdatafolding]") REQUIRE(results1.Matches[0].first == results2.Matches[0].first); } -TEST_CASE("SQLiteIndex_MapDataFolding_PFNs", "[sqliteindex][mapdatafolding]") +TEST_CASE("SQLiteIndex_MapDataFolding_PFNs", "[sqliteindex][mapdatafolding][V1_7]") { TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; INFO("Using temporary file named: " << tempFile.GetPath()); @@ -3255,7 +3293,7 @@ TEST_CASE("SQLiteIndex_MapDataFolding_PFNs", "[sqliteindex][mapdatafolding]") SQLiteIndex index = SearchTestSetup(tempFile, { { "Id", "Name", "Publisher", "Moniker", "Version1", "", { }, { "Command" }, "Path1", { pfn1 }, { } }, { "Id", "Name", "Publisher", "Moniker", "Version2", "", { }, { "Command" }, "Path2", { pfn2 }, { } }, - }); + }, SQLiteVersion{ 1, 7 }); // Apply the map data folding if it is present in the created test index. index.PrepareForPackaging(); @@ -3312,7 +3350,7 @@ TEST_CASE("SQLiteIndex_MapDataFolding_PFNs", "[sqliteindex][mapdatafolding]") } } -TEST_CASE("SQLiteIndex_MapDataFolding_ProductCodes", "[sqliteindex][mapdatafolding]") +TEST_CASE("SQLiteIndex_MapDataFolding_ProductCodes", "[sqliteindex][mapdatafolding][V1_7]") { TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; INFO("Using temporary file named: " << tempFile.GetPath()); @@ -3323,7 +3361,7 @@ TEST_CASE("SQLiteIndex_MapDataFolding_ProductCodes", "[sqliteindex][mapdatafoldi SQLiteIndex index = SearchTestSetup(tempFile, { { "Id", "Name", "Publisher", "Moniker", "Version1", "", { }, { "Command" }, "Path1", { }, { pc1 } }, { "Id", "Name", "Publisher", "Moniker", "Version2", "", { }, { "Command" }, "Path2", { }, { pc2 } }, - }); + }, SQLiteVersion{ 1, 7 }); // Apply the map data folding if it is present in the created test index. index.PrepareForPackaging(); @@ -3355,3 +3393,370 @@ TEST_CASE("SQLiteIndex_MapDataFolding_ProductCodes", "[sqliteindex][mapdatafoldi REQUIRE(pcValues2.size() == 1); REQUIRE(pcValues1[0] != pcValues2[0]); } + +TEST_CASE("SQLiteIndex_FilePath_Memory", "[sqliteindex]") +{ + SQLiteIndex index = SQLiteIndex::CreateNew(SQLITE_MEMORY_DB_CONNECTION_TARGET); + auto contextData = index.GetContextData(); + REQUIRE(!contextData.Contains(Schema::Property::DatabaseFilePath)); +} + +TEST_CASE("SQLiteIndex_FilePath_Create", "[sqliteindex]") +{ + TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; + + SQLiteIndex index = SQLiteIndex::CreateNew(tempFile); + auto contextData = index.GetContextData(); + REQUIRE(contextData.Contains(Schema::Property::DatabaseFilePath)); + REQUIRE(contextData.Get<Schema::Property::DatabaseFilePath>() == tempFile.GetPath()); +} + +TEST_CASE("SQLiteIndex_FilePath_Open", "[sqliteindex]") +{ + TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; + + { + SQLiteIndex index = SQLiteIndex::CreateNew(tempFile); + } + + SQLiteIndex index = SQLiteIndex::Open(tempFile, SQLiteStorageBase::OpenDisposition::Read); + auto contextData = index.GetContextData(); + REQUIRE(contextData.Contains(Schema::Property::DatabaseFilePath)); + REQUIRE(contextData.Get<Schema::Property::DatabaseFilePath>() == tempFile.GetPath()); +} + +TEST_CASE("SQLiteIndex_MigrateTo_Unsupported", "[sqliteindex][V1_7]") +{ + SQLiteIndex index = SQLiteIndex::CreateNew(SQLITE_MEMORY_DB_CONNECTION_TARGET, SQLiteVersion{ 1, 6 }); + REQUIRE(!index.MigrateTo(SQLiteVersion{ 1, 7 })); +} + +TEST_CASE("SQLiteIndex_MigrateTo_Empty", "[sqliteindex][V2_0]") +{ + TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; + INFO("Using temporary file named: " << tempFile.GetPath()); + + { + SQLiteIndex index = SQLiteIndex::CreateNew(tempFile, SQLiteVersion{ 1, 7 }); + REQUIRE(index.MigrateTo(SQLiteVersion{ 2, 0 })); + REQUIRE(index.GetVersion() == SQLiteVersion{ 2, 0 }); + } + + { + SQLiteIndex index = SQLiteIndex::Open(tempFile, SQLiteStorageBase::OpenDisposition::Read); + REQUIRE(index.GetVersion() == SQLiteVersion{ 2, 0 }); + } +} + +TEST_CASE("SQLiteIndex_MigrateTo_Data", "[sqliteindex][V2_0]") +{ + TempFile tempFile{ "repolibtest_tempdb"s, ".db"s }; + INFO("Using temporary file named: " << tempFile.GetPath()); + + std::string packageId1 = "Id1"; + std::string packageId2 = "Id2"; + std::string packageId3 = "Id3"; + + SQLiteIndex index = SearchTestSetup(tempFile, { + { packageId1, "Name1", "Moniker", "Version", "", { "Tag" }, { "Command" }, "Path1", { "PFN1" }, { "PC1" } }, + { packageId2, "Name2", "Moniker", "Version", "", { "ID3" }, { "Command" }, "Path2", { "PFN2" }, { "PC2" } }, + { packageId3, "Name3", "Moniker", "Version", "", { "Tag" }, { "Command" }, "Path3", { "PFN3" }, { "PC3" } }, + }); + + auto preMigrationVersion = index.GetVersion(); + + if (preMigrationVersion == SQLiteVersion{ 1, 7 }) + { + REQUIRE(index.MigrateTo(SQLiteVersion{ 2, 0 })); + REQUIRE(index.GetVersion() == SQLiteVersion{ 2, 0 }); + + Connection connection = Connection::Create(tempFile, Connection::OpenDisposition::ReadWrite); + auto updateData = Schema::V2_0::PackageUpdateTrackingTable::GetUpdatesSince(connection, 0); + + REQUIRE(updateData.size() == 3); + REQUIRE(std::count_if(updateData.begin(), updateData.end(), [&](const auto& x) { return x.PackageIdentifier == packageId1; }) == 1); + REQUIRE(std::count_if(updateData.begin(), updateData.end(), [&](const auto& x) { return x.PackageIdentifier == packageId2; }) == 1); + REQUIRE(std::count_if(updateData.begin(), updateData.end(), [&](const auto& x) { return x.PackageIdentifier == packageId3; }) == 1); + } + else + { + REQUIRE(!index.MigrateTo(SQLiteVersion{ 2, 0 })); + REQUIRE(index.GetVersion() == preMigrationVersion); + } +} + +TEST_CASE("SQLiteIndex_Property_IntermediateFilePath", "[sqliteindex]") +{ + SQLiteIndex index = SQLiteIndex::CreateNew(SQLITE_MEMORY_DB_CONNECTION_TARGET); + std::filesystem::path intermediateFilePath = "A:\\Path"; + index.SetProperty(SQLiteIndex::Property::IntermediateFileOutputPath, intermediateFilePath.u8string()); + + auto contextData = index.GetContextData(); + REQUIRE(contextData.Contains(Schema::Property::IntermediateFileOutputPath)); + REQUIRE(contextData.Get<Schema::Property::IntermediateFileOutputPath>() == intermediateFilePath); +} + +struct ManifestAndPath +{ + Manifest Manifest; + std::string Path; +}; + +void CreateFakeManifestAndPath( + ManifestAndPath& manifestAndPath, + const string_t& publisher, + std::string_view version = "1.0.0", + std::optional<std::string_view> arpMinVersion = {}, + std::optional<std::string_view> arpMaxVersion = {}) +{ + CreateFakeManifest(manifestAndPath.Manifest, publisher, version); + manifestAndPath.Path = ConvertToUTF8(CreateNewGuidNameWString()); + manifestAndPath.Manifest.StreamSha256 = SHA256::ComputeHash(manifestAndPath.Path); + + if (arpMinVersion) + { + manifestAndPath.Manifest.Installers[0].BaseInstallerType = InstallerTypeEnum::Exe; + manifestAndPath.Manifest.Installers[0].AppsAndFeaturesEntries.push_back({}); + manifestAndPath.Manifest.Installers[0].AppsAndFeaturesEntries.back().DisplayVersion = arpMinVersion.value(); + } + + if (arpMaxVersion) + { + manifestAndPath.Manifest.Installers[0].BaseInstallerType = InstallerTypeEnum::Exe; + manifestAndPath.Manifest.Installers[0].AppsAndFeaturesEntries.push_back({}); + manifestAndPath.Manifest.Installers[0].AppsAndFeaturesEntries.back().DisplayVersion = arpMaxVersion.value(); + } +} + +std::filesystem::path GetOnlyChild(const std::filesystem::path& parent) +{ + auto parentDirectoryIterator = std::filesystem::directory_iterator{ parent }; + std::filesystem::path result = parentDirectoryIterator->path(); + REQUIRE(++parentDirectoryIterator == std::filesystem::directory_iterator{}); + return result; +} + +void CheckIntermediates(const std::filesystem::path& intermediatesDirectory, const std::vector<std::vector<ManifestAndPath>>& expectedIntermediatesData, std::chrono::seconds sleep = 1s) +{ + size_t intermediatePackageCount = std::count_if(std::filesystem::directory_iterator{ intermediatesDirectory }, std::filesystem::directory_iterator{}, [](const auto&){ return true; }); + REQUIRE(intermediatePackageCount == expectedIntermediatesData.size()); + + for (const auto& versions : expectedIntermediatesData) + { + REQUIRE(!versions.empty()); + INFO(versions[0].Manifest.Id); + std::filesystem::path packageDirectory = intermediatesDirectory / ConvertToUTF16(versions[0].Manifest.Id); + + REQUIRE(std::filesystem::exists(packageDirectory)); + std::filesystem::path hashDirectory = GetOnlyChild(packageDirectory); + + SHA256::HashBuffer hashBytes = SHA256::ConvertToBytes(hashDirectory.filename().u8string()); + std::filesystem::path versionDataFile = GetOnlyChild(hashDirectory); + std::ifstream versionDataStream{ versionDataFile, std::ios_base::in | std::ios_base::binary }; + auto versionDataBytes = ReadEntireStreamAsByteArray(versionDataStream); + SHA256::HashBuffer versionDataHash = SHA256::ComputeHash(versionDataBytes); + REQUIRE(SHA256::AreEqual(hashBytes, versionDataHash)); + + PackageVersionDataManifest versionDataManifest; + versionDataManifest.Deserialize(PackageVersionDataManifest::CreateDecompressor().Decompress(versionDataBytes)); + + const auto& versionDataVersions = versionDataManifest.Versions(); + REQUIRE(versionDataVersions.size() == versions.size()); + + for (const auto& manifestAndPath : versions) + { + const auto& versionDataItr = std::find_if(versionDataVersions.begin(), versionDataVersions.end(), [&](const auto& v) { return v.Version == manifestAndPath.Manifest.Version; }); + REQUIRE(versionDataItr != versionDataVersions.end()); + const auto& versionData = *versionDataItr; + + REQUIRE(manifestAndPath.Path == versionData.ManifestRelativePath); + REQUIRE(SHA256::ConvertToString(manifestAndPath.Manifest.StreamSha256) == versionData.ManifestHash); + + auto versionRange = manifestAndPath.Manifest.GetArpVersionRange(); + if (!versionRange.IsEmpty()) + { + REQUIRE(versionData.ArpMinVersion); + REQUIRE(versionRange.GetMinVersion() == versionData.ArpMinVersion.value()); + REQUIRE(versionData.ArpMaxVersion); + REQUIRE(versionRange.GetMaxVersion() == versionData.ArpMaxVersion.value()); + } + } + } + + // This is needed to force the timestamp to roll over to a new value for the next call to this function. + // An alternate solution would be to hook the timestamp function and control the values it returns + // so that we can advance/halt time arbitrarily. + std::this_thread::sleep_for(sleep); +} + +void PrepareAndCheckIntermediates(const std::filesystem::path& baseFile, const std::filesystem::path& preparedFile, const std::vector<std::vector<ManifestAndPath>>& expectedIntermediatesData, std::chrono::seconds sleep = 1s) +{ + TempDirectory intermediatesDirectory{ "v2_0_intermediates" }; + INFO("Intermediates directory: " << intermediatesDirectory.GetPath()); + + std::filesystem::copy_file(baseFile, preparedFile, std::filesystem::copy_options::overwrite_existing); + + SQLiteIndex index = SQLiteIndex::Open(preparedFile.u8string(), SQLiteStorageBase::OpenDisposition::ReadWrite); + index.SetProperty(SQLiteIndex::Property::IntermediateFileOutputPath, intermediatesDirectory); + index.PrepareForPackaging(); + + CheckIntermediates(intermediatesDirectory, expectedIntermediatesData, sleep); +} + +TEST_CASE("SQLiteIndex_V2_0_UsageFlow_Simple", "[sqliteindex][V2_0]") +{ + TempFile baseFile{ "v2_0_index_tempdb"s, ".db"s }; + TempFile preparedFile{ "v2_0_index_prepared_tempdb"s, ".db"s }; + INFO("Using files named: [" << baseFile.GetPath() << "] and [" << preparedFile.GetPath() << "]"); + + // Create empty index + std::ignore = SQLiteIndex::CreateNew(baseFile, SQLiteVersion{ 2, 0 }); + + std::string publisher = "Publisher"; + ManifestAndPath manifest1; + CreateFakeManifestAndPath(manifest1, publisher, "1.0"); + + { + // Open existing file to add a manifest + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.SetProperty(SQLiteIndex::Property::PackageUpdateTrackingBaseTime, ""); + index.AddManifest(manifest1.Manifest, manifest1.Path); + } + + PrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest1 } }, 0s); +} + +TEST_CASE("SQLiteIndex_V2_0_UsageFlow_Complex", "[sqliteindex][V2_0]") +{ + TempFile baseFile{ "v2_0_index_tempdb"s, ".db"s }; + TempFile preparedFile{ "v2_0_index_prepared_tempdb"s, ".db"s }; + INFO("Using files named: [" << baseFile.GetPath() << "] and [" << preparedFile.GetPath() << "]"); + + // Create empty index + std::ignore = SQLiteIndex::CreateNew(baseFile, SQLiteVersion{ 2, 0 }); + + // Open existing file to add a new package + std::string Publisher1 = "Publisher1"; + ManifestAndPath manifest1; + CreateFakeManifestAndPath(manifest1, Publisher1, "1.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.SetProperty(SQLiteIndex::Property::PackageUpdateTrackingBaseTime, ""); + index.AddManifest(manifest1.Manifest, manifest1.Path); + } + + PrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest1 } }); + + // Open existing file to add another new package + ManifestAndPath manifest2; + CreateFakeManifestAndPath(manifest2, "Publisher2", "1.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.SetProperty(SQLiteIndex::Property::PackageUpdateTrackingBaseTime, ""); + index.AddManifest(manifest2.Manifest, manifest2.Path); + } + + PrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest2 } }); + + // Open existing file to add a new version of existing package + ManifestAndPath manifest3; + CreateFakeManifestAndPath(manifest3, Publisher1, "2.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.SetProperty(SQLiteIndex::Property::PackageUpdateTrackingBaseTime, ""); + index.AddManifest(manifest3.Manifest, manifest3.Path); + } + + PrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest1, manifest3 } }); + + // Open existing file to add a new verion of existing package and update an existing version + manifest2.Manifest.StreamSha256 = SHA256::ComputeHash(manifest2.Manifest.Id); + + ManifestAndPath manifest4; + CreateFakeManifestAndPath(manifest4, Publisher1, "3.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.SetProperty(SQLiteIndex::Property::PackageUpdateTrackingBaseTime, ""); + index.UpdateManifest(manifest2.Manifest, manifest2.Path); + index.AddManifest(manifest4.Manifest, manifest4.Path); + } + + PrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest2 }, { manifest1, manifest3, manifest4 } }, 0s); +} + +void MigratePrepareAndCheckIntermediates(const std::filesystem::path& baseFile, const std::filesystem::path& preparedFile, const std::vector<std::vector<ManifestAndPath>>& expectedIntermediatesData) +{ + TempDirectory intermediatesDirectory{ "v2_0_intermediates" }; + INFO("Intermediates directory: " << intermediatesDirectory.GetPath()); + + std::filesystem::copy_file(baseFile, preparedFile, std::filesystem::copy_options::overwrite_existing); + + SQLiteIndex index = SQLiteIndex::Open(preparedFile.u8string(), SQLiteStorageBase::OpenDisposition::ReadWrite); + index.MigrateTo({ 2, 0 }); + index.SetProperty(SQLiteIndex::Property::IntermediateFileOutputPath, intermediatesDirectory); + index.PrepareForPackaging(); + + CheckIntermediates(intermediatesDirectory, expectedIntermediatesData, 0s); +} + +TEST_CASE("SQLiteIndex_V2_0_UsageFlow_ComplexMigration", "[sqliteindex][V2_0]") +{ + TempFile baseFile{ "v1_7_index_tempdb"s, ".db"s }; + TempFile preparedFile{ "v2_0_index_prepared_tempdb"s, ".db"s }; + INFO("Using files named: [" << baseFile.GetPath() << "] and [" << preparedFile.GetPath() << "]"); + + // Create empty index + std::ignore = SQLiteIndex::CreateNew(baseFile, SQLiteVersion{ 1, 7 }); + + // Open existing file to add a new package + std::string Publisher1 = "Publisher1"; + ManifestAndPath manifest1; + CreateFakeManifestAndPath(manifest1, Publisher1, "1.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.AddManifest(manifest1.Manifest, manifest1.Path); + } + + MigratePrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest1 } }); + + // Open existing file to add another new package + ManifestAndPath manifest2; + CreateFakeManifestAndPath(manifest2, "Publisher2", "1.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.AddManifest(manifest2.Manifest, manifest2.Path); + } + + MigratePrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest2 }, { manifest1 } }); + + // Open existing file to add a new version of existing package + ManifestAndPath manifest3; + CreateFakeManifestAndPath(manifest3, Publisher1, "2.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.AddManifest(manifest3.Manifest, manifest3.Path); + } + + MigratePrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest2 }, { manifest1, manifest3 } }); + + // Open existing file to add a new verion of existing package and update an existing version + manifest2.Manifest.StreamSha256 = SHA256::ComputeHash(manifest2.Manifest.Id); + + ManifestAndPath manifest4; + CreateFakeManifestAndPath(manifest4, Publisher1, "3.0"); + + { + SQLiteIndex index = SQLiteIndex::Open(baseFile, SQLiteStorageBase::OpenDisposition::ReadWrite); + index.UpdateManifest(manifest2.Manifest, manifest2.Path); + index.AddManifest(manifest4.Manifest, manifest4.Path); + } + + MigratePrepareAndCheckIntermediates(baseFile, preparedFile, { { manifest2 }, { manifest1, manifest3, manifest4 } }); +} diff --git a/src/AppInstallerCommonCore/AppInstallerCommonCore.vcxproj b/src/AppInstallerCommonCore/AppInstallerCommonCore.vcxproj @@ -440,6 +440,7 @@ <ClInclude Include="Public\winget\Filesystem.h" /> <ClInclude Include="Public\winget\NetworkSettings.h" /> <ClInclude Include="Public\winget\PackageDependenciesValidationUtil.h" /> + <ClInclude Include="Public\winget\PackageVersionDataManifest.h" /> <ClInclude Include="Public\winget\Pin.h" /> <ClInclude Include="Public\winget\Reboot.h" /> <ClInclude Include="Public\winget\Regex.h" /> @@ -491,6 +492,7 @@ <ClCompile Include="NameNormalization.cpp" /> <ClCompile Include="NetworkSettings.cpp" /> <ClCompile Include="PackageDependenciesValidationUtil.cpp" /> + <ClCompile Include="PackageVersionDataManifest.cpp" /> <ClCompile Include="Pin.cpp" /> <ClCompile Include="Progress.cpp" /> <ClCompile Include="Reboot.cpp" /> diff --git a/src/AppInstallerCommonCore/AppInstallerCommonCore.vcxproj.filters b/src/AppInstallerCommonCore/AppInstallerCommonCore.vcxproj.filters @@ -192,6 +192,9 @@ <ClInclude Include="Public\winget\NetworkSettings.h"> <Filter>Public\winget</Filter> </ClInclude> + <ClInclude Include="Public\winget\PackageVersionDataManifest.h"> + <Filter>Public\winget</Filter> + </ClInclude> <ClInclude Include="Public\winget\HttpClientHelper.h"> <Filter>Public\winget</Filter> </ClInclude> @@ -350,6 +353,9 @@ <ClCompile Include="NetworkSettings.cpp"> <Filter>Source Files</Filter> </ClCompile> + <ClCompile Include="PackageVersionDataManifest.cpp"> + <Filter>Source Files</Filter> + </ClCompile> <ClCompile Include="HttpClientHelper.cpp"> <Filter>Source Files</Filter> </ClCompile> diff --git a/src/AppInstallerCommonCore/PackageVersionDataManifest.cpp b/src/AppInstallerCommonCore/PackageVersionDataManifest.cpp @@ -0,0 +1,177 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "Public/winget/PackageVersionDataManifest.h" +#include "Public/winget/Yaml.h" +#include <AppInstallerErrors.h> + +using namespace std::string_view_literals; + +namespace AppInstaller::Manifest +{ + // These shortened names save some bytes since humans are neither authoring them nor reading them (except to debug). + static constexpr std::string_view s_FieldName_SchemaVersion = "sV"sv; + static constexpr std::string_view s_FieldName_VersionData = "vD"sv; + static constexpr std::string_view s_FieldName_Version = "v"sv; + static constexpr std::string_view s_FieldName_ArpMinVersion = "aMiV"sv; + static constexpr std::string_view s_FieldName_ArpMaxVersion = "aMaV"sv; + static constexpr std::string_view s_FieldName_RelativePath = "rP"sv; + static constexpr std::string_view s_FieldName_Sha256Hash = "s256H"sv; + + static constexpr std::string_view s_SchemaVersion_1_0 = "1.0"sv; + + static constexpr DWORD CompressionAlgorithm = COMPRESS_ALGORITHM_MSZIP; + static constexpr bool CompressionSetLevel1 = false; + + namespace anon + { + std::string GetRequiredChildString(const YAML::Node& node, std::string_view childName) + { + const YAML::Node& childNode = node.GetChildNode(childName); + THROW_HR_IF(APPINSTALLER_CLI_ERROR_INVALID_MANIFEST, !childNode.IsScalar()); + return childNode.as<std::string>(); + } + + std::optional<std::string> GetOptionalChildString(const YAML::Node& node, std::string_view childName) + { + const YAML::Node& childNode = node.GetChildNode(childName); + return childNode.IsScalar() ? std::make_optional(childNode.as<std::string>()) : std::nullopt; + } + + void Deserialize_1_0(const YAML::Node& document, PackageVersionDataManifest& manifest) + { + const YAML::Node& versionDataItems = document.GetChildNode(s_FieldName_VersionData); + THROW_HR_IF(APPINSTALLER_CLI_ERROR_INVALID_MANIFEST, !versionDataItems.IsSequence()); + + for (const YAML::Node& item : versionDataItems.Sequence()) + { + THROW_HR_IF(APPINSTALLER_CLI_ERROR_INVALID_MANIFEST, !item.IsMap()); + + PackageVersionDataManifest::VersionData versionData; + + versionData.Version.Assign(GetRequiredChildString(item, s_FieldName_Version)); + versionData.ArpMinVersion = GetOptionalChildString(item, s_FieldName_ArpMinVersion); + versionData.ArpMaxVersion = GetOptionalChildString(item, s_FieldName_ArpMaxVersion); + versionData.ManifestRelativePath = GetRequiredChildString(item, s_FieldName_RelativePath); + versionData.ManifestHash = GetRequiredChildString(item, s_FieldName_Sha256Hash); + + manifest.AddVersion(std::move(versionData)); + } + } + } + + std::string_view PackageVersionDataManifest::VersionManifestFileName() + { + return "versionData.yml"sv; + } + + std::string_view PackageVersionDataManifest::VersionManifestCompressedFileName() + { + return "versionData.mszyml"sv; + } + + Compression::Compressor PackageVersionDataManifest::CreateCompressor() + { + Compression::Compressor result(CompressionAlgorithm); + if constexpr (CompressionSetLevel1) + { + result.SetInformation(COMPRESS_INFORMATION_CLASS_LEVEL, 1); + } + return result; + } + + Compression::Decompressor PackageVersionDataManifest::CreateDecompressor() + { + return Compression::Decompressor(CompressionAlgorithm); + } + + PackageVersionDataManifest::VersionData::VersionData( + const Utility::VersionAndChannel& versionAndChannel, + std::optional<std::string> arpMinVersion, + std::optional<std::string> arpMaxVersion, + std::optional<std::string> relativePath, + std::optional<std::string> manifestHash) : + Version(versionAndChannel.GetVersion()), + ArpMinVersion(std::move(arpMinVersion)), + ArpMaxVersion(std::move(arpMaxVersion)), + ManifestRelativePath(std::move(relativePath).value_or("")), + ManifestHash(std::move(manifestHash).value_or("")) + { + if (ArpMinVersion && ArpMinVersion->empty()) + { + ArpMinVersion.reset(); + } + + if (ArpMaxVersion && ArpMaxVersion->empty()) + { + ArpMaxVersion.reset(); + } + } + + void PackageVersionDataManifest::AddVersion(VersionData&& versionData) + { + m_versions.emplace_back(std::move(versionData)); + } + + const std::vector<PackageVersionDataManifest::VersionData>& PackageVersionDataManifest::Versions() const + { + return m_versions; + } + + std::string PackageVersionDataManifest::Serialize() + { + YAML::Emitter out; + out << YAML::BeginMap; + out << YAML::Key << s_FieldName_SchemaVersion << YAML::Value << s_SchemaVersion_1_0; + + out << YAML::Key << s_FieldName_VersionData; + out << YAML::BeginSeq; + + for (const auto& version : m_versions) + { + out << YAML::BeginMap; + out << YAML::Key << s_FieldName_Version << YAML::Value << version.Version.ToString(); + if (version.ArpMinVersion) + { + out << YAML::Key << s_FieldName_ArpMinVersion << YAML::Value << version.ArpMinVersion.value(); + } + if (version.ArpMaxVersion) + { + out << YAML::Key << s_FieldName_ArpMaxVersion << YAML::Value << version.ArpMaxVersion.value(); + } + out << YAML::Key << s_FieldName_RelativePath << YAML::Value << version.ManifestRelativePath; + out << YAML::Key << s_FieldName_Sha256Hash << YAML::Value << version.ManifestHash; + out << YAML::EndMap; + } + + out << YAML::EndSeq; + out << YAML::EndMap; + + return out.str(); + } + + void PackageVersionDataManifest::Deserialize(std::string_view input) + { + YAML::Node document = YAML::Load(input); + THROW_HR_IF(APPINSTALLER_CLI_ERROR_INVALID_MANIFEST, !document.IsMap()); + + const YAML::Node& schemaVersionNode = document.GetChildNode(s_FieldName_SchemaVersion); + THROW_HR_IF(APPINSTALLER_CLI_ERROR_INVALID_MANIFEST, !schemaVersionNode.IsScalar()); + + Utility::Version schemaVersion{ schemaVersionNode.as<std::string>() }; + + if (schemaVersion.PartAt(0).Integer == 1) + { + anon::Deserialize_1_0(document, *this); + } + else + { + THROW_HR(APPINSTALLER_CLI_ERROR_UNSUPPORTED_MANIFESTVERSION); + } + } + + void PackageVersionDataManifest::Deserialize(const std::vector<uint8_t>& input) + { + Deserialize(std::string_view{ reinterpret_cast<const char*>(input.data()), input.size() }); + } +} diff --git a/src/AppInstallerCommonCore/Public/winget/PackageVersionDataManifest.h b/src/AppInstallerCommonCore/Public/winget/PackageVersionDataManifest.h @@ -0,0 +1,62 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <AppInstallerVersions.h> +#include <winget/Compression.h> + + +namespace AppInstaller::Manifest +{ + // Contains the manifest that stores package version data for index v2 + struct PackageVersionDataManifest + { + // The file name to use for the package version data manifest. + static std::string_view VersionManifestFileName(); + + // The file name to use for the compressed package version data manifest. + static std::string_view VersionManifestCompressedFileName(); + + // Creates the compressor used by the PackageVersionDataManifest. + static Compression::Compressor CreateCompressor(); + + // Creates the decompressor used by the PackageVersionDataManifest. + static Compression::Decompressor CreateDecompressor(); + + // Data on an individual version. + struct VersionData + { + VersionData() = default; + + VersionData( + const Utility::VersionAndChannel& versionAndChannel, + std::optional<std::string> arpMinVersion, + std::optional<std::string> arpMaxVersion, + std::optional<std::string> relativePath, + std::optional<std::string> manifestHash); + + Utility::Version Version; + std::optional<std::string> ArpMinVersion; + std::optional<std::string> ArpMaxVersion; + std::string ManifestRelativePath; + std::string ManifestHash; + }; + + // Adds the given version data to the manifest. + void AddVersion(VersionData&& versionData); + + // Gets the version data in this object. + const std::vector<VersionData>& Versions() const; + + // Returns a serialized version of the current manifest data. + std::string Serialize(); + + // Parses the input into this objects data. + void Deserialize(std::string_view input); + + // Parses the input into this objects data. + void Deserialize(const std::vector<uint8_t>& input); + + private: + std::vector<VersionData> m_versions; + }; +} diff --git a/src/AppInstallerRepositoryCore/AppInstallerRepositoryCore.vcxproj b/src/AppInstallerRepositoryCore/AppInstallerRepositoryCore.vcxproj @@ -384,10 +384,24 @@ <ClInclude Include="Microsoft\Schema\1_6\SearchResultsTable.h" /> <ClInclude Include="Microsoft\Schema\1_6\UpgradeCodeTable.h" /> <ClInclude Include="Microsoft\Schema\1_7\Interface.h" /> + <ClInclude Include="Microsoft\Schema\2_0\CommandsTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\Interface.h" /> + <ClInclude Include="Microsoft\Schema\2_0\NormalizedPackageNameTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\NormalizedPackagePublisherTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\PackageFamilyNameTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\PackagesTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\OneToManyTableWithMap.h" /> + <ClInclude Include="Microsoft\Schema\2_0\PackageUpdateTrackingTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\ProductCodeTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\SearchResultsTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\SystemReferenceStringTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\TagsTable.h" /> + <ClInclude Include="Microsoft\Schema\2_0\UpgradeCodeTable.h" /> <ClInclude Include="Microsoft\Schema\IPinningIndex.h" /> <ClInclude Include="Microsoft\Schema\IPortableIndex.h" /> <ClInclude Include="Microsoft\Schema\ICheckpointDatabase.h" /> <ClInclude Include="Microsoft\Schema\ISQLiteIndex.h" /> + <ClInclude Include="Microsoft\Schema\SQLiteIndexContextData.h" /> <ClInclude Include="Microsoft\Schema\Pinning_1_0\PinningIndexInterface.h" /> <ClInclude Include="Microsoft\Schema\Pinning_1_0\PinTable.h" /> <ClInclude Include="Microsoft\Schema\Portable_1_0\PortableIndexInterface.h" /> @@ -481,6 +495,13 @@ <ClCompile Include="Microsoft\Schema\1_6\Interface_1_6.cpp" /> <ClCompile Include="Microsoft\Schema\1_6\SearchResultsTable_1_6.cpp" /> <ClCompile Include="Microsoft\Schema\1_7\Interface_1_7.cpp" /> + <ClCompile Include="Microsoft\Schema\2_0\Interface_2_0.cpp" /> + <ClCompile Include="Microsoft\Schema\2_0\PackagesTable.cpp" /> + <ClCompile Include="Microsoft\Schema\2_0\OneToManyTableWithMap.cpp" /> + <ClCompile Include="Microsoft\Schema\2_0\PackageUpdateTrackingTable.cpp" /> + <ClCompile Include="Microsoft\Schema\2_0\SearchResultsTable_2_0.cpp" /> + <ClCompile Include="Microsoft\Schema\2_0\SystemReferenceStringTable.cpp" /> + <ClCompile Include="Microsoft\Schema\ISQLiteIndex.cpp" /> <ClCompile Include="Microsoft\Schema\Pinning_1_0\PinningIndexInterface_1_0.cpp" /> <ClCompile Include="Microsoft\Schema\Pinning_1_0\PinTable.cpp" /> <ClCompile Include="Microsoft\Schema\Portable_1_0\PortableIndexInterface_1_0.cpp" /> diff --git a/src/AppInstallerRepositoryCore/AppInstallerRepositoryCore.vcxproj.filters b/src/AppInstallerRepositoryCore/AppInstallerRepositoryCore.vcxproj.filters @@ -100,6 +100,9 @@ <Filter Include="Rest\Schema\1_7\Json"> <UniqueIdentifier>{7fd6c265-81c0-4c6e-87b2-24bef117e21d}</UniqueIdentifier> </Filter> + <Filter Include="Microsoft\Schema\2_0"> + <UniqueIdentifier>{34442899-29e5-4183-96ba-a1e8740146be}</UniqueIdentifier> + </Filter> </ItemGroup> <ItemGroup> <ClInclude Include="pch.h"> @@ -411,6 +414,48 @@ <ClInclude Include="Rest\Schema\1_7\Json\ManifestDeserializer.h"> <Filter>Rest\Schema\1_7\Json</Filter> </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\CommandsTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\Interface.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\PackagesTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\OneToManyTableWithMap.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\SearchResultsTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\TagsTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\SystemReferenceStringTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\PackageFamilyNameTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\NormalizedPackageNameTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\NormalizedPackagePublisherTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\ProductCodeTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\UpgradeCodeTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\2_0\PackageUpdateTrackingTable.h"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClInclude> + <ClInclude Include="Microsoft\Schema\SQLiteIndexContextData.h"> + <Filter>Microsoft\Schema</Filter> + </ClInclude> </ItemGroup> <ItemGroup> <ClCompile Include="pch.cpp"> @@ -644,6 +689,27 @@ <ClCompile Include="Rest\Schema\1_7\Json\ManifestDeserializer_1_7.cpp"> <Filter>Rest\Schema\1_7\Json</Filter> </ClCompile> + <ClCompile Include="Microsoft\Schema\2_0\Interface_2_0.cpp"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClCompile> + <ClCompile Include="Microsoft\Schema\2_0\PackagesTable.cpp"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClCompile> + <ClCompile Include="Microsoft\Schema\2_0\OneToManyTableWithMap.cpp"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClCompile> + <ClCompile Include="Microsoft\Schema\2_0\SearchResultsTable_2_0.cpp"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClCompile> + <ClCompile Include="Microsoft\Schema\ISQLiteIndex.cpp"> + <Filter>Microsoft\Schema</Filter> + </ClCompile> + <ClCompile Include="Microsoft\Schema\2_0\SystemReferenceStringTable.cpp"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClCompile> + <ClCompile Include="Microsoft\Schema\2_0\PackageUpdateTrackingTable.cpp"> + <Filter>Microsoft\Schema\2_0</Filter> + </ClCompile> </ItemGroup> <ItemGroup> <None Include="PropertySheet.props" /> diff --git a/src/AppInstallerRepositoryCore/Microsoft/SQLiteIndex.cpp b/src/AppInstallerRepositoryCore/Microsoft/SQLiteIndex.cpp @@ -6,15 +6,6 @@ #include "ArpVersionValidation.h" #include <winget/ManifestYamlParser.h> -#include "Schema/1_0/Interface.h" -#include "Schema/1_1/Interface.h" -#include "Schema/1_2/Interface.h" -#include "Schema/1_3/Interface.h" -#include "Schema/1_4/Interface.h" -#include "Schema/1_5/Interface.h" -#include "Schema/1_6/Interface.h" -#include "Schema/1_7/Interface.h" - namespace AppInstaller::Repository::Microsoft { SQLiteIndex SQLiteIndex::CreateNew(const std::string& filePath, SQLite::Version version, CreateOptions options) @@ -46,37 +37,12 @@ namespace AppInstaller::Repository::Microsoft return { filePath, source }; } - std::unique_ptr<Schema::ISQLiteIndex> SQLiteIndex::CreateISQLiteIndex(const SQLite::Version& version) - { - using namespace Schema; - - if (version.MajorVersion == 1 || - version.IsLatest()) - { - constexpr std::array<std::unique_ptr<Schema::ISQLiteIndex>(*)(), 8> versionCreatorMap = - { - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_0::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_1::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_2::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_3::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_4::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_5::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_6::Interface>()); }, - []() { return std::unique_ptr<Schema::ISQLiteIndex>(std::make_unique<V1_7::Interface>()); }, - }; - - return versionCreatorMap[std::min(static_cast<size_t>(version.MinorVersion), versionCreatorMap.size() - 1)](); - } - - // We do not have the capacity to operate on this schema version - THROW_HR(HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED)); - } - SQLiteIndex::SQLiteIndex(const std::string& target, const SQLite::Version& version) : SQLiteStorageBase(target, version) { m_dbconn.EnableICU(); - m_interface = CreateISQLiteIndex(version); + m_interface = Schema::CreateISQLiteIndex(version); m_version = m_interface->GetVersion(); + SetDatabaseFilePath(target); } SQLiteIndex::SQLiteIndex(const std::string& target, SQLiteStorageBase::OpenDisposition disposition, Utility::ManagedFile&& indexFile) : @@ -84,26 +50,41 @@ namespace AppInstaller::Repository::Microsoft { m_dbconn.EnableICU(); AICLI_LOG(Repo, Info, << "Opened SQLite Index with version [" << m_version << "], last write [" << GetLastWriteTime() << "]"); - m_interface = CreateISQLiteIndex(m_version); + m_interface = Schema::CreateISQLiteIndex(m_version); THROW_HR_IF(APPINSTALLER_CLI_ERROR_CANNOT_WRITE_TO_UPLEVEL_INDEX, disposition == SQLiteStorageBase::OpenDisposition::ReadWrite && m_version != m_interface->GetVersion()); + SetDatabaseFilePath(target); } SQLiteIndex::SQLiteIndex(const std::string& target, SQLiteIndex& source) : SQLiteStorageBase(target, source) { m_dbconn.EnableICU(); - m_interface = CreateISQLiteIndex(m_version); + m_interface = Schema::CreateISQLiteIndex(m_version); + SetDatabaseFilePath(target); + } + + void SQLiteIndex::SetDatabaseFilePath(const std::string& target) + { + if (target != SQLITE_MEMORY_DB_CONNECTION_TARGET) + { + m_contextData.Add<Schema::Property::DatabaseFilePath>(Utility::ConvertToUTF16(target)); + } } #ifndef AICLI_DISABLE_TEST_HOOKS void SQLiteIndex::ForceVersion(const SQLite::Version& version) { - m_interface = CreateISQLiteIndex(version); + m_interface = Schema::CreateISQLiteIndex(version); } SQLite::Version SQLiteIndex::GetLatestVersion() { - return CreateISQLiteIndex(SQLite::Version::Latest())->GetVersion(); + return Schema::CreateISQLiteIndex(SQLite::Version::Latest())->GetVersion(); + } + + const Schema::SQLiteIndexContextData& SQLiteIndex::GetContextData() const + { + return m_contextData; } #endif @@ -220,7 +201,7 @@ namespace AppInstaller::Repository::Microsoft std::lock_guard<std::mutex> lockInterface{ *m_interfaceLock }; AICLI_LOG(Repo, Info, << "Preparing index for packaging"); - m_interface->PrepareForPackaging(m_dbconn); + m_interface->PrepareForPackaging(Schema::SQLiteIndexContext{ m_dbconn, m_contextData }); } bool SQLiteIndex::CheckConsistency(bool log) const @@ -299,4 +280,47 @@ namespace AppInstaller::Repository::Microsoft { return m_interface->GetDependentsById(m_dbconn, packageId); } + + bool SQLiteIndex::MigrateTo(SQLite::Version version) + { + std::lock_guard<std::mutex> lockInterface{ *m_interfaceLock }; + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(m_dbconn, "sqliteindex_migrate_to"); + + AICLI_LOG(Repo, Info, << "Attempting to migrate index from [" << m_interface->GetVersion() << "] to [" << version << "]..."); + std::unique_ptr<Schema::ISQLiteIndex> newInterface = Schema::CreateISQLiteIndex(version); + + bool result = newInterface->MigrateFrom(m_dbconn, m_interface.get()); + + AICLI_LOG(Repo, Info, << "...migration was " << (result ? "" : "NOT ") << "successful"); + if (result) + { + version.SetSchemaVersion(m_dbconn); + SetLastWriteTime(); + savepoint.Commit(); + + m_version = version; + m_interface = std::move(newInterface); + } + + return result; + } + + void SQLiteIndex::SetProperty(Property property, const std::string& value) + { + std::lock_guard<std::mutex> lockInterface{ *m_interfaceLock }; + + switch (property) + { + case Property::PackageUpdateTrackingBaseTime: + m_interface->SetProperty(m_dbconn, Schema::Property::PackageUpdateTrackingBaseTime, value); + break; + case Property::IntermediateFileOutputPath: + { + std::filesystem::path pathValue{ Utility::ConvertToUTF16(value) }; + THROW_HR_IF(E_INVALIDARG, pathValue.empty() || pathValue.is_relative()); + m_contextData.Add<Schema::Property::IntermediateFileOutputPath>(std::move(pathValue)); + } + break; + } + } } diff --git a/src/AppInstallerRepositoryCore/Microsoft/SQLiteIndex.h b/src/AppInstallerRepositoryCore/Microsoft/SQLiteIndex.h @@ -62,6 +62,9 @@ namespace AppInstaller::Repository::Microsoft // Gets the latest version of the index schema (the actual numbers, not just the latest sentinel values). static SQLite::Version GetLatestVersion(); + + // Gets the context data for testing. + const Schema::SQLiteIndexContextData& GetContextData() const; #endif // Adds the manifest at the repository relative path to the index. @@ -143,6 +146,21 @@ namespace AppInstaller::Repository::Microsoft std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependenciesByManifestRowId(SQLite::rowid_t manifestRowId) const; std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependentsById(AppInstaller::Manifest::string_t packageId) const; + // Migrates the index to the target version. + // Returns false to indicate that the requested migration is not supported. + bool MigrateTo(SQLite::Version version); + + // The property values that can be set. + enum class Property + { + PackageUpdateTrackingBaseTime, + IntermediateFileOutputPath, + }; + + // Sets the given property. + // Some properties will persist into the database. + void SetProperty(Property property, const std::string& value); + private: // Constructor used to create a new index. SQLiteIndex(const std::string& target, const SQLite::Version& version); @@ -153,13 +171,14 @@ namespace AppInstaller::Repository::Microsoft // Constructor used to copy the given index. SQLiteIndex(const std::string& target, SQLiteIndex& source); + // Sets the database file path in the context data if appropriate. + void SetDatabaseFilePath(const std::string& target); + // Internal functions to normalize on the relativePath being present. IdType AddManifestInternal(const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath); bool UpdateManifestInternal(const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath); - // Creates the ISQLiteIndex interface object for this version. - static std::unique_ptr<Schema::ISQLiteIndex> CreateISQLiteIndex(const SQLite::Version& version); - std::unique_ptr<Schema::ISQLiteIndex> m_interface; + Schema::SQLiteIndexContextData m_contextData; }; -}- \ No newline at end of file +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/Interface.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/Interface.h @@ -39,7 +39,13 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Version 1.4 Get all the dependencies for a specific manifest. std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t manifestRowId) const override; - std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const override; + std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const override; + + // Version 1.7 + void DropTables(SQLite::Connection& connection) override; + + // Version 2.0 + bool MigrateFrom(SQLite::Connection& connection, const ISQLiteIndex* current) override; protected: virtual bool NotNeeded(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t id) const; @@ -47,9 +53,6 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Creates the search results table. virtual std::unique_ptr<SearchResultsTable> CreateSearchResultsTable(const SQLite::Connection& connection) const; - // Gets the ordering of matches to execute, with more specific matches coming first. - virtual std::vector<MatchType> GetMatchTypeOrder(MatchType type) const; - // Executes all relevant searches for the query. virtual void PerformQuerySearch(SearchResultsTable& resultsTable, const RequestMatch& query) const; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/Interface_1_0.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/Interface_1_0.cpp @@ -465,7 +465,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 { for (auto include : request.Inclusions) { - for (MatchType match : GetMatchTypeOrder(include.Type)) + for (MatchType match : GetDefaultMatchTypeOrder(include.Type)) { include.Type = match; resultsTable->SearchOnField(include); @@ -483,7 +483,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Perform search for just the field matching the first filter PackageMatchFilter filter = request.Filters[0]; - for (MatchType match : GetMatchTypeOrder(filter.Type)) + for (MatchType match : GetDefaultMatchTypeOrder(filter.Type)) { filter.Type = match; resultsTable->SearchOnField(filter); @@ -503,7 +503,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 resultsTable->PrepareToFilter(); - for (MatchType match : GetMatchTypeOrder(filter.Type)) + for (MatchType match : GetDefaultMatchTypeOrder(filter.Type)) { filter.Type = match; resultsTable->FilterOnField(filter); @@ -520,9 +520,17 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 return GetPropertyByManifestIdInternal(connection, manifestId, property); } - std::vector<std::string> Interface::GetMultiPropertyByManifestId(const SQLite::Connection&, SQLite::rowid_t, PackageVersionMultiProperty) const + std::vector<std::string> Interface::GetMultiPropertyByManifestId(const SQLite::Connection& connection, SQLite::rowid_t rowid, PackageVersionMultiProperty property) const { - return {}; + switch (property) + { + case PackageVersionMultiProperty::Tag: + return TagsTable::GetValuesByManifestId(connection, rowid); + case PackageVersionMultiProperty::Command: + return CommandsTable::GetValuesByManifestId(connection, rowid); + default: + return {}; + } } std::optional<SQLite::rowid_t> Interface::GetManifestIdByKey(const SQLite::Connection& connection, SQLite::rowid_t id, std::string_view version, std::string_view channel) const @@ -545,6 +553,31 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 return {}; } + void Interface::DropTables(SQLite::Connection& connection) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "drop_tables_v1_0"); + + IdTable::Drop(connection); + NameTable::Drop(connection); + MonikerTable::Drop(connection); + VersionTable::Drop(connection); + ChannelTable::Drop(connection); + + PathPartTable::Drop(connection); + + ManifestTable::Drop(connection); + + TagsTable::Drop(connection); + CommandsTable::Drop(connection); + + savepoint.Commit(); + } + + bool Interface::MigrateFrom(SQLite::Connection&, const ISQLiteIndex*) + { + return false; + } + std::vector<ISQLiteIndex::VersionKey> Interface::GetVersionKeysById(const SQLite::Connection& connection, SQLite::rowid_t id) const { auto versionsAndChannels = ManifestTable::GetAllValuesById<IdTable, VersionTable, ChannelTable>(connection, id); @@ -583,35 +616,12 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 return std::make_unique<SearchResultsTable>(connection); } - std::vector<MatchType> Interface::GetMatchTypeOrder(MatchType type) const - { - switch (type) - { - case MatchType::Exact: - return { MatchType::Exact }; - case MatchType::CaseInsensitive: - return { MatchType::CaseInsensitive }; - case MatchType::StartsWith: - return { MatchType::CaseInsensitive, MatchType::StartsWith }; - case MatchType::Substring: - return { MatchType::CaseInsensitive, MatchType::Substring }; - case MatchType::Wildcard: - return { MatchType::Wildcard }; - case MatchType::Fuzzy: - return { MatchType::CaseInsensitive, MatchType::Fuzzy }; - case MatchType::FuzzySubstring: - return { MatchType::CaseInsensitive, MatchType::Fuzzy, MatchType::Substring, MatchType::FuzzySubstring }; - default: - THROW_HR(E_UNEXPECTED); - } - } - void Interface::PerformQuerySearch(SearchResultsTable& resultsTable, const RequestMatch& query) const { // Arbitrary values to create a reusable filter with the given value. PackageMatchFilter filter(PackageMatchField::Id, MatchType::Exact, query.Value); - for (MatchType match : GetMatchTypeOrder(query.Type)) + for (MatchType match : GetDefaultMatchTypeOrder(query.Type)) { filter.Type = match; @@ -637,6 +647,8 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 return ManifestTable::GetValueById<ChannelTable>(connection, manifestId); case AppInstaller::Repository::PackageVersionProperty::RelativePath: return PathPartTable::GetPathById(connection, std::get<0>(ManifestTable::GetIdsById<PathPartTable>(connection, manifestId))); + case AppInstaller::Repository::PackageVersionProperty::Moniker: + return ManifestTable::GetValueById<MonikerTable>(connection, manifestId); default: return {}; } diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/ManifestTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/ManifestTable.cpp @@ -433,6 +433,14 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 savepoint.Commit(); } + void ManifestTable::Drop(SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(s_ManifestTable_Table_Name); + + dropTableBuilder.Execute(connection); + } + SQLite::rowid_t ManifestTable::Insert(SQLite::Connection& connection, std::initializer_list<ManifestOneToOneValue> values) { SQLite::Builder::StatementBuilder builder; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/ManifestTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/ManifestTable.h @@ -121,6 +121,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Creates the table with standard primary keys. static void Create_deprecated(SQLite::Connection& connection, std::initializer_list<ManifestColumnInfo> values); + // Drops the table. + static void Drop(SQLite::Connection& connection); + // Insert the given values into the table. static SQLite::rowid_t Insert(SQLite::Connection& connection, std::initializer_list<ManifestOneToOneValue> values); diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToManyTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToManyTable.cpp @@ -171,6 +171,20 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 savepoint.Commit(); } + void DropOneToManyTable(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_drop_v1_0"); + + DropOneToOneTable(connection, tableName); + + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable({ tableName, s_OneToManyTable_MapTable_Suffix }); + + dropTableBuilder.Execute(connection); + + savepoint.Commit(); + } + std::vector<std::string> OneToManyTableGetValuesByManifestId( const SQLite::Connection& connection, std::string_view tableName, diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToManyTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToManyTable.h @@ -34,6 +34,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Create the tables. void CreateOneToManyTable(SQLite::Connection& connection, OneToManyTableSchema schemaVersion, std::string_view tableName, std::string_view valueName); + // Drop the tables. + void DropOneToManyTable(SQLite::Connection& connection, std::string_view tableName); + // Gets all values associated with the given manifest id. std::vector<std::string> OneToManyTableGetValuesByManifestId( const SQLite::Connection& connection, @@ -99,6 +102,12 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 details::CreateOneToManyTable(connection, schemaVersion, TableInfo::TableName(), TableInfo::ValueName()); } + // Drops the table. + static void Drop(SQLite::Connection& connection) + { + details::DropOneToManyTable(connection, TableInfo::TableName()); + } + // Gets all values associated with the given manifest id. static std::vector<std::string> GetValuesByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId) { diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToOneTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToOneTable.cpp @@ -50,6 +50,14 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 } } + void DropOneToOneTable(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(tableName); + + dropTableBuilder.Execute(connection); + } + std::optional<SQLite::rowid_t> OneToOneTableSelectIdByValue(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, std::string_view value, bool useLike) { SQLite::Builder::StatementBuilder selectBuilder; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToOneTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/OneToOneTable.h @@ -15,6 +15,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Creates the table. void CreateOneToOneTable(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, bool useNamedIndices); + // Drops the table. + void DropOneToOneTable(SQLite::Connection& connection, std::string_view tableName); + // Selects the value from the table, returning the rowid if it exists. std::optional<SQLite::rowid_t> OneToOneTableSelectIdByValue(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, std::string_view value, bool useLike = false); @@ -26,6 +29,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Ensures that the values exists in the table. SQLite::rowid_t OneToOneTableEnsureExists(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, std::string_view value, bool overwriteLikeMatch = false); + // Removes data that is no longer needed for an index that is to be published. void OneToOneTablePrepareForPackaging(SQLite::Connection& connection, std::string_view tableName, bool useNamedIndices, bool preserveValuesIndex); @@ -64,6 +68,12 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 details::CreateOneToOneTable(connection, TableInfo::TableName(), TableInfo::ValueName(), false); } + // Drops the table. + static void Drop(SQLite::Connection& connection) + { + details::DropOneToOneTable(connection, TableInfo::TableName()); + } + // The name of the table. static constexpr std::string_view TableName() { diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/PathPartTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/PathPartTable.cpp @@ -159,6 +159,14 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 savepoint.Commit(); } + void PathPartTable::Drop(SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(s_PathPartTable_Table_Name); + + dropTableBuilder.Execute(connection); + } + std::string_view PathPartTable::TableName() { return s_PathPartTable_Table_Name; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/PathPartTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_0/PathPartTable.h @@ -26,6 +26,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_0 // Creates the table with standard primary keys. static void Create_deprecated(SQLite::Connection& connection); + // Drops the table. + static void Drop(SQLite::Connection& connection); + // Gets the table name. static std::string_view TableName(); diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/Interface.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/Interface.h @@ -25,6 +25,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_1 MetadataResult GetMetadataByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId) const override; void SetMetadataByManifestId(SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionMetadata metadata, std::string_view value) override; + // Version 1.7 + void DropTables(SQLite::Connection& connection) override; + protected: std::unique_ptr<V1_0::SearchResultsTable> CreateSearchResultsTable(const SQLite::Connection& connection) const override; void PerformQuerySearch(V1_0::SearchResultsTable& resultsTable, const RequestMatch& query) const override; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/Interface_1_1.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/Interface_1_1.cpp @@ -177,6 +177,20 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_1 savepoint.Commit(); } + void Interface::DropTables(SQLite::Connection& connection) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "drop_tables_v1_1"); + + V1_0::Interface::DropTables(connection); + + PackageFamilyNameTable::Drop(connection); + ProductCodeTable::Drop(connection); + + ManifestMetadataTable::Drop(connection); + + savepoint.Commit(); + } + std::unique_ptr<V1_0::SearchResultsTable> Interface::CreateSearchResultsTable(const SQLite::Connection& connection) const { return std::make_unique<V1_1::SearchResultsTable>(connection); diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/ManifestMetadataTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/ManifestMetadataTable.cpp @@ -49,6 +49,14 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_1 savepoint.Commit(); } + void ManifestMetadataTable::Drop(SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTableIfExists(s_ManifestMetadataTable_Table_Name); + + dropTableBuilder.Execute(connection); + } + ISQLiteIndex::MetadataResult ManifestMetadataTable::GetMetadataByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId) { using namespace Builder; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/ManifestMetadataTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_1/ManifestMetadataTable.h @@ -22,6 +22,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_1 // Creates the table in the database. static void Create(SQLite::Connection& connection); + // Drops the table. + static void Drop(SQLite::Connection& connection); + // Gets all metadata associated with the given manifest. // The table must exist. static ISQLiteIndex::MetadataResult GetMetadataByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId); diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_2/Interface.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_2/Interface.h @@ -24,6 +24,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 // Version 1.2 Utility::NormalizedName NormalizeName(std::string_view name, std::string_view publisher) const override; + // Version 1.7 + void DropTables(SQLite::Connection& connection) override; + protected: std::unique_ptr<V1_0::SearchResultsTable> CreateSearchResultsTable(const SQLite::Connection& connection) const override; SearchResult SearchInternal(const SQLite::Connection& connection, SearchRequest& request) const override; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_2/Interface_1_2.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_2/Interface_1_2.cpp @@ -11,7 +11,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 { - namespace + namespace anon { void AddNormalizedName( const Utility::NameNormalizer& normalizer, @@ -114,7 +114,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 filter.Value = normalized.GetNormalizedName(fieldsToInclude); filter.Additional = normalized.Publisher(); return WI_AreAllFlagsSet(normalized.GetNormalizedFields(), fieldsToInclude); - }; + } // Update NormalizedNameAndPublisher with normalization and folding // Returns true if any of normalized name contains normalization field of fieldsToInclude @@ -180,8 +180,8 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 // Add the new 1.2 data // These normalized strings are all stored with their cases folded so that they can be // looked up ordinally; enabling the index to provide efficient searches. - NormalizedPackageNameTable::EnsureExistsAndInsert(connection, GetNormalizedNames(m_normalizer, manifest), manifestId); - NormalizedPackagePublisherTable::EnsureExistsAndInsert(connection, GetNormalizedPublishers(m_normalizer, manifest), manifestId); + NormalizedPackageNameTable::EnsureExistsAndInsert(connection, anon::GetNormalizedNames(m_normalizer, manifest), manifestId); + NormalizedPackagePublisherTable::EnsureExistsAndInsert(connection, anon::GetNormalizedPublishers(m_normalizer, manifest), manifestId); savepoint.Commit(); @@ -195,8 +195,8 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 auto [indexModified, manifestId] = V1_1::Interface::UpdateManifest(connection, manifest, relativePath); // Update new 1.2 tables as necessary - indexModified = NormalizedPackageNameTable::UpdateIfNeededByManifestId(connection, GetNormalizedNames(m_normalizer, manifest), manifestId) || indexModified; - indexModified = NormalizedPackagePublisherTable::UpdateIfNeededByManifestId(connection, GetNormalizedPublishers(m_normalizer, manifest), manifestId) || indexModified; + indexModified = NormalizedPackageNameTable::UpdateIfNeededByManifestId(connection, anon::GetNormalizedNames(m_normalizer, manifest), manifestId) || indexModified; + indexModified = NormalizedPackagePublisherTable::UpdateIfNeededByManifestId(connection, anon::GetNormalizedPublishers(m_normalizer, manifest), manifestId) || indexModified; savepoint.Commit(); @@ -253,6 +253,18 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 return m_normalizer.Normalize(name, publisher); } + void Interface::DropTables(SQLite::Connection& connection) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "drop_tables_v1_2"); + + V1_1::Interface::DropTables(connection); + + NormalizedPackageNameTable::Drop(connection); + NormalizedPackagePublisherTable::Drop(connection); + + savepoint.Commit(); + } + std::unique_ptr<V1_0::SearchResultsTable> Interface::CreateSearchResultsTable(const SQLite::Connection& connection) const { return std::make_unique<V1_2::SearchResultsTable>(connection); @@ -266,7 +278,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 // For available package to installed package mapping, only one try is needed. // For example, if ARP DisplayName contains arch, then the installed package's ARP DisplayName should also include arch. auto candidateInclusionsWithArch = request.Inclusions; - if (UpdatePackageMatchFilters(candidateInclusionsWithArch, m_normalizer, Utility::NormalizationField::Architecture)) + if (anon::UpdatePackageMatchFilters(candidateInclusionsWithArch, m_normalizer, Utility::NormalizationField::Architecture)) { // If DisplayNames contain arch, only use Inclusions with arch for search request.Inclusions = candidateInclusionsWithArch; @@ -274,7 +286,7 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 else { // Otherwise, just update the Inclusions with normalization - UpdatePackageMatchFilters(request.Inclusions, m_normalizer); + anon::UpdatePackageMatchFilters(request.Inclusions, m_normalizer); } return V1_1::Interface::SearchInternal(connection, request); @@ -286,11 +298,11 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 // This can be extended in the future for more granular search requests. std::vector<SearchRequest> candidateSearches; auto candidateSearchWithArch = request; - if (UpdatePackageMatchFilters(candidateSearchWithArch.Inclusions, m_normalizer, Utility::NormalizationField::Architecture)) + if (anon::UpdatePackageMatchFilters(candidateSearchWithArch.Inclusions, m_normalizer, Utility::NormalizationField::Architecture)) { candidateSearches.emplace_back(std::move(candidateSearchWithArch)); } - UpdatePackageMatchFilters(request.Inclusions, m_normalizer); + anon::UpdatePackageMatchFilters(request.Inclusions, m_normalizer); candidateSearches.emplace_back(request); SearchResult result; @@ -307,8 +319,8 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_2 } else { - UpdatePackageMatchFilters(request.Inclusions, m_normalizer); - UpdatePackageMatchFilters(request.Filters, m_normalizer); + anon::UpdatePackageMatchFilters(request.Inclusions, m_normalizer); + anon::UpdatePackageMatchFilters(request.Filters, m_normalizer); return V1_1::Interface::SearchInternal(connection, request); } diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/DependenciesTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/DependenciesTable.cpp @@ -236,6 +236,14 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_4 savepoint.Commit(); } + void DependenciesTable::Drop(SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTableIfExists(s_DependenciesTable_Table_Name); + + dropTableBuilder.Execute(connection); + } + void DependenciesTable::AddDependencies(SQLite::Connection& connection, const Manifest::Manifest& manifest, SQLite::rowid_t manifestRowId) { if (!Exists(connection)) @@ -566,4 +574,4 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_4 return result; } -}- \ No newline at end of file +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/DependenciesTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/DependenciesTable.h @@ -21,7 +21,10 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_4 static std::string_view TableName(); // Creates the table with named indices. - static void Create(SQLite::Connection& connection); + static void Create(SQLite::Connection& connection); + + // Drops the table. + static void Drop(SQLite::Connection& connection); static bool Exists(const SQLite::Connection& connection); @@ -55,4 +58,4 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_4 // Get all dependencies with min versions in the dependencies table, used during consistency check. Returning a list of <PackageRowId, VersionString> pair. static std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetAllDependenciesWithMinVersions(const SQLite::Connection& connection); }; -}- \ No newline at end of file +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/Interface.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/Interface.h @@ -23,6 +23,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_4 std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t manifestRowId) const override; std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const override; + // Version 1.7 + void DropTables(SQLite::Connection& connection) override; + protected: bool NotNeeded(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t id) const override; @@ -30,4 +33,4 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_4 // Semantic check to validate dependencies with min versions are satisfied. bool ValidateDependenciesWithMinVersions(const SQLite::Connection& connection, bool log) const; }; -}- \ No newline at end of file +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/Interface_1_4.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_4/Interface_1_4.cpp @@ -1,177 +1,188 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. -#include "pch.h" -#include "Microsoft/Schema/1_4/Interface.h" -#include "Microsoft/Schema/1_0/VersionTable.h" - -#include "Microsoft/Schema/1_4/DependenciesTable.h" - -namespace AppInstaller::Repository::Microsoft::Schema::V1_4 -{ - Interface::Interface(Utility::NormalizationVersion normVersion) : V1_3::Interface(normVersion) - { - } - - SQLite::Version Interface::GetVersion() const - { - return { 1, 4 }; - } - - void Interface::CreateTables(SQLite::Connection& connection, CreateOptions options) - { - SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "createtables_v1_4"); - - V1_3::Interface::CreateTables(connection, options); - - if (WI_IsFlagClear(options, CreateOptions::DisableDependenciesSupport)) - { - DependenciesTable::Create(connection); - } - - savepoint.Commit(); - } - - SQLite::rowid_t Interface::AddManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) - { - SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "addmanifest_v1_4"); - - SQLite::rowid_t manifestId = V1_3::Interface::AddManifest(connection, manifest, relativePath); - - DependenciesTable::AddDependencies(connection, manifest, manifestId); - - savepoint.Commit(); - - return manifestId; - } - - std::pair<bool, SQLite::rowid_t> Interface::UpdateManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) - { - SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "updatemanifest_v1_4"); - - auto [indexModified, manifestId] = V1_3::Interface::UpdateManifest(connection, manifest, relativePath); - - bool dependenciesModified = DependenciesTable::UpdateDependencies(connection, manifest, manifestId); - indexModified = indexModified || dependenciesModified; - - savepoint.Commit(); - - return { indexModified, manifestId }; - } - - void Interface::RemoveManifestById(SQLite::Connection& connection, SQLite::rowid_t manifestId) - { - // Get all versions that need cleaning from the version table. - auto minVersions = DependenciesTable::GetDependenciesMinVersionsRowIdByManifestId(connection, manifestId); - - SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "removemanifest_v1_4"); - - // Removes dependences for the manifest id. - DependenciesTable::RemoveDependencies(connection, manifestId); - - // Removes the manifest. - V1_3::Interface::RemoveManifestById(connection, manifestId); - - // Remove the versions that are not needed. - for (auto minVersion : minVersions) - { - if (NotNeeded(connection, Schema::V1_0::VersionTable::TableName(), Schema::V1_0::VersionTable::ValueName(), minVersion)) - { - Schema::V1_0::VersionTable::DeleteById(connection, minVersion); - } - } - - savepoint.Commit(); - } - - bool Interface::NotNeeded(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t id) const - { - bool result = V1_0::Interface::NotNeeded(connection, tableName, valueName, id); - - return !DependenciesTable::IsValueReferenced(connection, tableName, id) && result; - } - - void Interface::PrepareForPackaging(SQLite::Connection& connection, bool vacuum) - { - SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "prepareforpackaging_v1_4"); - - V1_3::Interface::PrepareForPackaging(connection, false); - - DependenciesTable::PrepareForPackaging(connection); - - savepoint.Commit(); - - if (vacuum) - { - Vacuum(connection); - } - } - - bool Interface::CheckConsistency(const SQLite::Connection& connection, bool log) const - { - bool result = V1_3::Interface::CheckConsistency(connection, log); - - // If the v1.3 index was consistent, or if full logging of inconsistency was requested, check the v1.4 data. - if (result || log) - { - result = DependenciesTable::CheckConsistency(connection, log) && result; - } - - if (result || log) - { - result = ValidateDependenciesWithMinVersions(connection, log) && result; - } - - return result; - } - - std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> Interface::GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t manifestRowId) const - { - return DependenciesTable::GetDependenciesByManifestRowId(connection, manifestRowId); - } - - std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> Interface::GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const - { - return DependenciesTable::GetDependentsById(connection, packageId); - } - - bool Interface::ValidateDependenciesWithMinVersions(const SQLite::Connection& connection, bool log) const - { - try - { - bool result = true; - // A map to store already checked dependency package latest versions. - std::map<SQLite::rowid_t, Utility::Version> checkedVersions; - - auto dependencies = DependenciesTable::GetAllDependenciesWithMinVersions(connection); - for (auto const& dependency : dependencies) - { - // If the dependency package has not been checked yet, add to the map. - if (checkedVersions.find(dependency.first) == checkedVersions.end()) - { - auto versionKeys = GetVersionKeysById(connection, dependency.first); - THROW_HR_IF(E_UNEXPECTED, versionKeys.empty()); - checkedVersions.emplace(dependency.first, versionKeys[0].VersionAndChannel.GetVersion()); - } - - // If the latest version is less than min version required, fail the validation. - if (checkedVersions[dependency.first] < Utility::Version{ dependency.second }) - { - AICLI_LOG(Repo, Error, << "Dependency with min version not satisfied. Dependency package row id: " << dependency.first << " min version: " << dependency.second); - result = false; - - if (!log) - { - break; - } - } - } - - return result; - } - catch (...) - { - AICLI_LOG(Repo, Error, << "ValidateDependenciesWithMinVersions() encountered internal error. Returning false."); - return false; - } - } -}- \ No newline at end of file +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "Microsoft/Schema/1_4/Interface.h" +#include "Microsoft/Schema/1_0/VersionTable.h" + +#include "Microsoft/Schema/1_4/DependenciesTable.h" + +namespace AppInstaller::Repository::Microsoft::Schema::V1_4 +{ + Interface::Interface(Utility::NormalizationVersion normVersion) : V1_3::Interface(normVersion) + { + } + + SQLite::Version Interface::GetVersion() const + { + return { 1, 4 }; + } + + void Interface::CreateTables(SQLite::Connection& connection, CreateOptions options) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "createtables_v1_4"); + + V1_3::Interface::CreateTables(connection, options); + + if (WI_IsFlagClear(options, CreateOptions::DisableDependenciesSupport)) + { + DependenciesTable::Create(connection); + } + + savepoint.Commit(); + } + + SQLite::rowid_t Interface::AddManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "addmanifest_v1_4"); + + SQLite::rowid_t manifestId = V1_3::Interface::AddManifest(connection, manifest, relativePath); + + DependenciesTable::AddDependencies(connection, manifest, manifestId); + + savepoint.Commit(); + + return manifestId; + } + + std::pair<bool, SQLite::rowid_t> Interface::UpdateManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "updatemanifest_v1_4"); + + auto [indexModified, manifestId] = V1_3::Interface::UpdateManifest(connection, manifest, relativePath); + + bool dependenciesModified = DependenciesTable::UpdateDependencies(connection, manifest, manifestId); + indexModified = indexModified || dependenciesModified; + + savepoint.Commit(); + + return { indexModified, manifestId }; + } + + void Interface::RemoveManifestById(SQLite::Connection& connection, SQLite::rowid_t manifestId) + { + // Get all versions that need cleaning from the version table. + auto minVersions = DependenciesTable::GetDependenciesMinVersionsRowIdByManifestId(connection, manifestId); + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "removemanifest_v1_4"); + + // Removes dependences for the manifest id. + DependenciesTable::RemoveDependencies(connection, manifestId); + + // Removes the manifest. + V1_3::Interface::RemoveManifestById(connection, manifestId); + + // Remove the versions that are not needed. + for (auto minVersion : minVersions) + { + if (NotNeeded(connection, Schema::V1_0::VersionTable::TableName(), Schema::V1_0::VersionTable::ValueName(), minVersion)) + { + Schema::V1_0::VersionTable::DeleteById(connection, minVersion); + } + } + + savepoint.Commit(); + } + + bool Interface::NotNeeded(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t id) const + { + bool result = V1_0::Interface::NotNeeded(connection, tableName, valueName, id); + + return !DependenciesTable::IsValueReferenced(connection, tableName, id) && result; + } + + void Interface::PrepareForPackaging(SQLite::Connection& connection, bool vacuum) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "prepareforpackaging_v1_4"); + + V1_3::Interface::PrepareForPackaging(connection, false); + + DependenciesTable::PrepareForPackaging(connection); + + savepoint.Commit(); + + if (vacuum) + { + Vacuum(connection); + } + } + + bool Interface::CheckConsistency(const SQLite::Connection& connection, bool log) const + { + bool result = V1_3::Interface::CheckConsistency(connection, log); + + // If the v1.3 index was consistent, or if full logging of inconsistency was requested, check the v1.4 data. + if (result || log) + { + result = DependenciesTable::CheckConsistency(connection, log) && result; + } + + if (result || log) + { + result = ValidateDependenciesWithMinVersions(connection, log) && result; + } + + return result; + } + + std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> Interface::GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t manifestRowId) const + { + return DependenciesTable::GetDependenciesByManifestRowId(connection, manifestRowId); + } + + std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> Interface::GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const + { + return DependenciesTable::GetDependentsById(connection, packageId); + } + + void Interface::DropTables(SQLite::Connection& connection) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "drop_tables_v1_4"); + + V1_2::Interface::DropTables(connection); + + DependenciesTable::Drop(connection); + + savepoint.Commit(); + } + + bool Interface::ValidateDependenciesWithMinVersions(const SQLite::Connection& connection, bool log) const + { + try + { + bool result = true; + // A map to store already checked dependency package latest versions. + std::map<SQLite::rowid_t, Utility::Version> checkedVersions; + + auto dependencies = DependenciesTable::GetAllDependenciesWithMinVersions(connection); + for (auto const& dependency : dependencies) + { + // If the dependency package has not been checked yet, add to the map. + if (checkedVersions.find(dependency.first) == checkedVersions.end()) + { + auto versionKeys = GetVersionKeysById(connection, dependency.first); + THROW_HR_IF(E_UNEXPECTED, versionKeys.empty()); + checkedVersions.emplace(dependency.first, versionKeys[0].VersionAndChannel.GetVersion()); + } + + // If the latest version is less than min version required, fail the validation. + if (checkedVersions[dependency.first] < Utility::Version{ dependency.second }) + { + AICLI_LOG(Repo, Error, << "Dependency with min version not satisfied. Dependency package row id: " << dependency.first << " min version: " << dependency.second); + result = false; + + if (!log) + { + break; + } + } + } + + return result; + } + catch (...) + { + AICLI_LOG(Repo, Error, << "ValidateDependenciesWithMinVersions() encountered internal error. Returning false."); + return false; + } + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_6/Interface.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_6/Interface.h @@ -20,6 +20,9 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_6 bool CheckConsistency(const SQLite::Connection& connection, bool log) const override; std::vector<std::string> GetMultiPropertyByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionMultiProperty property) const override; + // Version 1.7 + void DropTables(SQLite::Connection& connection) override; + protected: std::unique_ptr<V1_0::SearchResultsTable> CreateSearchResultsTable(const SQLite::Connection& connection) const override; void PerformQuerySearch(V1_0::SearchResultsTable& resultsTable, const RequestMatch& query) const override; diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/1_6/Interface_1_6.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/1_6/Interface_1_6.cpp @@ -96,6 +96,17 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_6 } } + void Interface::DropTables(SQLite::Connection& connection) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "drop_tables_v1_6"); + + V1_4::Interface::DropTables(connection); + + UpgradeCodeTable::Drop(connection); + + savepoint.Commit(); + } + std::unique_ptr<V1_0::SearchResultsTable> Interface::CreateSearchResultsTable(const SQLite::Connection& connection) const { return std::make_unique<V1_6::SearchResultsTable>(connection); @@ -151,4 +162,4 @@ namespace AppInstaller::Repository::Microsoft::Schema::V1_6 Vacuum(connection); } } -}- \ No newline at end of file +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/CommandsTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/CommandsTable.h @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/OneToManyTableWithMap.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct CommandsTableInfo + { + inline static constexpr std::string_view TableName() { return "commands2"sv; } + inline static constexpr std::string_view ValueName() { return "command"sv; } + }; + } + + using CommandsTable = OneToManyTableWithMap<details::CommandsTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/Interface.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/Interface.h @@ -0,0 +1,98 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/ISQLiteIndex.h" +#include "Microsoft/Schema/2_0/SearchResultsTable.h" +#include "Microsoft/Schema/2_0/OneToManyTableWithMap.h" + +#include <memory> +#include <vector> + +using namespace std::string_view_literals; + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + // Version 2.0 + static constexpr std::string_view s_MetadataValueName_PackageUpdateTrackingBaseTime = "updateTrackingBase"sv; + + // Interface to this schema version exposed through ISQLiteIndex. + struct Interface : public ISQLiteIndex + { + Interface(Utility::NormalizationVersion normVersion = Utility::NormalizationVersion::Initial); + + // Version 1.0 + SQLite::Version GetVersion() const override; + void CreateTables(SQLite::Connection& connection, CreateOptions options) override; + SQLite::rowid_t AddManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) override; + std::pair<bool, SQLite::rowid_t> UpdateManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) override; + SQLite::rowid_t RemoveManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest) override; + void RemoveManifestById(SQLite::Connection& connection, SQLite::rowid_t manifestId) override; + void PrepareForPackaging(SQLite::Connection& connection) override; + void PrepareForPackaging(const SQLiteIndexContext& context) override; + bool CheckConsistency(const SQLite::Connection& connection, bool log) const override; + SearchResult Search(const SQLite::Connection& connection, const SearchRequest& request) const override; + std::optional<std::string> GetPropertyByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionProperty property) const override; + std::vector<std::string> GetMultiPropertyByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionMultiProperty property) const override; + std::optional<SQLite::rowid_t> GetManifestIdByKey(const SQLite::Connection& connection, SQLite::rowid_t id, std::string_view version, std::string_view channel) const override; + std::optional<SQLite::rowid_t> GetManifestIdByManifest(const SQLite::Connection& connection, const Manifest::Manifest& manifest) const override; + std::vector<VersionKey> GetVersionKeysById(const SQLite::Connection& connection, SQLite::rowid_t id) const override; + + // Version 1.1 + MetadataResult GetMetadataByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId) const override; + void SetMetadataByManifestId(SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionMetadata metadata, std::string_view value) override; + + // Version 1.2 + Utility::NormalizedName NormalizeName(std::string_view name, std::string_view publisher) const override; + + // Version 1.4 Get all the dependencies for a specific manifest. + std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t manifestRowId) const override; + std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const override; + + // Version 1.7 + void DropTables(SQLite::Connection& connection) override; + + // Version 2.0 + bool MigrateFrom(SQLite::Connection& connection, const ISQLiteIndex* current) override; + void SetProperty(SQLite::Connection& connection, Property property, const std::string& value) override; + + protected: + // Creates the search results table. + virtual std::unique_ptr<SearchResultsTable> CreateSearchResultsTable(const SQLite::Connection& connection) const; + + // Executes all relevant searches for the query. + virtual void PerformQuerySearch(SearchResultsTable& resultsTable, const RequestMatch& query) const; + + // Gets the one to many table schema to use. + virtual OneToManyTableSchema GetOneToManyTableSchema() const; + + // Executes search on a request that can be modified. + virtual SearchResult SearchInternal(const SQLite::Connection& connection, SearchRequest& request) const; + + // Executes search on the given request. + SearchResult BasicSearchInternal(const SQLite::Connection& connection, const SearchRequest& request) const; + + // Prepares for packaging, optionally vacuuming the database. + virtual void PrepareForPackaging(const SQLiteIndexContext& context, bool vacuum); + + // Force the database to shrink the file size. + // This *must* be done outside of an active transaction. + void Vacuum(const SQLite::Connection& connection); + + // If before PrepareForPackaging is called, this should find the internal interface schema version and create the object. + // If after PrepareForPackaging is called, this should not find the internal interface schema version and allow the code to fall through. + // requireInternalInterface should be set to true for modifying functions. + void EnsureInternalInterface(const SQLite::Connection& connection, bool requireInternalInterface = false) const; + + // Allows derived types to move to a different internal schema version. + virtual std::unique_ptr<Schema::ISQLiteIndex> CreateInternalInterface() const; + + // If EnsureInternalInterface has been called. + mutable bool m_internalInterfaceChecked = false; + + // Interface to the data before PrepareForPackaging is called. + mutable std::unique_ptr<Schema::ISQLiteIndex> m_internalInterface; + + // The name normalization utility + Utility::NameNormalizer m_normalizer; + }; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/Interface_2_0.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/Interface_2_0.cpp @@ -0,0 +1,788 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include <winget/SQLiteMetadataTable.h> +#include "Microsoft/Schema/2_0/Interface.h" + +#include "Microsoft/Schema/2_0/PackagesTable.h" + +#include "Microsoft/Schema/2_0/TagsTable.h" +#include "Microsoft/Schema/2_0/CommandsTable.h" +#include "Microsoft/Schema/2_0/PackageFamilyNameTable.h" +#include "Microsoft/Schema/2_0/ProductCodeTable.h" +#include "Microsoft/Schema/2_0/NormalizedPackageNameTable.h" +#include "Microsoft/Schema/2_0/NormalizedPackagePublisherTable.h" +#include "Microsoft/Schema/2_0/UpgradeCodeTable.h" + +#include "Microsoft/Schema/2_0/SearchResultsTable.h" +#include "Microsoft/Schema/2_0/PackageUpdateTrackingTable.h" + +#include <winget/PackageVersionDataManifest.h> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace anon + { + // Folds the values of the fields that are stored folded. + void FoldPackageMatchFilters(std::vector<PackageMatchFilter>& filters) + { + for (auto& filter : filters) + { + if ((filter.Field == PackageMatchField::PackageFamilyName || filter.Field == PackageMatchField::ProductCode || filter.Field == PackageMatchField::UpgradeCode) && + filter.Type == MatchType::Exact) + { + filter.Value = Utility::FoldCase(filter.Value); + } + } + } + + // Update NormalizedNameAndPublisher with normalization and folding + // Returns true if the normalized name contains normalization field of fieldsToInclude + bool UpdateNormalizedNameAndPublisher( + PackageMatchFilter& filter, + const Utility::NameNormalizer& normalizer, + Utility::NormalizationField fieldsToInclude) + { + Utility::NormalizedName normalized = normalizer.Normalize(Utility::FoldCase(filter.Value), Utility::FoldCase(filter.Additional.value())); + filter.Value = normalized.GetNormalizedName(fieldsToInclude); + filter.Additional = normalized.Publisher(); + return WI_AreAllFlagsSet(normalized.GetNormalizedFields(), fieldsToInclude); + } + + bool UpdatePackageMatchFilters( + std::vector<PackageMatchFilter>& filters, + const Utility::NameNormalizer& normalizer, + Utility::NormalizationField normalizedNameFieldsToFilter = Utility::NormalizationField::None) + { + bool normalizedNameFieldsFound = false; + for (auto itr = filters.begin(); itr != filters.end();) + { + if (itr->Field == PackageMatchField::NormalizedNameAndPublisher && itr->Type == MatchType::Exact) + { + if (!UpdateNormalizedNameAndPublisher(*itr, normalizer, normalizedNameFieldsToFilter)) + { + // If not matched, this package match filter will be removed. + // For example, if caller is trying to search with arch info only, values without arch will be removed from search. + itr = filters.erase(itr); + continue; + } + + normalizedNameFieldsFound = true; + } + + ++itr; + } + + return normalizedNameFieldsFound; + } + } + + Interface::Interface(Utility::NormalizationVersion normVersion) : m_normalizer(normVersion) + { + } + + SQLite::Version Interface::GetVersion() const + { + return { 2, 0 }; + } + + void Interface::CreateTables(SQLite::Connection& connection, CreateOptions options) + { + m_internalInterface = CreateInternalInterface(); + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "createtables_v2_0"); + + // We only create the internal tables at this point, the actual 2.0 tables are created in PrepareForPackaging + m_internalInterface->CreateTables(connection, options); + + savepoint.Commit(); + + m_internalInterfaceChecked = true; + } + + SQLite::rowid_t Interface::AddManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) + { + EnsureInternalInterface(connection, true); + SQLite::rowid_t manifestId = m_internalInterface->AddManifest(connection, manifest, relativePath); + PackageUpdateTrackingTable::Update(connection, m_internalInterface.get(), m_internalInterface->GetPropertyByManifestId(connection, manifestId, PackageVersionProperty::Id).value()); + return manifestId; + } + + std::pair<bool, SQLite::rowid_t> Interface::UpdateManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest, const std::optional<std::filesystem::path>& relativePath) + { + EnsureInternalInterface(connection, true); + std::pair<bool, SQLite::rowid_t> result = m_internalInterface->UpdateManifest(connection, manifest, relativePath); + if (result.first) + { + PackageUpdateTrackingTable::Update(connection, m_internalInterface.get(), m_internalInterface->GetPropertyByManifestId(connection, result.second, PackageVersionProperty::Id).value()); + } + return result; + } + + SQLite::rowid_t Interface::RemoveManifest(SQLite::Connection& connection, const Manifest::Manifest& manifest) + { + EnsureInternalInterface(connection, true); + std::optional<SQLite::rowid_t> result = m_internalInterface->GetManifestIdByManifest(connection, manifest); + + // If the manifest doesn't actually exist, fail the remove. + THROW_HR_IF(E_NOT_SET, !result); + + SQLite::rowid_t manifestId = result.value(); + RemoveManifestById(connection, manifestId); + + return manifestId; + } + + void Interface::RemoveManifestById(SQLite::Connection& connection, SQLite::rowid_t manifestId) + { + EnsureInternalInterface(connection, true); + std::optional<std::string> identifier = m_internalInterface->GetPropertyByManifestId(connection, manifestId, PackageVersionProperty::Id); + m_internalInterface->RemoveManifestById(connection, manifestId); + if (identifier) + { + PackageUpdateTrackingTable::Update(connection, m_internalInterface.get(), identifier.value()); + } + } + + void Interface::PrepareForPackaging(SQLite::Connection&) + { + // We implement the context version + THROW_HR(E_NOTIMPL); + } + + void Interface::PrepareForPackaging(const SQLiteIndexContext& context) + { + EnsureInternalInterface(context.Connection, true); + PrepareForPackaging(context, true); + } + + bool Interface::CheckConsistency(const SQLite::Connection& connection, bool log) const + { + EnsureInternalInterface(connection); + + bool result = true; + +#define AICLI_CHECK_CONSISTENCY(_check_) \ + if (result || log) \ + { \ + result = _check_ && result; \ + } + + if (m_internalInterface) + { + AICLI_CHECK_CONSISTENCY(m_internalInterface->CheckConsistency(connection, log)); + AICLI_CHECK_CONSISTENCY(PackageUpdateTrackingTable::CheckConsistency(connection, m_internalInterface.get(), log)); + + return result; + } + + AICLI_CHECK_CONSISTENCY((PackagesTable::CheckConsistency< + PackagesTable::IdColumn, + PackagesTable::NameColumn, + PackagesTable::MonikerColumn, + PackagesTable::LatestVersionColumn, + PackagesTable::ARPMinVersionColumn, + PackagesTable::ARPMaxVersionColumn>(connection, log))); + + // Check the 1:N map tables for consistency + AICLI_CHECK_CONSISTENCY(TagsTable::CheckConsistency(connection, log)); + AICLI_CHECK_CONSISTENCY(CommandsTable::CheckConsistency(connection, log)); + + AICLI_CHECK_CONSISTENCY(PackageFamilyNameTable::CheckConsistency(connection, log)); + AICLI_CHECK_CONSISTENCY(ProductCodeTable::CheckConsistency(connection, log)); + AICLI_CHECK_CONSISTENCY(NormalizedPackageNameTable::CheckConsistency(connection, log)); + AICLI_CHECK_CONSISTENCY(NormalizedPackagePublisherTable::CheckConsistency(connection, log)); + AICLI_CHECK_CONSISTENCY(UpgradeCodeTable::CheckConsistency(connection, log)); + +#undef AICLI_CHECK_CONSISTENCY + + return result; + } + + ISQLiteIndex::SearchResult Interface::Search(const SQLite::Connection& connection, const SearchRequest& request) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->Search(connection, request); + } + + SearchRequest requestCopy = request; + return SearchInternal(connection, requestCopy); + } + + std::optional<std::string> Interface::GetPropertyByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionProperty property) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetPropertyByManifestId(connection, manifestId, property); + } + + switch (property) + { + case PackageVersionProperty::Id: + return PackagesTable::GetValueById<PackagesTable::IdColumn>(connection, manifestId); + case PackageVersionProperty::Name: + return PackagesTable::GetValueById<PackagesTable::NameColumn>(connection, manifestId); + case PackageVersionProperty::Version: + return PackagesTable::GetValueById<PackagesTable::LatestVersionColumn>(connection, manifestId); + case PackageVersionProperty::Channel: + return ""; + case PackageVersionProperty::ManifestSHA256Hash: + { + std::optional<SQLite::blob_t> hash = PackagesTable::GetValueById<PackagesTable::HashColumn>(connection, manifestId); + return (!hash || hash->empty()) ? std::optional<std::string>{} : Utility::SHA256::ConvertToString(hash.value()); + } + case PackageVersionProperty::ArpMinVersion: + return PackagesTable::GetValueById<PackagesTable::ARPMinVersionColumn>(connection, manifestId); + case PackageVersionProperty::ArpMaxVersion: + return PackagesTable::GetValueById<PackagesTable::ARPMaxVersionColumn>(connection, manifestId); + case PackageVersionProperty::Moniker: + return PackagesTable::GetValueById<PackagesTable::MonikerColumn>(connection, manifestId); + default: + return {}; + } + } + + std::vector<std::string> Interface::GetMultiPropertyByManifestId(const SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionMultiProperty property) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetMultiPropertyByManifestId(connection, manifestId, property); + } + + switch (property) + { + case PackageVersionMultiProperty::PackageFamilyName: + return PackageFamilyNameTable::GetValuesByPrimaryId(connection, manifestId); + case PackageVersionMultiProperty::ProductCode: + return ProductCodeTable::GetValuesByPrimaryId(connection, manifestId); + // These values are not right, as they are normalized. But they are good enough for now and all we have. + case PackageVersionMultiProperty::Name: + return NormalizedPackageNameTable::GetValuesByPrimaryId(connection, manifestId); + case PackageVersionMultiProperty::Publisher: + return NormalizedPackagePublisherTable::GetValuesByPrimaryId(connection, manifestId); + case PackageVersionMultiProperty::UpgradeCode: + return UpgradeCodeTable::GetValuesByPrimaryId(connection, manifestId); + case PackageVersionMultiProperty::Tag: + return TagsTable::GetValuesByPrimaryId(connection, manifestId); + case PackageVersionMultiProperty::Command: + return CommandsTable::GetValuesByPrimaryId(connection, manifestId); + default: + return {}; + } + } + + std::optional<SQLite::rowid_t> Interface::GetManifestIdByKey(const SQLite::Connection& connection, SQLite::rowid_t id, std::string_view version, std::string_view channel) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetManifestIdByKey(connection, id, version, channel); + } + + THROW_HR(E_NOT_VALID_STATE); + } + + std::optional<SQLite::rowid_t> Interface::GetManifestIdByManifest(const SQLite::Connection& connection, const Manifest::Manifest& manifest) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetManifestIdByManifest(connection, manifest); + } + + THROW_HR(E_NOT_VALID_STATE); + } + + std::vector<ISQLiteIndex::VersionKey> Interface::GetVersionKeysById(const SQLite::Connection& connection, SQLite::rowid_t id) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetVersionKeysById(connection, id); + } + + THROW_HR(E_NOT_VALID_STATE); + } + + ISQLiteIndex::MetadataResult Interface::GetMetadataByManifestId(const SQLite::Connection&, SQLite::rowid_t) const + { + return {}; + } + + void Interface::SetMetadataByManifestId(SQLite::Connection&, SQLite::rowid_t, PackageVersionMetadata, std::string_view) + { + } + + Utility::NormalizedName Interface::NormalizeName(std::string_view name, std::string_view publisher) const + { + if (m_internalInterface) + { + return m_internalInterface->NormalizeName(name, publisher); + } + + return m_normalizer.Normalize(name, publisher); + } + + std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> Interface::GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t rowid) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetDependenciesByManifestRowId(connection, rowid); + } + + THROW_HR(E_NOT_VALID_STATE); + } + + std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> Interface::GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t id) const + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->GetDependentsById(connection, id); + } + + THROW_HR(E_NOT_VALID_STATE); + } + + void Interface::DropTables(SQLite::Connection& connection) + { + EnsureInternalInterface(connection); + + if (m_internalInterface) + { + return m_internalInterface->DropTables(connection); + } + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "drop_tables_v2_0"); + + PackagesTable::Drop(connection); + + TagsTable::Drop(connection); + CommandsTable::Drop(connection); + + PackageFamilyNameTable::Drop(connection); + ProductCodeTable::Drop(connection); + NormalizedPackageNameTable::Drop(connection); + NormalizedPackagePublisherTable::Drop(connection); + UpgradeCodeTable::Drop(connection); + + savepoint.Commit(); + } + + bool Interface::MigrateFrom(SQLite::Connection& connection, const ISQLiteIndex* current) + { + THROW_HR_IF_NULL(E_POINTER, current); + + auto currentVersion = current->GetVersion(); + if (currentVersion.MajorVersion != 1 || currentVersion.MinorVersion != 7) + { + return false; + } + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "migrate_from_v2_0"); + + // We only need to insert all of the existing packages into the update tracking table. + PackageUpdateTrackingTable::EnsureExists(connection); + SearchResult allPackages = current->Search(connection, {}); + + for (const auto& packageMatch : allPackages.Matches) + { + std::vector<ISQLiteIndex::VersionKey> versionKeys = current->GetVersionKeysById(connection, packageMatch.first); + ISQLiteIndex::VersionKey& latestVersionKey = versionKeys[0]; + PackageUpdateTrackingTable::Update(connection, current, current->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::Id).value(), false); + } + + savepoint.Commit(); + return true; + } + + void Interface::SetProperty(SQLite::Connection& connection, Property property, const std::string& value) + { + switch (property) + { + case Property::PackageUpdateTrackingBaseTime: + { + int64_t baseTime = 0; + if (value.empty()) + { + baseTime = Utility::GetCurrentUnixEpoch(); + } + else + { + baseTime = std::stoll(value); + } + SQLite::MetadataTable::SetNamedValue(connection, s_MetadataValueName_PackageUpdateTrackingBaseTime, std::to_string(baseTime)); + } + break; + + default: + THROW_WIN32(ERROR_NOT_SUPPORTED); + } + } + + std::unique_ptr<SearchResultsTable> Interface::CreateSearchResultsTable(const SQLite::Connection& connection) const + { + return std::make_unique<SearchResultsTable>(connection); + } + + void Interface::PerformQuerySearch(SearchResultsTable& resultsTable, const RequestMatch& query) const + { + // First, do an exact match search for the folded system reference strings + // We do this first because it is exact, and likely won't match anything else if it matches this. + PackageMatchFilter filter(PackageMatchField::Unknown, MatchType::Exact, Utility::FoldCase(query.Value)); + + for (PackageMatchField field : { PackageMatchField::PackageFamilyName, PackageMatchField::ProductCode, PackageMatchField::UpgradeCode }) + { + filter.Field = field; + resultsTable.SearchOnField(filter); + } + + // Now search on the unfolded value + filter.Value = query.Value; + + for (MatchType match : GetDefaultMatchTypeOrder(query.Type)) + { + filter.Type = match; + + for (auto field : { PackageMatchField::Id, PackageMatchField::Name, PackageMatchField::Moniker, PackageMatchField::Command, PackageMatchField::Tag }) + { + filter.Field = field; + resultsTable.SearchOnField(filter); + } + } + } + + OneToManyTableSchema Interface::GetOneToManyTableSchema() const + { + return OneToManyTableSchema::Version_2_0; + } + + ISQLiteIndex::SearchResult Interface::SearchInternal(const SQLite::Connection& connection, SearchRequest& request) const + { + anon::FoldPackageMatchFilters(request.Inclusions); + anon::FoldPackageMatchFilters(request.Filters); + + if (request.Purpose == SearchPurpose::CorrelationToInstalled) + { + // Correlate from available package to installed package + // For available package to installed package mapping, only one try is needed. + // For example, if ARP DisplayName contains arch, then the installed package's ARP DisplayName should also include arch. + auto candidateInclusionsWithArch = request.Inclusions; + if (anon::UpdatePackageMatchFilters(candidateInclusionsWithArch, m_normalizer, Utility::NormalizationField::Architecture)) + { + // If DisplayNames contain arch, only use Inclusions with arch for search + request.Inclusions = candidateInclusionsWithArch; + } + else + { + // Otherwise, just update the Inclusions with normalization + anon::UpdatePackageMatchFilters(request.Inclusions, m_normalizer); + } + + return BasicSearchInternal(connection, request); + } + else if (request.Purpose == SearchPurpose::CorrelationToAvailable) + { + // For installed package to available package correlation, + // try the search with NormalizedName with Arch first, if not found, try with all values. + // This can be extended in the future for more granular search requests. + std::vector<SearchRequest> candidateSearches; + auto candidateSearchWithArch = request; + if (anon::UpdatePackageMatchFilters(candidateSearchWithArch.Inclusions, m_normalizer, Utility::NormalizationField::Architecture)) + { + candidateSearches.emplace_back(std::move(candidateSearchWithArch)); + } + anon::UpdatePackageMatchFilters(request.Inclusions, m_normalizer); + candidateSearches.emplace_back(request); + + SearchResult result; + for (auto& candidateSearch : candidateSearches) + { + result = BasicSearchInternal(connection, candidateSearch); + if (!result.Matches.empty()) + { + break; + } + } + + return result; + } + else + { + anon::UpdatePackageMatchFilters(request.Inclusions, m_normalizer); + anon::UpdatePackageMatchFilters(request.Filters, m_normalizer); + + return BasicSearchInternal(connection, request); + } + } + + ISQLiteIndex::SearchResult Interface::BasicSearchInternal(const SQLite::Connection& connection, const SearchRequest& request) const + { + if (request.IsForEverything()) + { + std::vector<SQLite::rowid_t> ids = PackagesTable::GetAllRowIds(connection, PackagesTable::IdColumn::Name, request.MaximumResults); + + SearchResult result; + for (SQLite::rowid_t id : ids) + { + result.Matches.emplace_back(std::make_pair(id, PackageMatchFilter(PackageMatchField::Id, MatchType::Wildcard))); + } + + result.Truncated = (request.MaximumResults && PackagesTable::GetCount(connection) > request.MaximumResults); + + return result; + } + + // First phase, create the search results table and populate it with the initial results. + // If the Query is provided, we search across many fields and put results in together. + // If Inclusions has fields, we add these to the data. + // If neither is defined, we take the first filter and use it as the initial results search. + std::unique_ptr<SearchResultsTable> resultsTable = CreateSearchResultsTable(connection); + bool inclusionsAttempted = false; + + if (request.Query) + { + // Perform searches across multiple tables to populate the initial results. + PerformQuerySearch(*resultsTable.get(), request.Query.value()); + + inclusionsAttempted = true; + } + + if (!request.Inclusions.empty()) + { + for (auto include : request.Inclusions) + { + for (MatchType match : GetDefaultMatchTypeOrder(include.Type)) + { + include.Type = match; + resultsTable->SearchOnField(include); + } + } + + inclusionsAttempted = true; + } + + size_t filterIndex = 0; + if (!inclusionsAttempted) + { + THROW_HR_IF(E_UNEXPECTED, request.Filters.empty()); + + // Perform search for just the field matching the first filter + PackageMatchFilter filter = request.Filters[0]; + + for (MatchType match : GetDefaultMatchTypeOrder(filter.Type)) + { + filter.Type = match; + resultsTable->SearchOnField(filter); + } + + // Skip the filter as we already know everything matches + filterIndex = 1; + } + + // Remove any duplicate manifest entries + resultsTable->RemoveDuplicateManifestRows(); + + // Second phase, for remaining filters, flag matching search results, then remove unflagged values. + for (size_t i = filterIndex; i < request.Filters.size(); ++i) + { + PackageMatchFilter filter = request.Filters[i]; + + resultsTable->PrepareToFilter(); + + for (MatchType match : GetDefaultMatchTypeOrder(filter.Type)) + { + filter.Type = match; + resultsTable->FilterOnField(filter); + } + + resultsTable->CompleteFilter(); + } + + return resultsTable->GetSearchResults(request.MaximumResults); + } + + void Interface::PrepareForPackaging(const SQLiteIndexContext& context, bool vacuum) + { + SQLite::Connection& connection = context.Connection; + + // Get the base time from metadata + int64_t updateBaseTime = 0; + std::optional<std::string> updateBaseTimeString = SQLite::MetadataTable::TryGetNamedValue<std::string>(connection, s_MetadataValueName_PackageUpdateTrackingBaseTime); + if (updateBaseTimeString && !updateBaseTimeString->empty()) + { + updateBaseTime = std::stoll(updateBaseTimeString.value()); + } + + // Get the output directory or use the file path + std::filesystem::path baseOutputDirectory; + + if (context.Data.Contains(Property::IntermediateFileOutputPath)) + { + baseOutputDirectory = context.Data.Get<Property::IntermediateFileOutputPath>(); + } + else if (context.Data.Contains(Property::DatabaseFilePath)) + { + baseOutputDirectory = context.Data.Get<Property::DatabaseFilePath>(); + baseOutputDirectory = baseOutputDirectory.parent_path(); + } + + THROW_WIN32_IF(ERROR_INVALID_STATE, baseOutputDirectory.empty() || baseOutputDirectory.is_relative()); + + // Output all of the changed package version manifests since the base time to the target location + for (const auto& packageData : PackageUpdateTrackingTable::GetUpdatesSince(connection, updateBaseTime)) + { + std::filesystem::path packageDirectory = baseOutputDirectory / Utility::ConvertToUTF16(packageData.PackageIdentifier); + std::filesystem::path hashDirectory = packageDirectory / Utility::SHA256::ConvertToWideString(packageData.Hash); + + std::filesystem::create_directories(hashDirectory); + + std::filesystem::path manifestPath = hashDirectory / Manifest::PackageVersionDataManifest::VersionManifestCompressedFileName(); + + AICLI_LOG(Repo, Info, << "Writing PackageVersionDataManifest for [" << packageData.PackageIdentifier << "] to [" << manifestPath << "]"); + + std::ofstream stream(manifestPath, std::ios_base::out | std::ios_base::binary | std::ios_base::trunc); + THROW_LAST_ERROR_IF(stream.fail()); + stream.write(reinterpret_cast<const char*>(packageData.Manifest.data()), packageData.Manifest.size()); + THROW_LAST_ERROR_IF(stream.fail()); + stream.flush(); + } + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "prepareforpackaging_v2_0"); + + // Create the 2.0 data tables + PackagesTable::Create< + PackagesTable::IdColumn, + PackagesTable::NameColumn, + PackagesTable::MonikerColumn, + PackagesTable::LatestVersionColumn, + PackagesTable::ARPMinVersionColumn, + PackagesTable::ARPMaxVersionColumn, + PackagesTable::HashColumn + >(connection); + + TagsTable::Create(connection, GetOneToManyTableSchema()); + CommandsTable::Create(connection, GetOneToManyTableSchema()); + + PackageFamilyNameTable::Create(connection); + ProductCodeTable::Create(connection); + NormalizedPackageNameTable::Create(connection); + NormalizedPackagePublisherTable::Create(connection); + UpgradeCodeTable::Create(connection); + + // Copy data from 1.7 tables to 2.0 tables + SearchResult allPackages = m_internalInterface->Search(connection, {}); + + for (const auto& packageMatch : allPackages.Matches) + { + std::vector<ISQLiteIndex::VersionKey> versionKeys = m_internalInterface->GetVersionKeysById(connection, packageMatch.first); + ISQLiteIndex::VersionKey& latestVersionKey = versionKeys[0]; + + std::string packageIdentifier = m_internalInterface->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::Id).value(); + + std::vector<PackagesTable::NameValuePair> packageData{ + { PackagesTable::IdColumn::Name, packageIdentifier }, + { PackagesTable::NameColumn::Name, m_internalInterface->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::Name).value() }, + { PackagesTable::LatestVersionColumn::Name, latestVersionKey.VersionAndChannel.GetVersion().ToString() }, + }; + + auto addIfPresent = [&](std::string_view name, std::optional<std::string>&& value) + { + if (value && !value->empty()) + { + packageData.emplace_back(PackagesTable::NameValuePair{ name, std::move(value).value() }); + } + }; + + addIfPresent(PackagesTable::MonikerColumn::Name, m_internalInterface->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::Moniker).value()); + addIfPresent(PackagesTable::ARPMinVersionColumn::Name, m_internalInterface->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::ArpMinVersion).value()); + addIfPresent(PackagesTable::ARPMaxVersionColumn::Name, m_internalInterface->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::ArpMaxVersion).value()); + + SQLite::rowid_t packageId = PackagesTable::Insert(connection, packageData); + + PackagesTable::UpdateValueIdById<PackagesTable::HashColumn>(connection, packageId, PackageUpdateTrackingTable::GetDataHash(connection, packageIdentifier)); + + for (const auto& versionKey : versionKeys) + { + TagsTable::EnsureExistsAndInsert(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::Tag), packageId); + CommandsTable::EnsureExistsAndInsert(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::Command), packageId); + + PackageFamilyNameTable::EnsureExists(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::PackageFamilyName), packageId); + ProductCodeTable::EnsureExists(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::ProductCode), packageId); + NormalizedPackageNameTable::EnsureExists(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::Name), packageId); + NormalizedPackagePublisherTable::EnsureExists(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::Publisher), packageId); + UpgradeCodeTable::EnsureExists(connection, m_internalInterface->GetMultiPropertyByManifestId(connection, versionKey.ManifestId, PackageVersionMultiProperty::UpgradeCode), packageId); + } + } + + PackagesTable::PrepareForPackaging< + PackagesTable::IdColumn, + PackagesTable::NameColumn, + PackagesTable::MonikerColumn, + PackagesTable::LatestVersionColumn, + PackagesTable::ARPMinVersionColumn, + PackagesTable::ARPMaxVersionColumn, + PackagesTable::HashColumn + >(connection); + + TagsTable::PrepareForPackaging(connection); + CommandsTable::PrepareForPackaging(connection); + + PackageUpdateTrackingTable::Drop(connection); + + // The tables based on SystemReferenceStringTable don't need a prepare currently + + // Drop 1.7 tables + m_internalInterface->DropTables(connection); + + savepoint.Commit(); + + m_internalInterface.reset(); + + if (vacuum) + { + Vacuum(connection); + } + } + + void Interface::Vacuum(const SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder builder; + builder.Vacuum(); + builder.Execute(connection); + } + + void Interface::EnsureInternalInterface(const SQLite::Connection& connection, bool requireInternalInterface) const + { + if (!m_internalInterfaceChecked) + { + if (!PackagesTable::Exists(connection)) + { + m_internalInterface = CreateInternalInterface(); + } + + m_internalInterfaceChecked = true; + } + + THROW_HR_IF(E_NOT_VALID_STATE, requireInternalInterface && !m_internalInterface); + } + + std::unique_ptr<Schema::ISQLiteIndex> Interface::CreateInternalInterface() const + { + return CreateISQLiteIndex({ 1, 7 }); + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/NormalizedPackageNameTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/NormalizedPackageNameTable.h @@ -0,0 +1,22 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/SystemReferenceStringTable.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct NormalizedPackageNameTableInfo + { + inline static constexpr std::string_view TableName() { return "norm_names2"sv; } + inline static constexpr std::string_view ValueName() { return "norm_name"sv; } + }; + } + + // The table for Commands. + using NormalizedPackageNameTable = SystemReferenceStringTable<details::NormalizedPackageNameTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/NormalizedPackagePublisherTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/NormalizedPackagePublisherTable.h @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/SystemReferenceStringTable.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct NormalizedPackagePublisherTableInfo + { + inline static constexpr std::string_view TableName() { return "norm_publishers2"sv; } + inline static constexpr std::string_view ValueName() { return "norm_publisher"sv; } + }; + } + + using NormalizedPackagePublisherTable = SystemReferenceStringTable<details::NormalizedPackagePublisherTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/OneToManyTableWithMap.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/OneToManyTableWithMap.cpp @@ -0,0 +1,411 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "Microsoft/Schema/2_0/OneToManyTableWithMap.h" +#include "Microsoft/Schema/2_0/PackagesTable.h" +#include <winget/SQLiteStatementBuilder.h> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using PrimaryTable = PackagesTable; + + using namespace std::string_view_literals; + static constexpr std::string_view s_OneToManyTableWithMap_MapTable_PrimaryName = "package"sv; + static constexpr std::string_view s_OneToManyTableWithMap_MapTable_Suffix = "_map"sv; + static constexpr std::string_view s_OneToManyTableWithMap_MapTable_IndexSuffix = "_index"sv; + static constexpr std::string_view s_OneToManyTableWithMap_PrimaryKeyIndexSuffix = "_pkindex"sv; + + namespace anon + { + // Create the mapping table insert statement for multiple use. + // Bind the rowid of the value to 2. + SQLite::Statement CreateMappingInsertStatementForPrimaryId(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t manifestId) + { + SQLite::Builder::StatementBuilder insertMappingBuilder; + insertMappingBuilder.InsertOrIgnore({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }). + Columns({ s_OneToManyTableWithMap_MapTable_PrimaryName, valueName }).Values(manifestId, SQLite::Builder::Unbound); + + return insertMappingBuilder.Prepare(connection); + } + + // Get a collection of the value ids associated with the given primary id. + std::vector<SQLite::rowid_t> GetValueIdsByPrimaryId(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t manifestId) + { + std::vector<SQLite::rowid_t> result; + + SQLite::Builder::StatementBuilder selectMappingBuilder; + selectMappingBuilder.Select(valueName).From({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }).Where(s_OneToManyTableWithMap_MapTable_PrimaryName).Equals(manifestId); + + SQLite::Statement selectMappingStatement = selectMappingBuilder.Prepare(connection); + + while (selectMappingStatement.Step()) + { + result.push_back(selectMappingStatement.GetColumn<SQLite::rowid_t>(0)); + } + + return result; + } + + void CreateDataTable(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName) + { + using namespace SQLite::Builder; + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_create_v2_0"); + + StatementBuilder createTableBuilder; + + createTableBuilder.CreateTable(tableName).Columns({ + IntegerPrimaryKey(), + ColumnBuilder(valueName, Type::Text).NotNull() + }); + + createTableBuilder.Execute(connection); + + StatementBuilder indexBuilder; + indexBuilder.CreateUniqueIndex({ tableName, s_OneToManyTableWithMap_PrimaryKeyIndexSuffix }).On(tableName).Columns(valueName); + indexBuilder.Execute(connection); + + savepoint.Commit(); + } + + void DropDataTable(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(tableName); + + dropTableBuilder.Execute(connection); + } + + std::optional<SQLite::rowid_t> DataTableSelectIdByValue(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, std::string_view value, bool useLike) + { + SQLite::Builder::StatementBuilder selectBuilder; + selectBuilder.Select(SQLite::RowIDName).From(tableName).Where(valueName); + + if (useLike) + { + selectBuilder.LikeWithEscape(value); + } + else + { + selectBuilder.Equals(value); + } + + SQLite::Statement select = selectBuilder.Prepare(connection); + + if (select.Step()) + { + return select.GetColumn<SQLite::rowid_t>(0); + } + else + { + return {}; + } + } + + std::optional<std::string> DataTableSelectValueById(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, SQLite::rowid_t rowid) + { + SQLite::Builder::StatementBuilder selectBuilder; + selectBuilder.Select(valueName).From(tableName).Where(SQLite::RowIDName).Equals(rowid); + + SQLite::Statement select = selectBuilder.Prepare(connection); + + if (select.Step()) + { + return select.GetColumn<std::string>(0); + } + else + { + return {}; + } + } + + SQLite::rowid_t DataTableEnsureExists(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, std::string_view value, bool overwriteLikeMatch = false) + { + auto selectResult = DataTableSelectIdByValue(connection, tableName, valueName, value, overwriteLikeMatch); + if (selectResult) + { + if (overwriteLikeMatch) + { + // If the value in the table is not an exact match, overwrite it with the incoming value + auto tableValue = DataTableSelectValueById(connection, tableName, valueName, selectResult.value()); + if (tableValue.value() != value) + { + SQLite::Builder::StatementBuilder updateBuilder; + updateBuilder.Update(tableName).Set().Column(valueName).Equals(value).Where(SQLite::RowIDName).Equals(selectResult); + + updateBuilder.Execute(connection); + } + } + + return selectResult.value(); + } + + SQLite::Builder::StatementBuilder insertBuilder; + insertBuilder.InsertInto(tableName).Columns(valueName).Values(value); + + insertBuilder.Execute(connection); + + return connection.GetLastInsertRowID(); + } + + void DataTablePrepareForPackaging(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder dropIndexBuilder; + dropIndexBuilder.DropIndex({ tableName, s_OneToManyTableWithMap_PrimaryKeyIndexSuffix }); + dropIndexBuilder.Execute(connection); + } + + bool DataTableCheckConsistency(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, bool log) + { + // Build a select statement to find values that contain an embedded null character + // Such as: + // Select count(*) from table where instr(value,char(0))>0 + SQLite::Builder::StatementBuilder builder; + builder. + Select({ SQLite::RowIDName, valueName }). + From(tableName). + WhereValueContainsEmbeddedNullCharacter(valueName); + + SQLite::Statement select = builder.Prepare(connection); + bool result = true; + + while (select.Step()) + { + result = false; + + if (!log) + { + break; + } + + AICLI_LOG(Repo, Info, << " [INVALID] value in table [" << tableName << "] at row [" << select.GetColumn<SQLite::rowid_t>(0) << "] contains an embedded null character and starts with [" << select.GetColumn<std::string>(1) << "]"); + } + + return result; + } + } + + std::string OneToManyTableWithMapGetMapTableName(std::string_view tableName) + { + std::string result(tableName); + result += s_OneToManyTableWithMap_MapTable_Suffix; + return result; + } + + std::string_view OneToManyTableWithMapGetManifestColumnName() + { + return s_OneToManyTableWithMap_MapTable_PrimaryName; + } + + void CreateOneToManyTableWithMap(SQLite::Connection& connection, OneToManyTableSchema schemaVersion, std::string_view tableName, std::string_view valueName) + { + using namespace SQLite::Builder; + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_create_v2_0"); + + // Create the data table as a 1:1 + anon::CreateDataTable(connection, tableName, valueName); + + switch (schemaVersion) + { + case OneToManyTableSchema::Version_2_0: + { + // Create the mapping table + StatementBuilder createMapTableBuilder; + createMapTableBuilder.CreateTable({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }).Columns({ + ColumnBuilder(valueName, Type::Int64).NotNull(), + ColumnBuilder(s_OneToManyTableWithMap_MapTable_PrimaryName, Type::Int64).NotNull(), + PrimaryKeyBuilder({ valueName, s_OneToManyTableWithMap_MapTable_PrimaryName }) + }).WithoutRowID(); + + createMapTableBuilder.Execute(connection); + } + break; + default: + THROW_HR(E_UNEXPECTED); + } + + StatementBuilder createMapTableIndexBuilder; + createMapTableIndexBuilder.CreateIndex({ tableName, s_OneToManyTableWithMap_MapTable_Suffix, s_OneToManyTableWithMap_MapTable_IndexSuffix }). + On({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }).Columns({ s_OneToManyTableWithMap_MapTable_PrimaryName, valueName }); + + createMapTableIndexBuilder.Execute(connection); + + savepoint.Commit(); + } + + void DropOneToManyTableWithMap(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_drop_v2_0"); + + anon::DropDataTable(connection, tableName); + + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }); + + dropTableBuilder.Execute(connection); + + savepoint.Commit(); + } + + std::vector<std::string> OneToManyTableWithMapGetValuesByPrimaryId( + const SQLite::Connection& connection, + std::string_view tableName, + std::string_view valueName, + SQLite::rowid_t manifestId) + { + using QCol = SQLite::Builder::QualifiedColumn; + + std::vector<std::string> result; + + SQLite::Builder::StatementBuilder builder; + builder.Select(QCol(tableName, valueName)). + From({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }).As("map").Join(tableName). + On(QCol("map", valueName), QCol(tableName, SQLite::RowIDName)).Where(QCol("map", s_OneToManyTableWithMap_MapTable_PrimaryName)).Equals(manifestId); + + SQLite::Statement statement = builder.Prepare(connection); + + while (statement.Step()) + { + result.emplace_back(statement.GetColumn<std::string>(0)); + } + + return result; + } + + void OneToManyTableWithMapEnsureExistsAndInsert(SQLite::Connection& connection, + std::string_view tableName, std::string_view valueName, + const std::vector<std::string>& values, SQLite::rowid_t manifestId) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_ensureandinsert_v2_0"); + + SQLite::Statement insertMapping = anon::CreateMappingInsertStatementForPrimaryId(connection, tableName, valueName, manifestId); + + for (const std::string& value : values) + { + // First, ensure that the data exists + SQLite::rowid_t dataId = anon::DataTableEnsureExists(connection, tableName, valueName, value); + + // Second, insert into the mapping table + insertMapping.Reset(); + insertMapping.Bind(2, dataId); + + insertMapping.Execute(); + } + + savepoint.Commit(); + } + + void OneToManyTableWithMapPrepareForPackaging(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder dropMapTableIndexBuilder; + dropMapTableIndexBuilder.DropIndex({ tableName, s_OneToManyTableWithMap_MapTable_Suffix, s_OneToManyTableWithMap_MapTable_IndexSuffix }); + + dropMapTableIndexBuilder.Execute(connection); + + anon::DataTablePrepareForPackaging(connection, tableName); + } + + bool OneToManyTableWithMapCheckConsistency(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, bool log) + { + using QCol = SQLite::Builder::QualifiedColumn; + constexpr std::string_view s_map = "map"sv; + + bool result = true; + + { + // Build a select statement to find map rows containing references to primaries with nonexistent rowids + // Such as: + // Select map.rowid, map.primary from tags_map as map left outer join primary on map.primary = primary.rowid where primary.id is null + + SQLite::Builder::StatementBuilder builder; + builder. + Select({ QCol(s_map, s_OneToManyTableWithMap_MapTable_PrimaryName), QCol(s_map, valueName) }). + From({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }).As(s_map). + LeftOuterJoin(details::PrimaryTable::TableName()).On(QCol(s_map, s_OneToManyTableWithMap_MapTable_PrimaryName), QCol(details::PrimaryTable::TableName(), SQLite::RowIDName)). + Where(QCol(details::PrimaryTable::TableName(), SQLite::RowIDName)).IsNull(); + + SQLite::Statement select = builder.Prepare(connection); + + while (select.Step()) + { + result = false; + + if (!log) + { + break; + } + + AICLI_LOG(Repo, Info, << " [INVALID] " << tableName << s_OneToManyTableWithMap_MapTable_Suffix << " [" << select.GetColumn<SQLite::rowid_t>(0) << + ", " << select.GetColumn<SQLite::rowid_t>(1) << "] refers to invalid " << details::PrimaryTable::TableName()); + } + } + + if (!result && !log) + { + return result; + } + + { + // Build a select statement to find map rows containing references to 1:1 tables with nonexistent rowids + // Such as: + // Select map.rowid, map.tag from tags_map as map left outer join tags on map.tag = tags.rowid where tags.tag is null + SQLite::Builder::StatementBuilder builder; + builder. + Select({ QCol(s_map, s_OneToManyTableWithMap_MapTable_PrimaryName), QCol(s_map, valueName) }). + From({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }).As(s_map). + LeftOuterJoin(tableName).On(QCol(s_map, valueName), QCol(tableName, SQLite::RowIDName)). + Where(QCol(tableName, valueName)).IsNull(); + + SQLite::Statement select = builder.Prepare(connection); + bool secondaryResult = true; + + while (select.Step()) + { + secondaryResult = false; + + if (!log) + { + break; + } + + AICLI_LOG(Repo, Info, << " [INVALID] " << tableName << s_OneToManyTableWithMap_MapTable_Suffix << " [" << select.GetColumn<SQLite::rowid_t>(0) << + ", " << select.GetColumn<SQLite::rowid_t>(1) << "] refers to invalid " << tableName); + } + + result = result && secondaryResult; + } + + if (!result && !log) + { + return result; + } + + result = anon::DataTableCheckConsistency(connection, tableName, valueName, log) && result; + + return result; + } + + bool OneToManyTableWithMapIsEmpty(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder countBuilder; + countBuilder.Select(SQLite::Builder::RowCount).From(tableName); + + SQLite::Statement countStatement = countBuilder.Prepare(connection); + + THROW_HR_IF(E_UNEXPECTED, !countStatement.Step()); + + SQLite::Builder::StatementBuilder countMapBuilder; + countMapBuilder.Select(SQLite::Builder::RowCount).From({ tableName, s_OneToManyTableWithMap_MapTable_Suffix }); + + SQLite::Statement countMapStatement = countMapBuilder.Prepare(connection); + + THROW_HR_IF(E_UNEXPECTED, !countMapStatement.Step()); + + return ((countStatement.GetColumn<int>(0) == 0) && (countMapStatement.GetColumn<int>(0) == 0)); + } + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/OneToManyTableWithMap.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/OneToManyTableWithMap.h @@ -0,0 +1,119 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <winget/SQLiteWrapper.h> +#include <string> +#include <string_view> +#include <vector> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + // Allow the different schema version to indicate which they are. + enum class OneToManyTableSchema + { + // Uses a named unique index for data table. + // Map table has primary key and no rowid. + // Column order is consistent with primary key order. + Version_2_0, + }; + + namespace details + { + // Returns the map table name for a given table. + std::string OneToManyTableGetMapTableName(std::string_view tableName); + + // Returns the primary column name. + std::string_view OneToManyTableGetManifestColumnName(); + + // Create the tables. + void CreateOneToManyTableWithMap(SQLite::Connection& connection, OneToManyTableSchema schemaVersion, std::string_view tableName, std::string_view valueName); + + // Drops the tables. + void DropOneToManyTableWithMap(SQLite::Connection& connection, std::string_view tableName); + + // Gets all values associated with the given primary id. + std::vector<std::string> OneToManyTableWithMapGetValuesByPrimaryId( + const SQLite::Connection& connection, + std::string_view tableName, + std::string_view valueName, + SQLite::rowid_t primaryId); + + // Ensures that the value exists and inserts mapping entries. + void OneToManyTableWithMapEnsureExistsAndInsert(SQLite::Connection& connection, + std::string_view tableName, std::string_view valueName, + const std::vector<std::string>& values, SQLite::rowid_t primaryId); + + // Removes data that is no longer needed for an index that is to be published. + void OneToManyTableWithMapPrepareForPackaging(SQLite::Connection& connection, std::string_view tableName); + + // Checks the consistency of the index to ensure that every referenced row exists. + // Returns true if index is consistent; false if it is not. + bool OneToManyTableWithMapCheckConsistency(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, bool log); + + // Determines if the table is empty. + bool OneToManyTableWithMapIsEmpty(SQLite::Connection& connection, std::string_view tableName); + } + + // A table that represents a value that is 1:N with a primary entry. + template <typename TableInfo> + struct OneToManyTableWithMap + { + // The name of the table. + static constexpr std::string_view TableName() + { + return TableInfo::TableName(); + } + + // The value name of the table. + static constexpr std::string_view ValueName() + { + return TableInfo::ValueName(); + } + + // Creates the table with named indices. + static void Create(SQLite::Connection& connection, OneToManyTableSchema schemaVersion) + { + details::CreateOneToManyTableWithMap(connection, schemaVersion, TableInfo::TableName(), TableInfo::ValueName()); + } + + // Drops the tables. + static void Drop(SQLite::Connection& connection) + { + details::DropOneToManyTableWithMap(connection, TableInfo::TableName()); + } + + // Gets all values associated with the given primary id. + static std::vector<std::string> GetValuesByPrimaryId(const SQLite::Connection& connection, SQLite::rowid_t primaryId) + { + return details::OneToManyTableWithMapGetValuesByPrimaryId(connection, TableInfo::TableName(), TableInfo::ValueName(), primaryId); + } + + // Ensures that all values exist in the data table, and inserts into the mapping table for the given primary id. + static void EnsureExistsAndInsert(SQLite::Connection& connection, const std::vector<std::string>& values, SQLite::rowid_t primaryId) + { + details::OneToManyTableWithMapEnsureExistsAndInsert(connection, TableInfo::TableName(), TableInfo::ValueName(), values, primaryId); + } + + // Removes data that is no longer needed for an index that is to be published. + // Preserving the primary index will improve the efficiency of finding the values associated with a primary. + // Preserving the values index will improve searching when it is primarily done by equality. + static void PrepareForPackaging(SQLite::Connection& connection) + { + details::OneToManyTableWithMapPrepareForPackaging(connection, TableInfo::TableName()); + } + + // Checks the consistency of the index to ensure that every referenced row exists. + // Returns true if index is consistent; false if it is not. + static bool CheckConsistency(const SQLite::Connection& connection, bool log) + { + return details::OneToManyTableWithMapCheckConsistency(connection, TableInfo::TableName(), TableInfo::ValueName(), log); + } + + // Determines if the table is empty. + static bool IsEmpty(SQLite::Connection& connection) + { + return details::OneToManyTableWithMapIsEmpty(connection, TableInfo::TableName()); + } + }; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackageFamilyNameTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackageFamilyNameTable.h @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/SystemReferenceStringTable.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct PackageFamilyNameTableInfo + { + inline static constexpr std::string_view TableName() { return "pfns2"sv; } + inline static constexpr std::string_view ValueName() { return "pfn"sv; } + }; + } + + using PackageFamilyNameTable = SystemReferenceStringTable<details::PackageFamilyNameTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackageUpdateTrackingTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackageUpdateTrackingTable.cpp @@ -0,0 +1,249 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "PackageUpdateTrackingTable.h" +#include <winget/PackageVersionDataManifest.h> +#include <winget/SQLiteStatementBuilder.h> + +using namespace AppInstaller::SQLite; + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + using namespace std::string_view_literals; + static constexpr std::string_view s_PUTT_Table_Name = "update_tracking"sv; + static constexpr std::string_view s_PUTT_WriteTimeIndex_Name = "update_tracking_write_idx"sv; + static constexpr std::string_view s_PUTT_Package = "package"sv; + static constexpr std::string_view s_PUTT_WriteTime = "write_time"sv; + static constexpr std::string_view s_PUTT_Manifest = "manifest"sv; + static constexpr std::string_view s_PUTT_Hash = "hash"sv; + + std::string_view PackageUpdateTrackingTable::TableName() + { + return s_PUTT_Table_Name; + } + + void PackageUpdateTrackingTable::Create(SQLite::Connection& connection) + { + using namespace Builder; + + StatementBuilder builder; + builder.CreateTable(s_PUTT_Table_Name).BeginColumns(); + + builder.Column(IntegerPrimaryKey()); + builder.Column(ColumnBuilder(s_PUTT_Package, Type::Text).NotNull()); + builder.Column(ColumnBuilder(s_PUTT_WriteTime, Type::Int64).NotNull()); + builder.Column(ColumnBuilder(s_PUTT_Manifest, Type::Blob).NotNull()); + builder.Column(ColumnBuilder(s_PUTT_Hash, Type::Blob).NotNull()); + + builder.EndColumns(); + + builder.Execute(connection); + + StatementBuilder indexBuilder; + indexBuilder.CreateIndex(s_PUTT_WriteTimeIndex_Name).On(s_PUTT_Table_Name).Columns(s_PUTT_WriteTime); + indexBuilder.Execute(connection); + } + + void PackageUpdateTrackingTable::EnsureExists(SQLite::Connection& connection) + { + if (!Exists(connection)) + { + Create(connection); + } + } + + void PackageUpdateTrackingTable::Drop(SQLite::Connection& connection) + { + Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(s_PUTT_Table_Name); + + dropTableBuilder.Execute(connection); + } + + bool PackageUpdateTrackingTable::Exists(const SQLite::Connection& connection) + { + Builder::StatementBuilder builder; + builder.Select(Builder::RowCount).From(Builder::Schema::MainTable). + Where(Builder::Schema::TypeColumn).Equals(Builder::Schema::Type_Table).And(Builder::Schema::NameColumn).Equals(s_PUTT_Table_Name); + + Statement statement = builder.Prepare(connection); + THROW_HR_IF(E_UNEXPECTED, !statement.Step()); + return statement.GetColumn<int64_t>(0) != 0; + } + + void PackageUpdateTrackingTable::Update(SQLite::Connection& connection, const ISQLiteIndex* internalIndex, const std::string& packageIdentifier, bool ensureTable) + { + if (ensureTable) + { + EnsureExists(connection); + } + + SearchRequest request; + request.Inclusions.emplace_back(PackageMatchField::Id, MatchType::CaseInsensitive, packageIdentifier); + auto result = internalIndex->Search(connection, request); + + if (result.Matches.empty()) + { + // Remove any existing package update row + Builder::StatementBuilder deleteBuilder; + deleteBuilder.DeleteFrom(s_PUTT_Table_Name).Where(s_PUTT_Package).LikeWithEscape(packageIdentifier); + + deleteBuilder.Execute(connection); + } + else + { + THROW_HR_IF(E_UNEXPECTED, result.Matches.size() != 1); + + // Insert or update the package row + std::vector<ISQLiteIndex::VersionKey> versionKeys = internalIndex->GetVersionKeysById(connection, result.Matches[0].first); + + Manifest::PackageVersionDataManifest manifest; + + for (const auto& key : versionKeys) + { + Manifest::PackageVersionDataManifest::VersionData versionData{ + key.VersionAndChannel, + internalIndex->GetPropertyByManifestId(connection, key.ManifestId, PackageVersionProperty::ArpMinVersion), + internalIndex->GetPropertyByManifestId(connection, key.ManifestId, PackageVersionProperty::ArpMaxVersion), + internalIndex->GetPropertyByManifestId(connection, key.ManifestId, PackageVersionProperty::RelativePath), + internalIndex->GetPropertyByManifestId(connection, key.ManifestId, PackageVersionProperty::ManifestSHA256Hash) + }; + + manifest.AddVersion(std::move(versionData)); + } + + std::string manifestString = manifest.Serialize(); + + auto compressor = Manifest::PackageVersionDataManifest::CreateCompressor(); + std::vector<uint8_t> compressedManifest = compressor.Compress(manifestString); + + Utility::SHA256::HashBuffer manifestHash = Utility::SHA256::ComputeHash(compressedManifest); + int64_t currentTime = Utility::GetCurrentUnixEpoch(); + + // First attempt to update the row and then insert it if no modification occurred. + Builder::StatementBuilder updateBuilder; + updateBuilder.Update(s_PUTT_Table_Name).Set(). + Column(s_PUTT_WriteTime).Equals(currentTime). + Column(s_PUTT_Manifest).Equals(compressedManifest). + Column(s_PUTT_Hash).Equals(manifestHash). + Where(s_PUTT_Package).LikeWithEscape(packageIdentifier); + + updateBuilder.Execute(connection); + + if (connection.GetChanges() == 0) + { + Builder::StatementBuilder insertBuilder; + insertBuilder.InsertInto(s_PUTT_Table_Name). + Columns({ s_PUTT_Package, s_PUTT_WriteTime, s_PUTT_Manifest, s_PUTT_Hash }). + Values(packageIdentifier, currentTime, compressedManifest, manifestHash); + + insertBuilder.Execute(connection); + } + } + } + + bool PackageUpdateTrackingTable::CheckConsistency(const SQLite::Connection& connection, ISQLiteIndex* internalIndex, bool log) + { + bool result = true; + + // Ensure that all data in the update table matches the internal index + for (const PackageData& packageData : GetUpdatesSince(connection, 0)) + { + auto manifestHash = Utility::SHA256::ComputeHash(packageData.Manifest); + if (!Utility::SHA256::AreEqual(packageData.Hash, manifestHash)) + { + if (!log) + { + return false; + } + + result = false; + AICLI_LOG(Repo, Info, << " [INVALID] value [" << s_PUTT_Hash << "] in table [" << s_PUTT_Table_Name << + "] at row [" << packageData.RowID << "]; the hash of the manifest value does not match the hash in the row"); + } + + SearchRequest request; + request.Inclusions.emplace_back(PackageMatchField::Id, MatchType::CaseInsensitive, packageData.PackageIdentifier); + + if (internalIndex->Search(connection, request).Matches.empty()) + { + if (!log) + { + return false; + } + + result = false; + AICLI_LOG(Repo, Info, << " [INVALID] value [" << s_PUTT_Package << "] in table [" << s_PUTT_Table_Name << + "] at row [" << packageData.RowID << "]; the package [" << packageData.PackageIdentifier << "] was not found in the internal index"); + } + } + + // Ensure that all packages in the internal index are present in the update table + Builder::StatementBuilder builder; + builder.Select(Builder::RowCount).From(s_PUTT_Table_Name).Where(s_PUTT_Package).Like(Builder::Unbound).Escape(EscapeCharForLike); + + Statement select = builder.Prepare(connection); + + for (const auto& packageMatch : internalIndex->Search(connection, {}).Matches) + { + std::vector<ISQLiteIndex::VersionKey> versionKeys = internalIndex->GetVersionKeysById(connection, packageMatch.first); + ISQLiteIndex::VersionKey& latestVersionKey = versionKeys[0]; + + std::string packageIdentifier = internalIndex->GetPropertyByManifestId(connection, latestVersionKey.ManifestId, PackageVersionProperty::Id).value(); + + select.Reset(); + select.Bind(1, packageIdentifier); + select.Step(); + + if (select.GetColumn<int64_t>(0) != 1) + { + if (!log) + { + return false; + } + + result = false; + AICLI_LOG(Repo, Info, << " [INVALID] value [" << packageIdentifier << "] in the internal index was not found in [" << s_PUTT_Table_Name << "]"); + } + } + + return result; + } + + std::vector<PackageUpdateTrackingTable::PackageData> PackageUpdateTrackingTable::GetUpdatesSince(const SQLite::Connection& connection, int64_t updateBaseTime) + { + Builder::StatementBuilder builder; + builder.Select({ RowIDName, s_PUTT_Package, s_PUTT_WriteTime, s_PUTT_Manifest, s_PUTT_Hash }). + From(s_PUTT_Table_Name).Where(s_PUTT_WriteTime).IsGreaterThanOrEqualTo(updateBaseTime); + + Statement select = builder.Prepare(connection); + + std::vector<PackageData> result; + + while (select.Step()) + { + PackageData item; + item.RowID = select.GetColumn<rowid_t>(0); + item.PackageIdentifier = select.GetColumn<std::string>(1); + item.WriteTime = select.GetColumn<int64_t>(2); + item.Manifest = select.GetColumn<blob_t>(3); + item.Hash = select.GetColumn<blob_t>(4); + + result.emplace_back(std::move(item)); + } + + return result; + } + + SQLite::blob_t PackageUpdateTrackingTable::GetDataHash(const SQLite::Connection& connection, const std::string& packageIdentifier) + { + Builder::StatementBuilder builder; + builder.Select(s_PUTT_Hash).From(s_PUTT_Table_Name).Where(s_PUTT_Package).LikeWithEscape(packageIdentifier); + + Statement select = builder.Prepare(connection); + + THROW_HR_IF(E_NOT_SET, !select.Step()); + + return select.GetColumn<SQLite::blob_t>(0); + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackageUpdateTrackingTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackageUpdateTrackingTable.h @@ -0,0 +1,52 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/ISQLiteIndex.h" +#include <winget/SQLiteWrapper.h> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + // Table for tracking the updates to the internal table so that prepare can output + // only the necessary package manifests. + struct PackageUpdateTrackingTable + { + // Get the table name. + static std::string_view TableName(); + + // Creates the table. + static void Create(SQLite::Connection& connection); + + // Creates the table if it does not exist. + static void EnsureExists(SQLite::Connection& connection); + + // Drops the table. + static void Drop(SQLite::Connection& connection); + + // Determine if the table currently exists in the database. + static bool Exists(const SQLite::Connection& connection); + + // Updates the tracking table for the given package identifier in the internal index. + static void Update(SQLite::Connection& connection, const ISQLiteIndex* internalIndex, const std::string& packageIdentifier, bool ensureTable = true); + + // Checks the consistency of the index to ensure that every referenced row exists. + // Returns true if index is consistent; false if it is not. + static bool CheckConsistency(const SQLite::Connection& connection, ISQLiteIndex* internalIndex, bool log); + + // Data on a single row in the table. + struct PackageData + { + SQLite::rowid_t RowID = 0; + std::string PackageIdentifier; + int64_t WriteTime = 0; + SQLite::blob_t Manifest; + SQLite::blob_t Hash; + }; + + // Gets the data on updates that have been written since the given base time. + static std::vector<PackageData> GetUpdatesSince(const SQLite::Connection& connection, int64_t updateBaseTime); + + // Gets the data hash for the given package identifier. + static SQLite::blob_t GetDataHash(const SQLite::Connection& connection, const std::string& packageIdentifier); + }; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackagesTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackagesTable.cpp @@ -0,0 +1,359 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "PackagesTable.h" +#include <winget/SQLiteStatementBuilder.h> +#include "OneToManyTableWithMap.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + using namespace std::string_view_literals; + static constexpr std::string_view s_PackagesTable_Table_Name = "packages"sv; + static constexpr std::string_view s_PackagesTable_Index_Separator = "_"sv; + static constexpr std::string_view s_PackagesTable_Index_Suffix = "_index"sv; + + namespace details + { + void PackagesTableCreate(SQLite::Connection& connection, std::initializer_list<ColumnInfo> values) + { + using namespace SQLite::Builder; + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "createPackagesTable_v2_0"); + + StatementBuilder createTableBuilder; + createTableBuilder.CreateTable(s_PackagesTable_Table_Name).BeginColumns(); + + // Add an integer primary key to keep the manifest rowid consistent + createTableBuilder.Column(IntegerPrimaryKey()); + + for (const ColumnInfo& value : values) + { + ColumnBuilder columnBuilder(value.Name, Type::Int64); + + if (!value.AllowNull) + { + columnBuilder.NotNull(); + } + + createTableBuilder.Column(columnBuilder); + } + + createTableBuilder.EndColumns(); + + createTableBuilder.Execute(connection); + + // Create a unique index with the primary key values + StatementBuilder pkIndexBuilder; + + pkIndexBuilder.CreateUniqueIndex({ s_PackagesTable_Table_Name, s_PackagesTable_Index_Suffix }).On(s_PackagesTable_Table_Name).BeginColumns(); + + for (const ColumnInfo& value : values) + { + if (value.PrimaryKey) + { + pkIndexBuilder.Column(value.Name); + } + } + + pkIndexBuilder.EndColumns(); + + pkIndexBuilder.Execute(connection); + + // Create an index on every value to improve performance + for (const ColumnInfo& value : values) + { + StatementBuilder createIndexBuilder; + + createIndexBuilder.CreateIndex({ s_PackagesTable_Table_Name, s_PackagesTable_Index_Separator, value.Name, s_PackagesTable_Index_Suffix }); + createIndexBuilder.On(s_PackagesTable_Table_Name).Columns(value.Name); + + createIndexBuilder.Execute(connection); + } + + savepoint.Commit(); + } + + // Creates a statement and executes it, select the actual values for a given manifest id. + // Ex. + // SELECT [ids].[id] FROM [manifest] + // JOIN [ids] ON [manifest].[id] = [ids].[rowid] + // WHERE [manifest].[rowid] = 1 + SQLite::Statement PackagesTableGetValuesById_Statement( + const SQLite::Connection& connection, + SQLite::rowid_t id, + std::initializer_list<std::string_view> columns, + bool stepAndVerify) + { + SQLite::Builder::StatementBuilder builder; + builder.Select(columns).From(s_PackagesTable_Table_Name).Where(SQLite::RowIDName).Equals(id); + + SQLite::Statement result = builder.Prepare(connection); + + if (stepAndVerify) + { + THROW_HR_IF(E_NOT_SET, !result.Step()); + } + + return result; + } + + std::vector<int> PackagesTableBuildSearchStatement( + SQLite::Builder::StatementBuilder& builder, + std::initializer_list<std::string_view> columns, + std::string_view primaryAlias, + std::string_view valueAlias, + bool useLike) + { + using QCol = SQLite::Builder::QualifiedColumn; + + // Build a statement like: + // SELECT manifest.rowid as m, ids.id as v from manifest + // join ids on manifest.id = ids.rowid + // where ids.id = <value> + // OR + // SELECT manifest.rowid as m, tags.tag as v from manifest + // join tags_map on manifest.rowid = tags_map.manifest + // join tags on tags_map.tag = tags.rowid + // where tags.tag = <value> + // Where the joins and where portions are repeated for each table in question. + builder.Select(). + Column(QCol(s_PackagesTable_Table_Name, SQLite::RowIDName)).As(primaryAlias); + + // Value will be captured for single tables references, and left empty for multi-tables + if (columns.size() == 1) + { + builder.Column(*columns.begin()); + } + else + { + builder.LiteralColumn(""); + } + + builder.As(valueAlias).From(s_PackagesTable_Table_Name); + + std::vector<int> result; + + // Create where clause + for (const auto& column : columns) + { + if (result.empty()) + { + builder.Where(column); + } + else + { + builder.And(column); + } + + if (useLike) + { + builder.Like(SQLite::Builder::Unbound); + result.push_back(builder.GetLastBindIndex()); + builder.Escape(SQLite::EscapeCharForLike); + } + else + { + builder.Equals(SQLite::Builder::Unbound); + result.push_back(builder.GetLastBindIndex()); + } + } + + return result; + } + + SQLite::Statement PackagesTableUpdateValueIdById_Statement(SQLite::Connection& connection, std::string_view valueName) + { + SQLite::Builder::StatementBuilder builder; + builder.Update(s_PackagesTable_Table_Name).Set().Column(valueName).Equals(SQLite::Builder::Unbound).Where(SQLite::RowIDName).Equals(SQLite::Builder::Unbound); + + return builder.Prepare(connection); + } + + void PackagesTablePrepareForPackaging(SQLite::Connection& connection, std::initializer_list<ColumnInfo> values) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "pfpPackagesTable_v2_0"); + + // Drop the index on the requested values + for (const auto& value : values) + { + SQLite::Builder::StatementBuilder dropIndexBuilder; + dropIndexBuilder.DropIndex({ s_PackagesTable_Table_Name, s_PackagesTable_Index_Separator, value.Name, s_PackagesTable_Index_Suffix }); + + dropIndexBuilder.Execute(connection); + } + + SQLite::Builder::StatementBuilder dropPKIndexBuilder; + dropPKIndexBuilder.DropIndex({ s_PackagesTable_Table_Name, s_PackagesTable_Index_Suffix }); + dropPKIndexBuilder.Execute(connection); + + savepoint.Commit(); + } + + bool PackagesTableCheckColumnForNulls(const SQLite::Connection& connection, std::string_view valueName, bool log) + { + // Build a select statement to find values that contain an embedded null character + // Such as: + // Select count(*) from table where instr(value,char(0))>0 + SQLite::Builder::StatementBuilder builder; + builder. + Select({ SQLite::RowIDName, valueName }). + From(s_PackagesTable_Table_Name). + WhereValueContainsEmbeddedNullCharacter(valueName); + + SQLite::Statement select = builder.Prepare(connection); + bool result = true; + + while (select.Step()) + { + result = false; + + if (!log) + { + break; + } + + AICLI_LOG(Repo, Info, << " [INVALID] value [" << valueName << "] in table [" << s_PackagesTable_Table_Name << + "] at row [" << select.GetColumn<SQLite::rowid_t>(0) << "] contains an embedded null character and starts with [" << + select.GetColumn<std::string>(1) << "]"); + } + + return result; + } + + bool PackagesTableCheckConsistency(const SQLite::Connection& connection, std::initializer_list<std::string_view> values, bool log) + { + bool result = true; + + for (const auto& value : values) + { + if (result || log) + { + result = details::PackagesTableCheckColumnForNulls(connection, value, log) && result; + } + } + + return result; + } + } + + std::string_view PackagesTable::TableName() + { + return s_PackagesTable_Table_Name; + } + + void PackagesTable::Drop(SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(s_PackagesTable_Table_Name); + + dropTableBuilder.Execute(connection); + } + + bool PackagesTable::Exists(const SQLite::Connection& connection) + { + using namespace SQLite; + + Builder::StatementBuilder builder; + builder.Select(Builder::RowCount).From(Builder::Schema::MainTable). + Where(Builder::Schema::TypeColumn).Equals(Builder::Schema::Type_Table).And(Builder::Schema::NameColumn).Equals(s_PackagesTable_Table_Name); + + Statement statement = builder.Prepare(connection); + THROW_HR_IF(E_UNEXPECTED, !statement.Step()); + return statement.GetColumn<int64_t>(0) != 0; + } + + void PackagesTable::AddColumn(SQLite::Connection& connection, const ColumnInfo& value) + { + using namespace SQLite::Builder; + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, "addColumnPackagesTable_v2_0"); + + StatementBuilder alterTableBuilder; + alterTableBuilder.AlterTable(s_PackagesTable_Table_Name).Add(value.Name, value.Type); + + alterTableBuilder.Execute(connection); + + savepoint.Commit(); + } + + SQLite::rowid_t PackagesTable::Insert(SQLite::Connection& connection, const std::vector<NameValuePair>& values) + { + SQLite::Builder::StatementBuilder builder; + builder.InsertInto(s_PackagesTable_Table_Name).BeginColumns(); + + for (const NameValuePair& value : values) + { + builder.Column(value.Name); + } + + builder.EndColumns().BeginValues(); + + for (const NameValuePair& value : values) + { + builder.Value(value.Value); + } + + builder.EndValues(); + + builder.Execute(connection); + + return connection.GetLastInsertRowID(); + } + + bool PackagesTable::ExistsById(const SQLite::Connection& connection, SQLite::rowid_t id) + { + SQLite::Builder::StatementBuilder builder; + builder.Select(SQLite::Builder::RowCount).From(s_PackagesTable_Table_Name).Where(SQLite::RowIDName).Equals(id); + + SQLite::Statement countStatement = builder.Prepare(connection); + + THROW_HR_IF(E_UNEXPECTED, !countStatement.Step()); + + return (countStatement.GetColumn<int>(0) != 0); + } + + std::vector<SQLite::rowid_t> PackagesTable::GetAllRowIds(const SQLite::Connection& connection, std::string_view orderByColumn, size_t limit) + { + SQLite::Builder::StatementBuilder selectBuilder; + selectBuilder.Select(SQLite::RowIDName).From(s_PackagesTable_Table_Name).OrderBy(orderByColumn); + + if (limit) + { + selectBuilder.Limit(limit); + } + + SQLite::Statement select = selectBuilder.Prepare(connection); + + std::vector<SQLite::rowid_t> result; + while (select.Step()) + { + result.emplace_back(select.GetColumn<SQLite::rowid_t>(0)); + } + return result; + } + + uint64_t PackagesTable::GetCount(const SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder builder; + builder.Select(SQLite::Builder::RowCount).From(s_PackagesTable_Table_Name); + + SQLite::Statement countStatement = builder.Prepare(connection); + + THROW_HR_IF(E_UNEXPECTED, !countStatement.Step()); + + return static_cast<uint64_t>(countStatement.GetColumn<SQLite::rowid_t>(0)); + } + + bool PackagesTable::IsEmpty(SQLite::Connection& connection) + { + SQLite::Builder::StatementBuilder builder; + builder.Select(SQLite::Builder::RowCount).From(s_PackagesTable_Table_Name); + + SQLite::Statement countStatement = builder.Prepare(connection); + + THROW_HR_IF(E_UNEXPECTED, !countStatement.Step()); + + return (countStatement.GetColumn<int>(0) == 0); + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackagesTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/PackagesTable.h @@ -0,0 +1,219 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <winget/SQLiteWrapper.h> +#include <winget/SQLiteStatementBuilder.h> +#include <initializer_list> +#include <optional> +#include <string_view> +#include <utility> +#include <vector> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + // Info on the columns. + struct ColumnInfo + { + template<typename Column> + static constexpr ColumnInfo Create() + { + ColumnInfo result; + result.Name = Column::Name; + result.Type = Column::Type; + result.PrimaryKey = Column::PrimaryKey; + result.AllowNull = Column::AllowNull; + return result; + } + + std::string_view Name; + SQLite::Builder::Type Type = {}; + bool PrimaryKey = false; + bool AllowNull = false; + }; + + // Creates the table. + void PackagesTableCreate(SQLite::Connection& connection, std::initializer_list<ColumnInfo> values); + + // Gets the requested values for the manifest with the given rowid. + SQLite::Statement PackagesTableGetValuesById_Statement( + const SQLite::Connection& connection, + SQLite::rowid_t id, + std::initializer_list<std::string_view> columns, + bool stepAndVerify = true); + + // Builds the search select statement base on the given values. + std::vector<int> PackagesTableBuildSearchStatement( + SQLite::Builder::StatementBuilder& builder, + std::initializer_list<std::string_view> columns, + std::string_view manifestAlias, + std::string_view valueAlias, + bool useLike); + + // Prepares a statement to update the value of a single column for the manifest with the given rowid. + // The first bind value will be the value to set. + // The second bind value will be the manifest rowid to modify. + SQLite::Statement PackagesTableUpdateValueIdById_Statement(SQLite::Connection& connection, std::string_view valueName); + + // Removes data that is no longer needed for an index that is to be published. + void PackagesTablePrepareForPackaging(SQLite::Connection& connection, std::initializer_list<ColumnInfo> values); + + // Checks for embedded nulls in the database. + bool PackagesTableCheckConsistency(const SQLite::Connection& connection, std::initializer_list<std::string_view> values, bool log); + } + + // A table in which each row represents a single package. + struct PackagesTable + { + // Get the table name. + static std::string_view TableName(); + + struct IdColumn + { + static constexpr std::string_view Name = "id"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Text; + static constexpr bool PrimaryKey = true; + static constexpr bool AllowNull = false; + }; + + struct NameColumn + { + static constexpr std::string_view Name = "name"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Text; + static constexpr bool PrimaryKey = false; + static constexpr bool AllowNull = false; + }; + + struct MonikerColumn + { + static constexpr std::string_view Name = "moniker"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Text; + static constexpr bool PrimaryKey = false; + static constexpr bool AllowNull = true; + }; + + struct LatestVersionColumn + { + static constexpr std::string_view Name = "latest_version"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Text; + static constexpr bool PrimaryKey = false; + static constexpr bool AllowNull = false; + }; + + struct ARPMinVersionColumn + { + static constexpr std::string_view Name = "arp_min_version"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Text; + static constexpr bool PrimaryKey = false; + static constexpr bool AllowNull = true; + }; + + struct ARPMaxVersionColumn + { + static constexpr std::string_view Name = "arp_max_version"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Text; + static constexpr bool PrimaryKey = false; + static constexpr bool AllowNull = true; + }; + + struct HashColumn + { + static constexpr std::string_view Name = "hash"sv; + static constexpr SQLite::Builder::Type Type = SQLite::Builder::Type::Blob; + static constexpr bool PrimaryKey = false; + static constexpr bool AllowNull = true; + }; + + using ColumnInfo = details::ColumnInfo; + + // Creates the table. + template <typename... Columns> + static void Create(SQLite::Connection& connection) + { + details::PackagesTableCreate(connection, { ColumnInfo::Create<Columns>()... }); + } + + // Drops the table. + static void Drop(SQLite::Connection& connection); + + // Determine if the table currently exists in the database. + static bool Exists(const SQLite::Connection& connection); + + // Alters the table, adding the column provided. + static void AddColumn(SQLite::Connection& connection, const ColumnInfo& value); + + // A string value for the package. + struct NameValuePair + { + std::string_view Name; + std::string Value; + }; + + // Insert the given values into the table. + static SQLite::rowid_t Insert(SQLite::Connection& connection, const std::vector<NameValuePair>& values); + + // Gets a value indicating whether the package with rowid exists. + static bool ExistsById(const SQLite::Connection& connection, SQLite::rowid_t rowid); + + // Gets all row ids from the table. + static std::vector<SQLite::rowid_t> GetAllRowIds(const SQLite::Connection& connection, std::string_view orderByColumn, size_t limit = 0); + + // Gets the total number of rows in the table. + static uint64_t GetCount(const SQLite::Connection& connection); + + // Gets the values requested for the package with the given rowid. + template <typename... Columns> + static auto GetValuesById(const SQLite::Connection& connection, SQLite::rowid_t rowid) + { + return details::PackagesTableGetValuesById_Statement(connection, rowid, { Columns::Name... }).GetRow<typename SQLite::Builder::TypeInfo<Columns::Type>::value_t...>(); + } + + // Gets the value requested for the package with the given rowid, if it exists. + template <typename Column> + static std::optional<typename SQLite::Builder::TypeInfo<Column::Type>::value_t> GetValueById(const SQLite::Connection& connection, SQLite::rowid_t rowid) + { + auto statement = details::PackagesTableGetValuesById_Statement(connection, rowid, { Column::Name }, false); + if (statement.Step()) { return statement.GetColumn<typename SQLite::Builder::TypeInfo<Column::Type>::value_t>(0); } + else { return std::nullopt; } + } + + // Builds the search select statement base on the given values. + // If more than one table is provided, no value will be captured. + // The return value is the bind indices of the values to match against. + template <typename... Columns> + static std::vector<int> BuildSearchStatement(SQLite::Builder::StatementBuilder& builder, std::string_view primaryAlias, std::string_view valueAlias, bool useLike) + { + return details::PackagesTableBuildSearchStatement(builder, { Columns::Name... }, primaryAlias, valueAlias, useLike); + } + + // Update the value of a single column for the package with the given rowid. + template <typename Column> + static void UpdateValueIdById(SQLite::Connection& connection, SQLite::rowid_t id, const typename SQLite::Builder::TypeInfo<Column::Type>::value_t& value) + { + auto stmt = details::PackagesTableUpdateValueIdById_Statement(connection, Column::Name); + stmt.Bind(1, value); + stmt.Bind(2, id); + stmt.Execute(); + } + + // Removes data that is no longer needed for an index that is to be published. + template <typename... Columns> + static void PrepareForPackaging(SQLite::Connection& connection) + { + details::PackagesTablePrepareForPackaging(connection, { ColumnInfo::Create<Columns>()... }); + } + + // Checks the consistency of the index to ensure that every referenced row exists. + // Returns true if index is consistent; false if it is not. + template <typename... Columns> + static bool CheckConsistency(const SQLite::Connection& connection, bool log) + { + return details::PackagesTableCheckConsistency(connection, { Columns::Name... }, log); + } + + // Determines if the table is empty. + static bool IsEmpty(SQLite::Connection& connection); + }; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/ProductCodeTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/ProductCodeTable.h @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/SystemReferenceStringTable.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct ProductCodeTableInfo + { + inline static constexpr std::string_view TableName() { return "productcodes2"sv; } + inline static constexpr std::string_view ValueName() { return "productcode"sv; } + }; + } + + using ProductCodeTable = SystemReferenceStringTable<details::ProductCodeTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SearchResultsTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SearchResultsTable.h @@ -0,0 +1,65 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <winget/SQLiteWrapper.h> +#include <winget/SQLiteTempTable.h> +#include "Microsoft/Schema/ISQLiteIndex.h" +#include "Public/winget/RepositorySearch.h" + +#include <optional> +#include <utility> +#include <vector> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + // Table for holding temporary search results. + struct SearchResultsTable : public SQLite::TempTable + { + SearchResultsTable(const SQLite::Connection& connection); + + SearchResultsTable(const SearchResultsTable&) = delete; + SearchResultsTable& operator=(const SearchResultsTable&) = delete; + + SearchResultsTable(SearchResultsTable&&) = default; + SearchResultsTable& operator=(SearchResultsTable&&) = default; + + // Performs the requested search type on the requested field. + void SearchOnField(const PackageMatchFilter& filter); + + // Removes rows with manifest ids whose sort order is below the highest one. + void RemoveDuplicateManifestRows(); + + // Prepares the table for a filtering pass. + void PrepareToFilter(); + + // Performs the requested filter type on the requested field. + void FilterOnField(const PackageMatchFilter& filter); + + // Completes a filtering pass, removing filtered rows. + void CompleteFilter(); + + // Gets the results from the table. + ISQLiteIndex::SearchResult GetSearchResults(size_t limit = 0); + + protected: + // Builds the search statement for the specified field and match type. + std::vector<int> BuildSearchStatement(SQLite::Builder::StatementBuilder& builder, PackageMatchField field, MatchType match) const; + + virtual std::vector<int> BuildSearchStatement( + SQLite::Builder::StatementBuilder& builder, + PackageMatchField field, + std::string_view manifestAlias, + std::string_view valueAlias, + bool useLike) const; + + static bool MatchUsesLike(MatchType match); + void BindStatementForMatchType(SQLite::Statement& statement, MatchType match, int bindIndex, std::string_view value); + + virtual void BindStatementForMatchType(SQLite::Statement& statement, const PackageMatchFilter& filter, const std::vector<int>& bindIndex); + + private: + const SQLite::Connection& m_connection; + int m_sortOrdinalValue = 0; + }; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SearchResultsTable_2_0.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SearchResultsTable_2_0.cpp @@ -0,0 +1,301 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "SearchResultsTable.h" +#include <winget/SQLiteStatementBuilder.h> + +#include "Microsoft/Schema/2_0/PackagesTable.h" +#include "Microsoft/Schema/2_0/TagsTable.h" +#include "Microsoft/Schema/2_0/CommandsTable.h" + + +// TODO :: The code here was copied from a previous schema version and is currently a placeholder !!! + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace + { + using namespace std::string_literals; + using namespace std::string_view_literals; + + constexpr std::string_view s_SearchResultsTable_Manifest = "manifest"sv; + constexpr std::string_view s_SearchResultsTable_MatchField = "field"sv; + constexpr std::string_view s_SearchResultsTable_MatchType = "match"sv; + constexpr std::string_view s_SearchResultsTable_MatchValue = "value"sv; + constexpr std::string_view s_SearchResultsTable_SortValue = "sort"sv; + constexpr std::string_view s_SearchResultsTable_Filter = "filter"sv; + + constexpr std::string_view s_SearchResultsTable_Index_Suffix = "_i_m"sv; + + constexpr std::string_view s_SearchResultsTable_SubSelect_TableAlias = "valueTable"sv; + constexpr std::string_view s_SearchResultsTable_SubSelect_ManifestAlias = "m"sv; + constexpr std::string_view s_SearchResultsTable_SubSelect_ValueAlias = "v"sv; + } + + SearchResultsTable::SearchResultsTable(const SQLite::Connection& connection) : + m_connection(connection) + { + using namespace SQLite::Builder; + + { + StatementBuilder builder; + builder.CreateTable(GetQualifiedName()).BeginColumns(); + + builder.Column(ColumnBuilder(s_SearchResultsTable_Manifest, Type::RowId).NotNull()); + builder.Column(ColumnBuilder(s_SearchResultsTable_MatchField, Type::Int).NotNull()); + builder.Column(ColumnBuilder(s_SearchResultsTable_MatchType, Type::Int).NotNull()); + builder.Column(ColumnBuilder(s_SearchResultsTable_MatchValue, Type::Text).NotNull()); + builder.Column(ColumnBuilder(s_SearchResultsTable_SortValue, Type::Int).NotNull()); + builder.Column(ColumnBuilder(s_SearchResultsTable_Filter, Type::Bool).NotNull()); + + builder.EndColumns(); + + builder.Execute(m_connection); + } + + InitDropStatement(m_connection); + + { + SQLite::Builder::QualifiedTable index = GetQualifiedName(); + std::string indexName(index.Table); + indexName += s_SearchResultsTable_Index_Suffix; + index.Table = indexName; + + StatementBuilder builder; + builder.CreateIndex(indexName).On(GetQualifiedName().Table).Columns(s_SearchResultsTable_Manifest); + + builder.Execute(m_connection); + } + } + + void SearchResultsTable::SearchOnField(const PackageMatchFilter& filter) + { + using namespace SQLite::Builder; + + int sortOrdinal = m_sortOrdinalValue++; + + // Create an insert statement to select values into the table as requested. + // The goal is a statement like this: + // INSERT INTO <tempTable> + // SELECT valueTable.m, <field>, <match>, valueTable.v, <sort>, <filter> FROM + // (SELECT manifest.rowid as m, manifest.id as v from manifest join ids on manifest.id = ids.rowid where ids.id = <value>) AS valueTable + // Where the subselect is built by the owning table. + StatementBuilder builder; + builder.InsertInto(GetQualifiedName()).Select(). + Column(QualifiedColumn(s_SearchResultsTable_SubSelect_TableAlias, s_SearchResultsTable_SubSelect_ManifestAlias)). + Value(filter.Field). + Value(filter.Type). + Column(QualifiedColumn(s_SearchResultsTable_SubSelect_TableAlias, s_SearchResultsTable_SubSelect_ValueAlias)). + Value(sortOrdinal). + Value(false). + From().BeginParenthetical(); + + // Add the field specific portion + std::vector<int> bindIndex = BuildSearchStatement(builder, filter.Field, filter.Type); + + if (bindIndex.empty()) + { + AICLI_LOG(Repo, Verbose, << "PackageMatchField not supported in this version: " << ToString(filter.Field)); + return; + } + + builder.EndParenthetical().As(s_SearchResultsTable_SubSelect_TableAlias); + + SQLite::Statement statement = builder.Prepare(m_connection); + BindStatementForMatchType(statement, filter, bindIndex); + statement.Execute(); + AICLI_LOG(SQL, Verbose, << "Search found " << m_connection.GetChanges() << " rows"); + } + + void SearchResultsTable::RemoveDuplicateManifestRows() + { + using namespace SQLite::Builder; + + // Create a delete statement to leave only one row with a given manifest. + // This will arbitrarily choose one of the rows if multiple have the same lowest sort order. + // The goal is a statement like this: + // DELETE from <temp> where rowid not in ( + // SELECT rowid from ( + // SELECT rowid, min(sort) from <temp> group by manifest + // ) + // ) + StatementBuilder builder; + builder.DeleteFrom(GetQualifiedName()).Where(SQLite::RowIDName).Not().In().BeginParenthetical(). + Select(SQLite::RowIDName).From().BeginParenthetical(). + Select().Column(SQLite::RowIDName).Column(Aggregate::Min, s_SearchResultsTable_SortValue).From(GetQualifiedName()).GroupBy(s_SearchResultsTable_Manifest). + EndParenthetical(). + EndParenthetical(); + + builder.Execute(m_connection); + AICLI_LOG(SQL, Verbose, << "Removed " << m_connection.GetChanges() << " duplicate rows"); + } + + void SearchResultsTable::PrepareToFilter() + { + // Reset all filter values to unselected + SQLite::Builder::StatementBuilder builder; + builder.Update(GetQualifiedName()).Set().Column(s_SearchResultsTable_Filter).Equals(false); + + builder.Execute(m_connection); + } + + void SearchResultsTable::FilterOnField(const PackageMatchFilter& filter) + { + using namespace SQLite::Builder; + + // Create an update statement to mark rows that are found by the search. + // This will arbitrarily choose one of the rows if multiple have the same lowest sort order. + // The goal is a statement like this: + // UPDATE <temp> set filter = 1 where manifest in ( + // SELECT m from ( + // SELECT manifest.rowid as m, manifest.id as v from manifest join ids on manifest.id = ids.rowid where ids.id = <value> + // ) + // ) + StatementBuilder builder; + builder.Update(GetQualifiedName()).Set().Column(s_SearchResultsTable_Filter).Equals(true).Where(s_SearchResultsTable_Manifest).In().BeginParenthetical(). + Select(s_SearchResultsTable_SubSelect_ManifestAlias).From().BeginParenthetical(); + + // Add the field specific portion + std::vector<int> bindIndex = BuildSearchStatement(builder, filter.Field, filter.Type); + + if (bindIndex.empty()) + { + AICLI_LOG(Repo, Verbose, << "PackageMatchField not supported in this version: " << ToString(filter.Field)); + return; + } + + builder.EndParenthetical().EndParenthetical(); + + SQLite::Statement statement = builder.Prepare(m_connection); + BindStatementForMatchType(statement, filter, bindIndex); + statement.Execute(); + AICLI_LOG(SQL, Verbose, << "Filter kept " << m_connection.GetChanges() << " rows"); + } + + void SearchResultsTable::CompleteFilter() + { + // Delete all unselected values + SQLite::Builder::StatementBuilder builder; + builder.DeleteFrom(GetQualifiedName()).Where(s_SearchResultsTable_Filter).Equals(false); + + builder.Execute(m_connection); + AICLI_LOG(SQL, Verbose, << "Filter deleted " << m_connection.GetChanges() << " rows"); + } + + ISQLiteIndex::SearchResult SearchResultsTable::GetSearchResults(size_t limit) + { + constexpr std::string_view tempTableAlias = "t"sv; + + using namespace SQLite::Builder; + using QCol = QualifiedColumn; + + // Select all unique ids from the results table, and their highest ordered match. + // The goal is a statement like this: + // SELECT m.id, field, match, value, min(sort) from <temp> join manifest on rowid = manifest group by m.id order by t.sort + // Through the "group by m.id", we will only ever have one row per id, and the "min(sort)" returns us one of the rows that matched + // through the earliest search. We also order by the sort value to have the earliest search matches first in the list + StatementBuilder builder; + //builder.Select(). + // Column(QCol(ManifestTable::TableName(), IdTable::ValueName())). + // Column(QCol(tempTableAlias, s_SearchResultsTable_MatchField)). + // Column(QCol(tempTableAlias, s_SearchResultsTable_MatchType)). + // Column(QCol(tempTableAlias, s_SearchResultsTable_MatchValue)). + // Column(Aggregate::Min, QCol(tempTableAlias, s_SearchResultsTable_SortValue)). + //From(GetQualifiedName()).As(tempTableAlias). + // Join(ManifestTable::TableName()).On(QCol(tempTableAlias, s_SearchResultsTable_Manifest), QCol(ManifestTable::TableName(), SQLite::RowIDName)). + // GroupBy(QCol(ManifestTable::TableName(), IdTable::ValueName())).OrderBy(QCol(tempTableAlias, s_SearchResultsTable_SortValue)); + + SQLite::Statement select = builder.Prepare(m_connection); + + ISQLiteIndex::SearchResult result; + while (select.Step()) + { + if (limit && result.Matches.size() >= limit) + { + break; + } + + result.Matches.emplace_back(select.GetColumn<SQLite::rowid_t>(0), + PackageMatchFilter(select.GetColumn<PackageMatchField>(1), select.GetColumn<MatchType>(2), select.GetColumn<std::string>(3))); + } + + result.Truncated = (select.GetState() != SQLite::Statement::State::Completed); + + return result; + } + + std::vector<int> SearchResultsTable::BuildSearchStatement(SQLite::Builder::StatementBuilder& builder, PackageMatchField field, MatchType match) const + { + return BuildSearchStatement(builder, field, s_SearchResultsTable_SubSelect_ManifestAlias, s_SearchResultsTable_SubSelect_ValueAlias, MatchUsesLike(match)); + } + + std::vector<int> SearchResultsTable::BuildSearchStatement( + SQLite::Builder::StatementBuilder& builder, + PackageMatchField field, + std::string_view manifestAlias, + std::string_view valueAlias, + bool useLike) const + { + switch (field) + { + case PackageMatchField::Id: + return PackagesTable::BuildSearchStatement<PackagesTable::IdColumn>(builder, manifestAlias, valueAlias, useLike); + case PackageMatchField::Name: + return PackagesTable::BuildSearchStatement<PackagesTable::NameColumn>(builder, manifestAlias, valueAlias, useLike); + case PackageMatchField::Moniker: + return PackagesTable::BuildSearchStatement<PackagesTable::MonikerColumn>(builder, manifestAlias, valueAlias, useLike); + //case PackageMatchField::Tag: + // return PackagesTable::BuildSearchStatement<TagsTable>(builder, manifestAlias, valueAlias, useLike); + //case PackageMatchField::Command: + // return PackagesTable::BuildSearchStatement<CommandsTable>(builder, manifestAlias, valueAlias, useLike); + default: + return {}; + } + } + + bool SearchResultsTable::MatchUsesLike(MatchType match) + { + return (match != MatchType::Exact); + } + + void SearchResultsTable::BindStatementForMatchType(SQLite::Statement& statement, MatchType match, int bindIndex, std::string_view value) + { + std::string valueToUse; + + if (MatchUsesLike(match)) + { + valueToUse = SQLite::EscapeStringForLike(value); + } + else + { + valueToUse = value; + } + + switch (match) + { + case AppInstaller::Repository::MatchType::StartsWith: + valueToUse += '%'; + break; + case AppInstaller::Repository::MatchType::Substring: + valueToUse = "%"s + valueToUse + '%'; + break; + default: + // No changes required for others. + break; + } + + statement.Bind(bindIndex, valueToUse); + } + + void SearchResultsTable::BindStatementForMatchType(SQLite::Statement& statement, const PackageMatchFilter& filter, const std::vector<int>& bindIndex) + { + // TODO: Implement these more complex match types + if (filter.Type == MatchType::Wildcard || filter.Type == MatchType::Fuzzy || filter.Type == MatchType::FuzzySubstring) + { + AICLI_LOG(Repo, Verbose, << "Specific match type not implemented, skipping: " << ToString(filter.Type)); + return; + } + + BindStatementForMatchType(statement, filter.Type, bindIndex[0], filter.Value); + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SystemReferenceStringTable.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SystemReferenceStringTable.cpp @@ -0,0 +1,176 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "Microsoft/Schema/2_0/SystemReferenceStringTable.h" +#include "Microsoft/Schema/2_0/PackagesTable.h" +#include <winget/SQLiteStatementBuilder.h> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using PrimaryTable = PackagesTable; + + using namespace std::string_view_literals; + static constexpr std::string_view s_SystemReferenceStringTable_PrimaryName = "package"sv; + + std::string_view SystemReferenceStringTableGetPrimaryColumnName() + { + return s_SystemReferenceStringTable_PrimaryName; + } + + void SystemReferenceStringTableCreate(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName) + { + using namespace SQLite::Builder; + + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_create_v2_0"); + + StatementBuilder createTableBuilder; + createTableBuilder.CreateTable(tableName).Columns({ + ColumnBuilder(valueName, Type::Text).NotNull(), + ColumnBuilder(s_SystemReferenceStringTable_PrimaryName, Type::RowId).NotNull(), + PrimaryKeyBuilder({ valueName, s_SystemReferenceStringTable_PrimaryName }) + }).WithoutRowID(); + + createTableBuilder.Execute(connection); + + savepoint.Commit(); + } + + void SystemReferenceStringTableDrop(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder dropTableBuilder; + dropTableBuilder.DropTable(tableName); + + dropTableBuilder.Execute(connection); + } + + std::vector<std::string> SystemReferenceStringTableGetValuesByPrimaryId( + const SQLite::Connection& connection, + std::string_view tableName, + std::string_view valueName, + SQLite::rowid_t primaryId) + { + std::vector<std::string> result; + + SQLite::Builder::StatementBuilder builder; + builder.Select(valueName). + From(tableName).Where(s_SystemReferenceStringTable_PrimaryName).Equals(primaryId); + + SQLite::Statement statement = builder.Prepare(connection); + + while (statement.Step()) + { + result.emplace_back(statement.GetColumn<std::string>(0)); + } + + return result; + } + + void SystemReferenceStringTableEnsureExists( + SQLite::Connection& connection, + std::string_view tableName, + std::string_view valueName, + const std::vector<std::string>& values, + SQLite::rowid_t primaryId) + { + SQLite::Savepoint savepoint = SQLite::Savepoint::Create(connection, std::string{ tableName } + "_ensure_v2_0"); + + SQLite::Builder::StatementBuilder builder; + + builder.InsertOrIgnore(tableName). + Columns({ valueName, s_SystemReferenceStringTable_PrimaryName }).Values(SQLite::Builder::Unbound, primaryId); + + SQLite::Statement insertStatement = builder.Prepare(connection); + + for (const std::string& value : values) + { + // Second, insert into the mapping table + insertStatement.Reset(); + insertStatement.Bind(1, value); + + insertStatement.Execute(); + } + + savepoint.Commit(); + } + + bool SystemReferenceStringTableCheckConsistency(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, bool log) + { + using QCol = SQLite::Builder::QualifiedColumn; + + bool result = true; + + { + // Build a select statement to find rows containing references to primaries with nonexistent rowids + // Such as: + // Select data.data, data.primary from data left outer join primary on data.primary = primary.rowid where primary.id is null + + SQLite::Builder::StatementBuilder builder; + builder. + Select({ QCol(tableName, valueName), QCol(tableName, s_SystemReferenceStringTable_PrimaryName) }). + From(tableName). + LeftOuterJoin(details::PrimaryTable::TableName()).On(QCol(tableName, s_SystemReferenceStringTable_PrimaryName), QCol(details::PrimaryTable::TableName(), SQLite::RowIDName)). + Where(QCol(details::PrimaryTable::TableName(), SQLite::RowIDName)).IsNull(); + + SQLite::Statement select = builder.Prepare(connection); + + while (select.Step()) + { + result = false; + + if (!log) + { + break; + } + + AICLI_LOG(Repo, Info, << " [INVALID] " << tableName << " [" << select.GetColumn<std::string>(0) << + ", " << select.GetColumn<SQLite::rowid_t>(1) << "] refers to invalid " << details::PrimaryTable::TableName()); + } + } + + if (!result && !log) + { + return result; + } + + // Build a select statement to find values that contain an embedded null character + // Such as: + // Select count(*) from table where instr(value,char(0))>0 + SQLite::Builder::StatementBuilder builder; + builder. + Select({ valueName, s_SystemReferenceStringTable_PrimaryName }). + From(tableName). + WhereValueContainsEmbeddedNullCharacter(valueName); + + SQLite::Statement select = builder.Prepare(connection); + + while (select.Step()) + { + result = false; + + if (!log) + { + break; + } + + AICLI_LOG(Repo, Info, << " [INVALID] value in table [" << tableName << "] for primary [" << select.GetColumn<SQLite::rowid_t>(1) << "] contains an embedded null character and starts with [" << select.GetColumn<std::string>(0) << "]"); + } + + return result; + } + + bool SystemReferenceStringTableIsEmpty(SQLite::Connection& connection, std::string_view tableName) + { + SQLite::Builder::StatementBuilder countBuilder; + countBuilder.Select(SQLite::Builder::RowCount).From(tableName); + + SQLite::Statement countStatement = countBuilder.Prepare(connection); + + THROW_HR_IF(E_UNEXPECTED, !countStatement.Step()); + + return countStatement.GetColumn<int>(0) == 0; + } + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SystemReferenceStringTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/SystemReferenceStringTable.h @@ -0,0 +1,100 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <winget/SQLiteWrapper.h> +#include <AppInstallerStrings.h> +#include <string> +#include <string_view> +#include <vector> + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + // Returns the primary column name. + std::string_view SystemReferenceStringTableGetPrimaryColumnName(); + + // Create the table. + void SystemReferenceStringTableCreate(SQLite::Connection& connection, std::string_view tableName, std::string_view valueName); + + // Drops the table. + void SystemReferenceStringTableDrop(SQLite::Connection& connection, std::string_view tableName); + + // Gets all values associated with the given primary id. + std::vector<std::string> SystemReferenceStringTableGetValuesByPrimaryId( + const SQLite::Connection& connection, + std::string_view tableName, + std::string_view valueName, + SQLite::rowid_t primaryId); + + // Ensures that the value exists and inserts mapping entries. + void SystemReferenceStringTableEnsureExists( + SQLite::Connection& connection, + std::string_view tableName, + std::string_view valueName, + const std::vector<std::string>& values, + SQLite::rowid_t primaryId); + + // Checks the consistency of the index to ensure that every referenced row exists. + // Returns true if index is consistent; false if it is not. + bool SystemReferenceStringTableCheckConsistency(const SQLite::Connection& connection, std::string_view tableName, std::string_view valueName, bool log); + + // Determines if the table is empty. + bool SystemReferenceStringTableIsEmpty(SQLite::Connection& connection, std::string_view tableName); + } + + // A table that represents a value that is 1:N with a primary entry. + template <typename TableInfo> + struct SystemReferenceStringTable + { + // The name of the table. + static constexpr std::string_view TableName() + { + return TableInfo::TableName(); + } + + // The value name of the table. + static constexpr std::string_view ValueName() + { + return TableInfo::ValueName(); + } + + // Creates the table. + static void Create(SQLite::Connection& connection) + { + details::SystemReferenceStringTableCreate(connection, TableInfo::TableName(), TableInfo::ValueName()); + } + + // Drops the table. + static void Drop(SQLite::Connection& connection) + { + details::SystemReferenceStringTableDrop(connection, TableInfo::TableName()); + } + + // Gets all values associated with the given primary id. + static std::vector<std::string> GetValuesByPrimaryId(const SQLite::Connection& connection, SQLite::rowid_t primaryId) + { + return details::SystemReferenceStringTableGetValuesByPrimaryId(connection, TableInfo::TableName(), TableInfo::ValueName(), primaryId); + } + + // Ensures that all values exist in the data table, and inserts into the mapping table for the given primary id. + static void EnsureExists(SQLite::Connection& connection, const std::vector<std::string>& values, SQLite::rowid_t primaryId) + { + details::SystemReferenceStringTableEnsureExists(connection, TableInfo::TableName(), TableInfo::ValueName(), values, primaryId); + } + + // Checks the consistency of the index to ensure that every referenced row exists. + // Returns true if index is consistent; false if it is not. + static bool CheckConsistency(const SQLite::Connection& connection, bool log) + { + return details::SystemReferenceStringTableCheckConsistency(connection, TableInfo::TableName(), TableInfo::ValueName(), log); + } + + // Determines if the table is empty. + static bool IsEmpty(SQLite::Connection& connection) + { + return details::SystemReferenceStringTableIsEmpty(connection, TableInfo::TableName()); + } + }; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/TagsTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/TagsTable.h @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/OneToManyTableWithMap.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct TagsTableInfo + { + inline static constexpr std::string_view TableName() { return "tags2"sv; } + inline static constexpr std::string_view ValueName() { return "tag"sv; } + }; + } + + using TagsTable = OneToManyTableWithMap<details::TagsTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/UpgradeCodeTable.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/2_0/UpgradeCodeTable.h @@ -0,0 +1,21 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include "Microsoft/Schema/2_0/SystemReferenceStringTable.h" + + +namespace AppInstaller::Repository::Microsoft::Schema::V2_0 +{ + namespace details + { + using namespace std::string_view_literals; + + struct UpgradeCodeTableInfo + { + inline static constexpr std::string_view TableName() { return "upgradecodes2"sv; } + inline static constexpr std::string_view ValueName() { return "upgradecode"sv; } + }; + } + + using UpgradeCodeTable = SystemReferenceStringTable<details::UpgradeCodeTableInfo>; +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/ISQLiteIndex.cpp b/src/AppInstallerRepositoryCore/Microsoft/Schema/ISQLiteIndex.cpp @@ -0,0 +1,86 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "Microsoft/Schema/ISQLiteIndex.h" + +#include "Microsoft/Schema/1_0/Interface.h" +#include "Microsoft/Schema/1_1/Interface.h" +#include "Microsoft/Schema/1_2/Interface.h" +#include "Microsoft/Schema/1_3/Interface.h" +#include "Microsoft/Schema/1_4/Interface.h" +#include "Microsoft/Schema/1_5/Interface.h" +#include "Microsoft/Schema/1_6/Interface.h" +#include "Microsoft/Schema/1_7/Interface.h" +#include "Microsoft/Schema/2_0/Interface.h" + +namespace AppInstaller::Repository::Microsoft::Schema +{ + void ISQLiteIndex::PrepareForPackaging(const SQLiteIndexContext& context) + { + PrepareForPackaging(context.Connection); + } + + void ISQLiteIndex::SetProperty(SQLite::Connection&, Property, const std::string&) + { + THROW_WIN32(ERROR_NOT_SUPPORTED); + } + + std::unique_ptr<ISQLiteIndex> CreateISQLiteIndex(const SQLite::Version& version) + { + if (version.MajorVersion == 1 || + version.IsLatest()) + { + constexpr std::array<std::unique_ptr<ISQLiteIndex>(*)(), 8> versionCreatorMap = + { + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_0::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_1::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_2::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_3::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_4::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_5::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_6::Interface>()); }, + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V1_7::Interface>()); }, + }; + + return versionCreatorMap[std::min(static_cast<size_t>(version.MinorVersion), versionCreatorMap.size() - 1)](); + } + + // Version 2.0 is designed solely for minimizing the size of the index for transport. + // Unless it is prepared for packaging, it will be identical to a 1.N index. + if (version.MajorVersion == 2) + { + constexpr std::array<std::unique_ptr<ISQLiteIndex>(*)(), 1> versionCreatorMap = + { + []() { return std::unique_ptr<ISQLiteIndex>(std::make_unique<V2_0::Interface>()); }, + }; + + return versionCreatorMap[std::min(static_cast<size_t>(version.MinorVersion), versionCreatorMap.size() - 1)](); + } + + // We do not have the capacity to operate on this schema version + THROW_WIN32(ERROR_NOT_SUPPORTED); + } + + std::vector<MatchType> GetDefaultMatchTypeOrder(MatchType type) + { + switch (type) + { + case MatchType::Exact: + return { MatchType::Exact }; + case MatchType::CaseInsensitive: + return { MatchType::CaseInsensitive }; + case MatchType::StartsWith: + return { MatchType::CaseInsensitive, MatchType::StartsWith }; + case MatchType::Substring: + return { MatchType::CaseInsensitive, MatchType::Substring }; + case MatchType::Wildcard: + return { MatchType::Wildcard }; + case MatchType::Fuzzy: + return { MatchType::CaseInsensitive, MatchType::Fuzzy }; + case MatchType::FuzzySubstring: + return { MatchType::CaseInsensitive, MatchType::Fuzzy, MatchType::Substring, MatchType::FuzzySubstring }; + default: + THROW_HR(E_UNEXPECTED); + } + } +} diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/ISQLiteIndex.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/ISQLiteIndex.h @@ -4,6 +4,7 @@ #include <winget/SQLiteWrapper.h> #include <winget/SQLiteVersion.h> #include "ISource.h" +#include "Microsoft/Schema/SQLiteIndexContextData.h" #include <AppInstallerVersions.h> #include <winget/Manifest.h> #include <winget/NameNormalization.h> @@ -14,6 +15,13 @@ namespace AppInstaller::Repository::Microsoft::Schema { + // Contains the database connection and any other data that the owning index might need to pass in. + struct SQLiteIndexContext + { + SQLite::Connection& Connection; + SQLiteIndexContextData& Data; + }; + // The common interface used to interact with all schema versions of the index. struct ISQLiteIndex { @@ -77,6 +85,9 @@ namespace AppInstaller::Repository::Microsoft::Schema // Removes data that is no longer needed for an index that is to be published. virtual void PrepareForPackaging(SQLite::Connection& connection) = 0; + // Removes data that is no longer needed for an index that is to be published. + virtual void PrepareForPackaging(const SQLiteIndexContext& context); + // Checks the consistency of the index to ensure that every referenced row exists. // Returns true if index is consistent; false if it is not. virtual bool CheckConsistency(const SQLite::Connection& connection, bool log) const = 0; @@ -108,15 +119,40 @@ namespace AppInstaller::Repository::Microsoft::Schema // Sets the string for the given metadata and manifest id. virtual void SetMetadataByManifestId(SQLite::Connection& connection, SQLite::rowid_t manifestId, PackageVersionMetadata metadata, std::string_view value) = 0; + // Version 1.2 + // Normalizes a name using the internal rules used by the index. // Largely a utility function; should not be used to do work on behalf of the index by the caller. virtual Utility::NormalizedName NormalizeName(std::string_view name, std::string_view publisher) const = 0; + // Version 1.4 + // Get all the dependencies for a specific manifest. virtual std::set<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependenciesByManifestRowId(const SQLite::Connection& connection, SQLite::rowid_t manifestRowId) const = 0; virtual std::vector<std::pair<SQLite::rowid_t, Utility::NormalizedString>> GetDependentsById(const SQLite::Connection& connection, AppInstaller::Manifest::string_t packageId) const = 0; + + // Version 1.7 + + // Drops all tables that would have been created. + virtual void DropTables(SQLite::Connection& connection) = 0; + + // Version 2.0 + + // Migrates from the current interface given. + // Returns true if supported; false if not. + // Throws on errors that occur during an attempted migration. + virtual bool MigrateFrom(SQLite::Connection& connection, const ISQLiteIndex* current) = 0; + + // Set the property value. + virtual void SetProperty(SQLite::Connection& connection, Property property, const std::string& value); }; DEFINE_ENUM_FLAG_OPERATORS(ISQLiteIndex::CreateOptions); + + // Creates the ISQLiteIndex interface object for the given version. + std::unique_ptr<ISQLiteIndex> CreateISQLiteIndex(const SQLite::Version& version); + + // For a given match type, gets the set of match types that are more specific subsets of it. + std::vector<MatchType> GetDefaultMatchTypeOrder(MatchType type); } diff --git a/src/AppInstallerRepositoryCore/Microsoft/Schema/SQLiteIndexContextData.h b/src/AppInstallerRepositoryCore/Microsoft/Schema/SQLiteIndexContextData.h @@ -0,0 +1,50 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <AppInstallerLanguageUtilities.h> +#include <filesystem> + + +namespace AppInstaller::Repository::Microsoft::Schema +{ + // Names a property + enum class Property : size_t + { + PackageUpdateTrackingBaseTime, + IntermediateFileOutputPath, + DatabaseFilePath, + Max + }; + + namespace details + { + template <Property D> + struct PropertyMapping + { + // value_t type specifies the type of this property + }; + + template <> + struct PropertyMapping<Property::PackageUpdateTrackingBaseTime> + { + using value_t = int64_t; + static constexpr bool SetThroughInterface = true; + }; + + template <> + struct PropertyMapping<Property::IntermediateFileOutputPath> + { + using value_t = std::filesystem::path; + static constexpr bool SetThroughInterface = false; + }; + + template <> + struct PropertyMapping<Property::DatabaseFilePath> + { + using value_t = std::filesystem::path; + static constexpr bool SetThroughInterface = false; + }; + } + + using SQLiteIndexContextData = EnumBasedVariantMap<Property, details::PropertyMapping>; +} diff --git a/src/AppInstallerRepositoryCore/Public/winget/RepositorySearch.h b/src/AppInstallerRepositoryCore/Public/winget/RepositorySearch.h @@ -152,6 +152,7 @@ namespace AppInstaller::Repository Publisher, ArpMinVersion, ArpMaxVersion, + Moniker, }; // A property of a package version that can have multiple values. @@ -172,6 +173,10 @@ namespace AppInstaller::Repository // The locale of the matching Name and Publisher values; ideally these would match in number and order with both Name and Publisher. // May be empty if there is only a single value for Name and Publisher. Locale, + // The tags associated with a package version. + Tag, + // The commands associated with a package version. + Command, }; // A metadata item of a package version. These values are persisted and cannot be changed. diff --git a/src/AppInstallerSharedLib/AppInstallerSharedLib.vcxproj b/src/AppInstallerSharedLib/AppInstallerSharedLib.vcxproj @@ -409,6 +409,7 @@ <ClInclude Include="Public\Telemetry\WinEventLogLevels.h" /> <ClInclude Include="Public\winget\AsyncTokens.h" /> <ClInclude Include="Public\winget\Certificates.h" /> + <ClInclude Include="Public\winget\Compression.h" /> <ClInclude Include="Public\winget\ConfigurationSetProcessorHandlers.h" /> <ClInclude Include="Public\winget\GroupPolicy.h" /> <ClInclude Include="Public\winget\IConfigurationStaticsInternals.h" /> @@ -435,6 +436,7 @@ <ClCompile Include="AppInstallerLogging.cpp" /> <ClCompile Include="AppInstallerStrings.cpp" /> <ClCompile Include="Certificates.cpp" /> + <ClCompile Include="Compression.cpp" /> <ClCompile Include="DateTime.cpp" /> <ClCompile Include="Errors.cpp" /> <ClCompile Include="GroupPolicy.cpp" /> diff --git a/src/AppInstallerSharedLib/AppInstallerSharedLib.vcxproj.filters b/src/AppInstallerSharedLib/AppInstallerSharedLib.vcxproj.filters @@ -128,6 +128,9 @@ <ClInclude Include="Public\winget\SQLiteMetadataTable.h"> <Filter>Public\winget</Filter> </ClInclude> + <ClInclude Include="Public\winget\Compression.h"> + <Filter>Public\winget</Filter> + </ClInclude> </ItemGroup> <ItemGroup> <ClCompile Include="pch.cpp"> @@ -208,6 +211,9 @@ <ClCompile Include="ManagedFile.cpp"> <Filter>Source Files</Filter> </ClCompile> + <ClCompile Include="Compression.cpp"> + <Filter>Source Files</Filter> + </ClCompile> </ItemGroup> <ItemGroup> <None Include="PropertySheet.props" /> diff --git a/src/AppInstallerSharedLib/Compression.cpp b/src/AppInstallerSharedLib/Compression.cpp @@ -0,0 +1,93 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#include "pch.h" +#include "Public/winget/Compression.h" + +namespace AppInstaller::Compression +{ + Compressor::Compressor(DWORD algorithm) + { + THROW_IF_WIN32_BOOL_FALSE(CreateCompressor(algorithm, nullptr, &m_compressor)); + } + + std::vector<uint8_t> Compressor::Compress(std::string_view data) + { + std::vector<uint8_t> result; + + if (!data.empty()) + { + SIZE_T compressedBufferSize = 0; + THROW_HR_IF(E_UNEXPECTED, ::Compress(m_compressor.get(), data.data(), data.size(), nullptr, 0, &compressedBufferSize)); + THROW_LAST_ERROR_IF(GetLastError() != ERROR_INSUFFICIENT_BUFFER); + + result.resize(compressedBufferSize); + + SIZE_T compressedDataSize = 0; + THROW_IF_WIN32_BOOL_FALSE(::Compress(m_compressor.get(), data.data(), data.size(), &result[0], result.size(), &compressedDataSize)); + + result.resize(compressedDataSize); + } + + return result; + } + + void Compressor::Reset() + { + THROW_IF_WIN32_BOOL_FALSE(ResetCompressor(m_compressor.get())); + } + + void Compressor::SetInformation(COMPRESS_INFORMATION_CLASS information, DWORD value) + { + THROW_IF_WIN32_BOOL_FALSE(SetCompressorInformation(m_compressor.get(), information, &value, sizeof(value))); + } + + DWORD Compressor::GetInformation(COMPRESS_INFORMATION_CLASS information) + { + DWORD result = 0; + THROW_IF_WIN32_BOOL_FALSE(QueryCompressorInformation(m_compressor.get(), information, &result, sizeof(result))); + return result; + } + + Decompressor::Decompressor(DWORD algorithm) + { + THROW_IF_WIN32_BOOL_FALSE(CreateDecompressor(algorithm, nullptr, &m_decompressor)); + } + + std::vector<uint8_t> Decompressor::Decompress(const std::vector<uint8_t>& data) + { + std::vector<uint8_t> result; + + if (!data.empty()) + { + SIZE_T decompressedBufferSize = 0; + THROW_HR_IF(E_UNEXPECTED, ::Decompress(m_decompressor.get(), data.data(), data.size(), nullptr, 0, &decompressedBufferSize)); + THROW_LAST_ERROR_IF(GetLastError() != ERROR_INSUFFICIENT_BUFFER); + + result.resize(decompressedBufferSize); + + SIZE_T decompressedDataSize = 0; + THROW_IF_WIN32_BOOL_FALSE(::Decompress(m_decompressor.get(), data.data(), data.size(), &result[0], result.size(), &decompressedDataSize)); + + result.resize(decompressedDataSize); + } + + return result; + } + + void Decompressor::Reset() + { + THROW_IF_WIN32_BOOL_FALSE(ResetDecompressor(m_decompressor.get())); + } + + void Decompressor::SetInformation(COMPRESS_INFORMATION_CLASS information, DWORD value) + { + THROW_IF_WIN32_BOOL_FALSE(SetDecompressorInformation(m_decompressor.get(), information, &value, sizeof(value))); + } + + DWORD Decompressor::GetInformation(COMPRESS_INFORMATION_CLASS information) + { + DWORD result = 0; + THROW_IF_WIN32_BOOL_FALSE(QueryDecompressorInformation(m_decompressor.get(), information, &result, sizeof(result))); + return result; + } +} diff --git a/src/AppInstallerSharedLib/Public/AppInstallerSHA256.h b/src/AppInstallerSharedLib/Public/AppInstallerSHA256.h @@ -47,6 +47,9 @@ namespace AppInstaller::Utility { // Computes the hash of the given buffer immediately. static HashBuffer ComputeHash(const uint8_t* buffer, std::uint32_t cbBuffer); + // Computes the hash of the given buffer immediately. + static HashBuffer ComputeHash(const std::vector<uint8_t>& buffer); + // Computes the hash of the given string immediately. static HashBuffer ComputeHash(std::string_view buffer); @@ -75,4 +78,4 @@ namespace AppInstaller::Utility { std::unique_ptr<SHA256Context, SHA256ContextDeleter> context; }; -}- \ No newline at end of file +} diff --git a/src/AppInstallerSharedLib/Public/winget/Compression.h b/src/AppInstallerSharedLib/Public/winget/Compression.h @@ -0,0 +1,54 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +#pragma once +#include <wil/resource.h> +#include <compressapi.h> +#include <vector> +#include <string_view> + +namespace AppInstaller::Compression +{ + // Contains a compressor from the Windows Compression API. + struct Compressor + { + // Create a compressor using the given algorithm (see COMPRESS_ALGORITHM_*) + Compressor(DWORD algorithm); + + // Compresses the given data. + std::vector<uint8_t> Compress(std::string_view data); + + // Resets the compressor. + void Reset(); + + // Sets compressor information values. + void SetInformation(COMPRESS_INFORMATION_CLASS information, DWORD value); + + // Gets compressor information values. + DWORD GetInformation(COMPRESS_INFORMATION_CLASS information); + + private: + wil::unique_any<COMPRESSOR_HANDLE, decltype(CloseCompressor), CloseCompressor> m_compressor; + }; + + // Contains a decompressor from the Windows Compression API. + struct Decompressor + { + // Create a decompressor using the given algorithm (see COMPRESS_ALGORITHM_*) + Decompressor(DWORD algorithm); + + // Decompresses the given data. + std::vector<uint8_t> Decompress(const std::vector<uint8_t>& data); + + // Resets the decompressor. + void Reset(); + + // Sets decompressor information values. + void SetInformation(COMPRESS_INFORMATION_CLASS information, DWORD value); + + // Gets decompressor information values. + DWORD GetInformation(COMPRESS_INFORMATION_CLASS information); + + private: + wil::unique_any<DECOMPRESSOR_HANDLE, decltype(CloseDecompressor), CloseDecompressor> m_decompressor; + }; +} diff --git a/src/AppInstallerSharedLib/Public/winget/SQLiteStatementBuilder.h b/src/AppInstallerSharedLib/Public/winget/SQLiteStatementBuilder.h @@ -114,6 +114,23 @@ namespace AppInstaller::SQLite::Builder Integer, // Type for specifying a primary key column as a row id alias. }; + template <Type type> + struct TypeInfo + { + }; + + template <> + struct TypeInfo<Type::Text> + { + using value_t = std::string; + }; + + template <> + struct TypeInfo<Type::Blob> + { + using value_t = SQLite::blob_t; + }; + // Aggregate functions. enum class Aggregate { @@ -253,6 +270,22 @@ namespace AppInstaller::SQLite::Builder StatementBuilder& Equals(std::nullptr_t); StatementBuilder& Equals(); + template <typename ValueType> + StatementBuilder& IsGreaterThan(const ValueType& value) + { + AddBindFunctor(AppendOpAndBinder(Op::GreaterThan), value); + return *this; + } + StatementBuilder& IsGreaterThan(details::unbound_t, std::optional<size_t> index = {}); + + template <typename ValueType> + StatementBuilder& IsGreaterThanOrEqualTo(const ValueType& value) + { + AddBindFunctor(AppendOpAndBinder(Op::GreaterThanOrEqualTo), value); + return *this; + } + StatementBuilder& IsGreaterThanOrEqualTo(details::unbound_t, std::optional<size_t> index = {}); + StatementBuilder& LikeWithEscape(std::string_view value); StatementBuilder& Like(details::unbound_t); @@ -311,6 +344,12 @@ namespace AppInstaller::SQLite::Builder StatementBuilder& InsertInto(QualifiedTable table); StatementBuilder& InsertInto(std::initializer_list<std::string_view> table); + // Begin an insert or ignore statement for the given table. + // The initializer_list form enables the table name to be constructed from multiple parts. + StatementBuilder& InsertOrIgnore(std::string_view table); + StatementBuilder& InsertOrIgnore(QualifiedTable table); + StatementBuilder& InsertOrIgnore(std::initializer_list<std::string_view> table); + // Set the columns for a statement (typically insert). StatementBuilder& Columns(std::string_view column); StatementBuilder& Columns(std::initializer_list<std::string_view> columns); @@ -362,12 +401,18 @@ namespace AppInstaller::SQLite::Builder // Complete an alter table statement by adding a column. StatementBuilder& Add(std::string_view column, Type type); - // Begin an table deletion statement. + // Begin a table deletion statement. // The initializer_list form enables the table name to be constructed from multiple parts. StatementBuilder& DropTable(std::string_view table); StatementBuilder& DropTable(QualifiedTable table); StatementBuilder& DropTable(std::initializer_list<std::string_view> table); + // Begin a table deletion statement. + // The initializer_list form enables the table name to be constructed from multiple parts. + StatementBuilder& DropTableIfExists(std::string_view table); + StatementBuilder& DropTableIfExists(QualifiedTable table); + StatementBuilder& DropTableIfExists(std::initializer_list<std::string_view> table); + // Begin an index creation statement. // The initializer_list form enables the index name to be constructed from multiple parts. StatementBuilder& CreateIndex(std::string_view table); @@ -441,6 +486,8 @@ namespace AppInstaller::SQLite::Builder Like, Escape, Literal, + GreaterThan, + GreaterThanOrEqualTo, }; // Appends given the operation. diff --git a/src/AppInstallerSharedLib/SHA256.cpp b/src/AppInstallerSharedLib/SHA256.cpp @@ -109,6 +109,12 @@ namespace AppInstaller::Utility { return hasher.Get(); } + SHA256::HashBuffer SHA256::ComputeHash(const std::vector<uint8_t>& buffer) + { + THROW_HR_IF(HRESULT_FROM_WIN32(ERROR_INSUFFICIENT_BUFFER), buffer.size() > std::numeric_limits<uint32_t>::max()); + return ComputeHash(buffer.data(), static_cast<uint32_t>(buffer.size())); + } + SHA256::HashBuffer SHA256::ComputeHash(std::string_view buffer) { return ComputeHash(reinterpret_cast<const std::uint8_t*>(buffer.data()), static_cast<std::uint32_t>(buffer.size())); @@ -171,4 +177,4 @@ namespace AppInstaller::Utility { THROW_HR_MSG(E_UNEXPECTED, "The hash is already finished"); } } -}- \ No newline at end of file +} diff --git a/src/AppInstallerSharedLib/SQLiteMetadataTable.cpp b/src/AppInstallerSharedLib/SQLiteMetadataTable.cpp @@ -16,7 +16,7 @@ namespace AppInstaller::SQLite static constexpr std::string_view s_MetadataTable_Table_Create = R"( CREATE TABLE [metadata]( [name] TEXT PRIMARY KEY NOT NULL, - [value] TEXT NOT NULL) + [value] TEXT NOT NULL) WITHOUT ROWID )"sv; // Statements diff --git a/src/AppInstallerSharedLib/SQLiteStatementBuilder.cpp b/src/AppInstallerSharedLib/SQLiteStatementBuilder.cpp @@ -356,6 +356,18 @@ namespace AppInstaller::SQLite::Builder return *this; } + StatementBuilder& StatementBuilder::IsGreaterThan(details::unbound_t, std::optional<size_t> index) + { + AppendOpAndBinder(Op::GreaterThan, index); + return *this; + } + + StatementBuilder& StatementBuilder::IsGreaterThanOrEqualTo(details::unbound_t, std::optional<size_t> index) + { + AppendOpAndBinder(Op::GreaterThanOrEqualTo, index); + return *this; + } + StatementBuilder& StatementBuilder::LikeWithEscape(std::string_view value) { AddBindFunctor(AppendOpAndBinder(Op::Like), EscapeStringForLike(value)); @@ -537,6 +549,24 @@ namespace AppInstaller::SQLite::Builder return *this; } + StatementBuilder& StatementBuilder::InsertOrIgnore(std::string_view table) + { + OutputOperationAndTable(m_stream, "INSERT OR IGNORE INTO", table); + return *this; + } + + StatementBuilder& StatementBuilder::InsertOrIgnore(QualifiedTable table) + { + OutputOperationAndTable(m_stream, "INSERT OR IGNORE INTO", table); + return *this; + } + + StatementBuilder& StatementBuilder::InsertOrIgnore(std::initializer_list<std::string_view> table) + { + OutputOperationAndTable(m_stream, "INSERT OR IGNORE INTO", table); + return *this; + } + StatementBuilder& StatementBuilder::Columns(std::string_view column) { OutputColumns(m_stream, "(", column); @@ -716,6 +746,24 @@ namespace AppInstaller::SQLite::Builder return *this; } + StatementBuilder& StatementBuilder::DropTableIfExists(std::string_view table) + { + OutputOperationAndTable(m_stream, "DROP TABLE IF EXISTS", table); + return *this; + } + + StatementBuilder& StatementBuilder::DropTableIfExists(QualifiedTable table) + { + OutputOperationAndTable(m_stream, "DROP TABLE IF EXISTS", table); + return *this; + } + + StatementBuilder& StatementBuilder::DropTableIfExists(std::initializer_list<std::string_view> table) + { + OutputOperationAndTable(m_stream, "DROP TABLE IF EXISTS", table); + return *this; + } + StatementBuilder& StatementBuilder::CreateIndex(std::string_view table) { OutputOperationAndTable(m_stream, "CREATE INDEX", table); @@ -905,6 +953,12 @@ namespace AppInstaller::SQLite::Builder case Op::Literal: m_stream << " ?"; break; + case Op::GreaterThan: + m_stream << " > ?"; + break; + case Op::GreaterThanOrEqualTo: + m_stream << " >= ?"; + break; default: THROW_HR(E_UNEXPECTED); } diff --git a/src/AppInstallerSharedLib/Yaml.cpp b/src/AppInstallerSharedLib/Yaml.cpp @@ -407,6 +407,11 @@ namespace AppInstaller::YAML std::optional<int64_t> Node::try_as_dispatch(int64_t*) const { + if (m_scalar.empty()) + { + return {}; + } + const char* begin = m_scalar.c_str(); char* end = nullptr; errno = 0; diff --git a/src/AppInstallerSharedLib/pch.h b/src/AppInstallerSharedLib/pch.h @@ -1,4 +1,4 @@ -// Copyright (c) Microsoft Corporation. +// Copyright (c) Microsoft Corporation. // Licensed under the MIT License. #pragma once @@ -6,7 +6,8 @@ #include <Windows.h> #include <appmodel.h> #include <icu.h> -#include <sddl.h> +#include <sddl.h> +#include <compressapi.h> #define YAML_DECLARE_STATIC #include <yaml.h> @@ -56,4 +57,4 @@ #include <winrt/Windows.ApplicationModel.Resources.h> #include <winrt/Windows.Foundation.h> #include <winrt/Windows.Globalization.h> -#include <winrt/Windows.System.Profile.h>- \ No newline at end of file +#include <winrt/Windows.System.Profile.h> diff --git a/src/WinGetUtil/Exports.cpp b/src/WinGetUtil/Exports.cpp @@ -28,6 +28,17 @@ namespace { return potentiallyNullPath ? std::filesystem::path{ potentiallyNullPath } : std::filesystem::path{}; } + + SQLiteIndex::Property GetSQLiteIndexProperty(WinGetSQLiteIndexProperty property) + { + switch (property) + { + case WinGetSQLiteIndexProperty_PackageUpdateTrackingBaseTime: return SQLiteIndex::Property::PackageUpdateTrackingBaseTime; + case WinGetSQLiteIndexProperty_IntermediateFileOutputPath: return SQLiteIndex::Property::IntermediateFileOutputPath; + } + + THROW_HR(E_INVALIDARG); + } } extern "C" @@ -119,6 +130,34 @@ extern "C" } CATCH_RETURN() + WINGET_UTIL_API WinGetSQLiteIndexMigrate( + WINGET_SQLITE_INDEX_HANDLE index, + UINT32 majorVersion, + UINT32 minorVersion) try + { + THROW_HR_IF(E_INVALIDARG, !index); + + return reinterpret_cast<SQLiteIndex*>(index)->MigrateTo({ majorVersion, minorVersion }) ? S_OK : HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED); + } + CATCH_RETURN() + + + WINGET_UTIL_API WinGetSQLiteIndexSetProperty( + WINGET_SQLITE_INDEX_HANDLE index, + WinGetSQLiteIndexProperty property, + WINGET_STRING value) try + { + THROW_HR_IF(E_INVALIDARG, !index); + THROW_HR_IF(E_INVALIDARG, !value); + + std::string valueUtf8 = ConvertToUTF8(value); + + reinterpret_cast<SQLiteIndex*>(index)->SetProperty(GetSQLiteIndexProperty(property), valueUtf8); + + return S_OK; + } + CATCH_RETURN() + WINGET_UTIL_API WinGetSQLiteIndexAddManifest( WINGET_SQLITE_INDEX_HANDLE index, WINGET_STRING manifestPath, WINGET_STRING relativePath) try diff --git a/src/WinGetUtil/WinGetUtil.h b/src/WinGetUtil/WinGetUtil.h @@ -125,6 +125,24 @@ extern "C" WINGET_UTIL_API WinGetSQLiteIndexClose( WINGET_SQLITE_INDEX_HANDLE index); + // Migrates the index to the new version specified. + WINGET_UTIL_API WinGetSQLiteIndexMigrate( + WINGET_SQLITE_INDEX_HANDLE index, + UINT32 majorVersion, + UINT32 minorVersion); + + enum WinGetSQLiteIndexProperty + { + WinGetSQLiteIndexProperty_PackageUpdateTrackingBaseTime = 0, + WinGetSQLiteIndexProperty_IntermediateFileOutputPath = 1, + }; + + // Sets the given property on the index. + WINGET_UTIL_API WinGetSQLiteIndexSetProperty( + WINGET_SQLITE_INDEX_HANDLE index, + WinGetSQLiteIndexProperty property, + WINGET_STRING value); + // Adds the manifest at the repository relative path to the index. // If the function succeeds, the manifest has been added. WINGET_UTIL_API WinGetSQLiteIndexAddManifest( diff --git a/src/WinGetUtilInterop/Api/WinGetSQLiteIndex.cs b/src/WinGetUtilInterop/Api/WinGetSQLiteIndex.cs @@ -1,4 +1,4 @@ -// ----------------------------------------------------------------------------- +// ----------------------------------------------------------------------------- // <copyright file="WinGetSQLiteIndex.cs" company="Microsoft Corporation"> // Copyright (c) Microsoft Corporation. Licensed under the MIT License. // </copyright> @@ -26,6 +26,32 @@ namespace Microsoft.WinGetUtil.Api internal WinGetSQLiteIndex(IntPtr indexHandle) { this.indexHandle = indexHandle; + } + + /// <inheritdoc/> + public void MigrateTo(uint majorVersion, uint minorVersion) + { + try + { + WinGetSQLiteIndexMigrate(this.indexHandle, majorVersion, minorVersion); + } + catch (Exception e) + { + throw new WinGetSQLiteIndexException(e); + } + } + + /// <inheritdoc/> + public void SetProperty(SQLiteIndexProperty property, string value) + { + try + { + WinGetSQLiteIndexSetProperty(this.indexHandle, property, value); + } + catch (Exception e) + { + throw new WinGetSQLiteIndexException(e); + } } /// <inheritdoc/> @@ -136,6 +162,26 @@ namespace Microsoft.WinGetUtil.Api } /// <summary> + /// Migrates the index to the target version. + /// </summary> + /// <param name="index">Handle of the index.</param> + /// <param name="majorVersion">Major version.</param> + /// <param name="minorVersion">Minor version.</param> + /// <returns>HRESULT.</returns> + [DllImport(Constants.DllName, CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Unicode, PreserveSig = false)] + private static extern IntPtr WinGetSQLiteIndexMigrate(IntPtr index, uint majorVersion, uint minorVersion); + + /// <summary> + /// Sets a property on the index. + /// </summary> + /// <param name="index">Handle of the index.</param> + /// <param name="property">The property to set.</param> + /// <param name="value">The value to set.</param> + /// <returns>HRESULT.</returns> + [DllImport(Constants.DllName, CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Unicode, PreserveSig = false)] + private static extern IntPtr WinGetSQLiteIndexSetProperty(IntPtr index, SQLiteIndexProperty property, string value); + + /// <summary> /// Closes the index. /// </summary> /// <param name="index">Handle of the index.</param> diff --git a/src/WinGetUtilInterop/Interfaces/IWinGetSQLiteIndex.cs b/src/WinGetUtilInterop/Interfaces/IWinGetSQLiteIndex.cs @@ -1,55 +1,89 @@ -// ----------------------------------------------------------------------------- -// <copyright file="IWinGetSQLiteIndex.cs" company="Microsoft Corporation"> -// Copyright (c) Microsoft Corporation. Licensed under the MIT License. -// </copyright> -// ----------------------------------------------------------------------------- - -namespace Microsoft.WinGetUtil.Interfaces -{ - using System; - - /// <summary> - /// Interface for index operations. - /// </summary> - public interface IWinGetSQLiteIndex : IDisposable - { - /// <summary> - /// Adds manifest to index. - /// </summary> - /// <param name="manifestPath">Manifest to add.</param> - /// <param name="relativePath">Path of the manifest in the repository.</param> - void AddManifest(string manifestPath, string relativePath); - - /// <summary> - /// Updates manifest in the index. - /// </summary> - /// <param name="manifestPath">Path to manifest to modify.</param> - /// <param name="relativePath">Path of the manifest in the repository.</param> - /// <returns>True if index was modified.</returns> - bool UpdateManifest(string manifestPath, string relativePath); - - /// <summary> - /// Delete manifest from index. - /// </summary> - /// <param name="manifestPath">Path to manifest to modify.</param> - /// <param name="relativePath">Path of the manifest in the repository.</param> - void RemoveManifest(string manifestPath, string relativePath); - - /// <summary> - /// Wrapper for WinGetSQLiteIndexPrepareForPackaging. - /// </summary> - void PrepareForPackaging(); - - /// <summary> - /// Checks the index for consistency, ensuring that at a minimum all referenced rows actually exist. - /// </summary> - /// <returns>Is index consistent.</returns> - bool IsIndexConsistent(); - - /// <summary> - /// Gets the managed index handle. It is used in additional manifest validation that requires an index. - /// </summary> - /// <returns>The managed index handle.</returns> - IntPtr GetIndexHandle(); - } -} +// ----------------------------------------------------------------------------- +// <copyright file="IWinGetSQLiteIndex.cs" company="Microsoft Corporation"> +// Copyright (c) Microsoft Corporation. Licensed under the MIT License. +// </copyright> +// ----------------------------------------------------------------------------- + +namespace Microsoft.WinGetUtil.Interfaces +{ + using System; + + /// <summary> + /// The properties that can be set with IWinGetSQLiteIndex::SetProperty. + /// The values must match those in WinGetUtil.h. + /// </summary> + public enum SQLiteIndexProperty + { + /// <summary> + /// The base time to use for update tracking. The value is in the Unix epoch. + /// Set to an empty string to use the current time. + /// Set to 0 to force all files to be output. + /// </summary> + PackageUpdateTrackingBaseTime = 0, + + /// <summary> + /// The full path to a base directory where intermediate files will be output. + /// The path does not need to exist, and may not be created if no files need to be written. + /// </summary> + IntermediateFileOutputPath = 1, + } + + /// <summary> + /// Interface for index operations. + /// </summary> + public interface IWinGetSQLiteIndex : IDisposable + { + /// <summary> + /// Migrates the index to the given version. + /// </summary> + /// <param name="majorVersion">Major version.</param> + /// <param name="minorVersion">Minor version.</param> + void MigrateTo(uint majorVersion, uint minorVersion); + + /// <summary> + /// Sets the given property to the given value. + /// </summary> + /// <param name="property">The property to set.</param> + /// <param name="value">The value to set.</param> + void SetProperty(SQLiteIndexProperty property, string value); + + /// <summary> + /// Adds manifest to index. + /// </summary> + /// <param name="manifestPath">Manifest to add.</param> + /// <param name="relativePath">Path of the manifest in the repository.</param> + void AddManifest(string manifestPath, string relativePath); + + /// <summary> + /// Updates manifest in the index. + /// </summary> + /// <param name="manifestPath">Path to manifest to modify.</param> + /// <param name="relativePath">Path of the manifest in the repository.</param> + /// <returns>True if index was modified.</returns> + bool UpdateManifest(string manifestPath, string relativePath); + + /// <summary> + /// Delete manifest from index. + /// </summary> + /// <param name="manifestPath">Path to manifest to modify.</param> + /// <param name="relativePath">Path of the manifest in the repository.</param> + void RemoveManifest(string manifestPath, string relativePath); + + /// <summary> + /// Wrapper for WinGetSQLiteIndexPrepareForPackaging. + /// </summary> + void PrepareForPackaging(); + + /// <summary> + /// Checks the index for consistency, ensuring that at a minimum all referenced rows actually exist. + /// </summary> + /// <returns>Is index consistent.</returns> + bool IsIndexConsistent(); + + /// <summary> + /// Gets the managed index handle. It is used in additional manifest validation that requires an index. + /// </summary> + /// <returns>The managed index handle.</returns> + IntPtr GetIndexHandle(); + } +}