Initial commit
This commit is contained in:
3
.gitignore
vendored
Normal file
3
.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
*.tsbuildinfo
|
||||||
44
Plan.md
Normal file
44
Plan.md
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
# Obsidian Sync Plan
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Build a self-hosted sync system for an Obsidian vault that works across Windows, Linux, and Android. The system will use an Obsidian plugin for clients, a central sync server, and end-to-end encrypted note transfer. Conflicts should try automatic merge first, then fall back to user-visible conflict handling.
|
||||||
|
|
||||||
|
## Current Implementation Slice
|
||||||
|
|
||||||
|
1. Create a TypeScript monorepo for the plugin, server, and shared libraries.
|
||||||
|
2. Define shared sync contracts for device registration, pull, push, tombstones, and conflict responses.
|
||||||
|
3. Add a shared sync engine with AES-GCM encryption, hashing, and safe text merge helpers.
|
||||||
|
4. Build a durable server that registers devices, stores encrypted note revisions on disk, and writes structured request and client-sync logs.
|
||||||
|
5. Build an Obsidian plugin with settings, manual sync, scheduled sync, remote pull, local push, attachment sync, conflict marker handling, and sync-run diagnostics upload.
|
||||||
|
6. Validate the scaffold with a workspace build and focused tests.
|
||||||
|
|
||||||
|
## Delivery Phases
|
||||||
|
|
||||||
|
### Phase 1
|
||||||
|
|
||||||
|
- Monorepo setup
|
||||||
|
- Protocol definitions
|
||||||
|
- Shared crypto and merge helpers
|
||||||
|
- Minimal server API
|
||||||
|
- Plugin settings and sync loop
|
||||||
|
|
||||||
|
### Phase 2
|
||||||
|
|
||||||
|
- Durable storage backend for the server
|
||||||
|
- Attachment sync
|
||||||
|
- Better device onboarding and recovery flow
|
||||||
|
- Structured logs and user-visible sync diagnostics
|
||||||
|
|
||||||
|
### Phase 3
|
||||||
|
|
||||||
|
- Hardening for large vaults and flaky networks
|
||||||
|
- Packaging and deployment docs
|
||||||
|
- Cross-device integration testing
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The server now persists encrypted sync state to a local data directory and keeps JSON-line diagnostics logs.
|
||||||
|
- Text conflicts are handled on the client and binary conflicts produce a remote conflict copy.
|
||||||
|
- Recovery bundle onboarding, device revocation, key rotation, and batched sync are implemented in the current repo.
|
||||||
|
- The next highest-priority gap is broader multi-instance Obsidian integration testing and a production-grade database-backed server store.
|
||||||
97
README.md
Normal file
97
README.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
# Obsidian Sync
|
||||||
|
|
||||||
|
Obsidian Sync is a self-hosted, end-to-end encrypted vault sync system for Obsidian across Windows, Linux, and Android. It consists of an Obsidian plugin, a sync server, and shared protocol and crypto packages inside a TypeScript monorepo.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- End-to-end encrypted text note and binary attachment sync
|
||||||
|
- Automatic text merge with conflict markers when both sides change
|
||||||
|
- Binary conflict copies when attachments diverge
|
||||||
|
- Durable server state stored on disk
|
||||||
|
- Structured JSON-line diagnostics and client sync log upload
|
||||||
|
- Recovery bundle export and import for onboarding new devices
|
||||||
|
- Device listing and revocation
|
||||||
|
- Vault key rotation with recovery-bundle refresh
|
||||||
|
- Paged pull and batched push behavior so interrupted sync runs can resume safely
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
- `apps/obsidian-plugin`: Obsidian plugin entry point, settings UI, and sync orchestration
|
||||||
|
- `apps/sync-server`: Express server for encrypted revision exchange and device management
|
||||||
|
- `packages/sync-protocol`: Shared request and response schemas
|
||||||
|
- `packages/sync-engine`: Shared crypto, hashing, and merge helpers
|
||||||
|
- `docs`: Architecture and deployment notes
|
||||||
|
- `tests`: Focused contract and store-level integration tests
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Node.js 22 or newer
|
||||||
|
- npm 10 or newer
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Run The Server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run dev:server
|
||||||
|
```
|
||||||
|
|
||||||
|
The default server URL is `http://localhost:8787`.
|
||||||
|
|
||||||
|
## First-Time Setup
|
||||||
|
|
||||||
|
1. Start the sync server.
|
||||||
|
2. Load the plugin into an Obsidian vault.
|
||||||
|
3. Set the server URL, vault ID, and device name in the plugin settings.
|
||||||
|
4. Run a manual sync to register the first device and generate the initial vault key.
|
||||||
|
5. Export a recovery bundle from the settings tab and store it securely.
|
||||||
|
|
||||||
|
## Add Another Device
|
||||||
|
|
||||||
|
1. Install the plugin on the second device.
|
||||||
|
2. Open the plugin settings and paste the recovery bundle into the import field.
|
||||||
|
3. Run a manual sync to register that device and pull the encrypted vault contents.
|
||||||
|
|
||||||
|
## Rotate The Vault Key
|
||||||
|
|
||||||
|
1. Open the plugin settings and use `Rotate key`.
|
||||||
|
2. Wait for the client to re-encrypt and upload the current vault contents.
|
||||||
|
3. Export the fresh recovery bundle.
|
||||||
|
4. Import that new bundle on every other device before the next sync.
|
||||||
|
|
||||||
|
## Revoke A Device
|
||||||
|
|
||||||
|
Use the plugin settings device list to revoke a device that should no longer have access. A revoked device can no longer authenticate to the server.
|
||||||
|
|
||||||
|
## Server Data
|
||||||
|
|
||||||
|
- `data/sync-state.json`: durable revision, device, tombstone, and key-status state
|
||||||
|
- `data/logs/server.jsonl`: structured server and client-sync diagnostics
|
||||||
|
|
||||||
|
## Current Limitations
|
||||||
|
|
||||||
|
- The server storage backend is file-based rather than SQLite or PostgreSQL.
|
||||||
|
- Sync validation is focused on shared libraries and the server store; there is not yet a full Obsidian plugin integration harness.
|
||||||
|
- Device onboarding is driven by a copy-and-paste recovery bundle instead of a richer pairing flow.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- `docs/architecture.md`
|
||||||
|
- `docs/deployment.md`
|
||||||
|
- `Plan.md`
|
||||||
2
Specsheet.md
Normal file
2
Specsheet.md
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
# Description
|
||||||
|
I need a system that syncs my obisian vault between different devices and is connected by a server. For this application, I will need a client side plugin for obisdian which works on both windows, linux, and android obsidian. I will also need a server application that will allow me to sync my notes to it and distribute it out to other devices connected. This system will require a version control system that allow the user to edit it on different devices and sort out the different edits. If there is an error create a log and alert the user.
|
||||||
9
apps/obsidian-plugin/manifest.json
Normal file
9
apps/obsidian-plugin/manifest.json
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"id": "obsidian-sync",
|
||||||
|
"name": "Obsidian Sync",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"minAppVersion": "1.5.0",
|
||||||
|
"description": "Sync a vault through a self-hosted encrypted sync server.",
|
||||||
|
"author": "Local",
|
||||||
|
"isDesktopOnly": false
|
||||||
|
}
|
||||||
18
apps/obsidian-plugin/package.json
Normal file
18
apps/obsidian-plugin/package.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"name": "@obsidian-sync/obsidian-plugin",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"main": "dist/main.js",
|
||||||
|
"types": "dist/main.d.ts",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc -b",
|
||||||
|
"typecheck": "tsc -b --pretty false"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@obsidian-sync/sync-engine": "0.1.0",
|
||||||
|
"@obsidian-sync/sync-protocol": "0.1.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"obsidian": "^1.8.7"
|
||||||
|
}
|
||||||
|
}
|
||||||
96
apps/obsidian-plugin/src/main.ts
Normal file
96
apps/obsidian-plugin/src/main.ts
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
import { Notice, Plugin } from "obsidian";
|
||||||
|
|
||||||
|
import { SyncSettingsTab } from "./settings/SyncSettingsTab";
|
||||||
|
import { DEFAULT_SETTINGS, type ObsidianSyncSettings } from "./settings/settings";
|
||||||
|
import { SyncService } from "./sync/SyncService";
|
||||||
|
|
||||||
|
export default class ObsidianSyncPlugin extends Plugin {
|
||||||
|
settings: ObsidianSyncSettings = DEFAULT_SETTINGS;
|
||||||
|
|
||||||
|
private syncService?: SyncService;
|
||||||
|
private scheduledSyncHandle?: number;
|
||||||
|
|
||||||
|
async onload(): Promise<void> {
|
||||||
|
await this.loadSettings();
|
||||||
|
|
||||||
|
this.syncService = new SyncService(
|
||||||
|
this.app,
|
||||||
|
() => this.settings,
|
||||||
|
async (settings) => {
|
||||||
|
this.settings = settings;
|
||||||
|
await this.saveSettings();
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
this.addSettingTab(new SyncSettingsTab(this.app, this, () => this.configureSyncSchedule()));
|
||||||
|
|
||||||
|
this.addCommand({
|
||||||
|
id: "obsidian-sync-run-manual",
|
||||||
|
name: "Run encrypted sync now",
|
||||||
|
callback: async () => {
|
||||||
|
await this.syncService?.runManualSync();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
this.addCommand({
|
||||||
|
id: "obsidian-sync-rotate-vault-key",
|
||||||
|
name: "Rotate vault key",
|
||||||
|
callback: async () => {
|
||||||
|
const result = await this.getSyncServiceOrThrow().rotateVaultKey();
|
||||||
|
new Notice(
|
||||||
|
`Vault key rotated. Uploaded ${result.uploadedFiles} files. Export the fresh recovery bundle to update other devices.`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
this.configureSyncSchedule();
|
||||||
|
new Notice("Obsidian Sync loaded.");
|
||||||
|
}
|
||||||
|
|
||||||
|
onunload(): void {
|
||||||
|
if (this.scheduledSyncHandle !== undefined) {
|
||||||
|
window.clearInterval(this.scheduledSyncHandle);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async loadSettings(): Promise<void> {
|
||||||
|
const loaded = (await this.loadData()) as Partial<ObsidianSyncSettings> | null;
|
||||||
|
this.settings = {
|
||||||
|
...DEFAULT_SETTINGS,
|
||||||
|
...loaded,
|
||||||
|
deviceName: loaded?.deviceName || this.app.vault.getName(),
|
||||||
|
syncState: {
|
||||||
|
...DEFAULT_SETTINGS.syncState,
|
||||||
|
...(loaded?.syncState ?? {}),
|
||||||
|
files: {
|
||||||
|
...DEFAULT_SETTINGS.syncState.files,
|
||||||
|
...(loaded?.syncState?.files ?? {})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
await this.saveSettings();
|
||||||
|
}
|
||||||
|
|
||||||
|
async saveSettings(): Promise<void> {
|
||||||
|
await this.saveData(this.settings);
|
||||||
|
}
|
||||||
|
|
||||||
|
getSyncServiceOrThrow(): SyncService {
|
||||||
|
if (!this.syncService) {
|
||||||
|
throw new Error("Sync service is not ready yet.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return this.syncService;
|
||||||
|
}
|
||||||
|
|
||||||
|
private configureSyncSchedule(): void {
|
||||||
|
if (this.scheduledSyncHandle !== undefined) {
|
||||||
|
window.clearInterval(this.scheduledSyncHandle);
|
||||||
|
}
|
||||||
|
|
||||||
|
const intervalMinutes = Math.max(1, this.settings.syncIntervalMinutes);
|
||||||
|
this.scheduledSyncHandle = window.setInterval(() => {
|
||||||
|
void this.syncService?.runScheduledSync();
|
||||||
|
}, intervalMinutes * 60_000);
|
||||||
|
}
|
||||||
|
}
|
||||||
228
apps/obsidian-plugin/src/settings/SyncSettingsTab.ts
Normal file
228
apps/obsidian-plugin/src/settings/SyncSettingsTab.ts
Normal file
@@ -0,0 +1,228 @@
|
|||||||
|
import { Notice, PluginSettingTab, Setting, TextAreaComponent } from "obsidian";
|
||||||
|
|
||||||
|
import type ObsidianSyncPlugin from "../main";
|
||||||
|
|
||||||
|
export class SyncSettingsTab extends PluginSettingTab {
|
||||||
|
private exportedRecoveryBundle = "";
|
||||||
|
private importedRecoveryBundle = "";
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
app: ObsidianSyncPlugin["app"],
|
||||||
|
private readonly plugin: ObsidianSyncPlugin,
|
||||||
|
private readonly onSettingsUpdated: () => void
|
||||||
|
) {
|
||||||
|
super(app, plugin);
|
||||||
|
}
|
||||||
|
|
||||||
|
display(): void {
|
||||||
|
const { containerEl } = this;
|
||||||
|
containerEl.empty();
|
||||||
|
|
||||||
|
containerEl.createEl("h2", { text: "Obsidian Sync" });
|
||||||
|
const syncService = this.plugin.getSyncServiceOrThrow();
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Server URL")
|
||||||
|
.setDesc("Base URL of the sync server.")
|
||||||
|
.addText((text) =>
|
||||||
|
text
|
||||||
|
.setPlaceholder("http://localhost:8787")
|
||||||
|
.setValue(this.plugin.settings.serverUrl)
|
||||||
|
.onChange(async (value) => {
|
||||||
|
this.plugin.settings.serverUrl = value.trim();
|
||||||
|
await this.plugin.saveSettings();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Vault ID")
|
||||||
|
.setDesc("Stable identifier used to group devices for the same vault.")
|
||||||
|
.addText((text) =>
|
||||||
|
text.setValue(this.plugin.settings.vaultId).onChange(async (value) => {
|
||||||
|
this.plugin.settings.vaultId = value.trim();
|
||||||
|
await this.plugin.saveSettings();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Device name")
|
||||||
|
.setDesc("Human-readable name shown on the server.")
|
||||||
|
.addText((text) =>
|
||||||
|
text.setValue(this.plugin.settings.deviceName).onChange(async (value) => {
|
||||||
|
this.plugin.settings.deviceName = value.trim();
|
||||||
|
await this.plugin.saveSettings();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Sync interval")
|
||||||
|
.setDesc("Minutes between scheduled sync runs.")
|
||||||
|
.addText((text) =>
|
||||||
|
text.setValue(String(this.plugin.settings.syncIntervalMinutes)).onChange(async (value) => {
|
||||||
|
const minutes = Number(value);
|
||||||
|
this.plugin.settings.syncIntervalMinutes = Number.isFinite(minutes) && minutes > 0 ? minutes : 5;
|
||||||
|
await this.plugin.saveSettings();
|
||||||
|
this.onSettingsUpdated();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Pull batch size")
|
||||||
|
.setDesc("Maximum remote changes to request in a single pull page.")
|
||||||
|
.addText((text) =>
|
||||||
|
text.setValue(String(this.plugin.settings.pullBatchSize)).onChange(async (value) => {
|
||||||
|
const batchSize = Number(value);
|
||||||
|
this.plugin.settings.pullBatchSize = Number.isFinite(batchSize) && batchSize > 0 ? Math.floor(batchSize) : 50;
|
||||||
|
await this.plugin.saveSettings();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Push batch size")
|
||||||
|
.setDesc("Maximum local changes to upload in a single push request.")
|
||||||
|
.addText((text) =>
|
||||||
|
text.setValue(String(this.plugin.settings.pushBatchSize)).onChange(async (value) => {
|
||||||
|
const batchSize = Number(value);
|
||||||
|
this.plugin.settings.pushBatchSize = Number.isFinite(batchSize) && batchSize > 0 ? Math.floor(batchSize) : 50;
|
||||||
|
await this.plugin.saveSettings();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
containerEl.createEl("h3", { text: "Recovery" });
|
||||||
|
|
||||||
|
let exportBundleTextArea: TextAreaComponent | undefined;
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Export recovery bundle")
|
||||||
|
.setDesc("Generate a bundle you can paste into another device to recover the same encrypted vault.")
|
||||||
|
.addTextArea((textArea) => {
|
||||||
|
exportBundleTextArea = textArea;
|
||||||
|
textArea.setValue(this.exportedRecoveryBundle);
|
||||||
|
textArea.inputEl.rows = 8;
|
||||||
|
textArea.inputEl.cols = 40;
|
||||||
|
})
|
||||||
|
.addButton((button) =>
|
||||||
|
button.setButtonText("Generate").onClick(async () => {
|
||||||
|
this.exportedRecoveryBundle = await syncService.exportRecoveryBundle();
|
||||||
|
exportBundleTextArea?.setValue(this.exportedRecoveryBundle);
|
||||||
|
new Notice("Recovery bundle generated.");
|
||||||
|
})
|
||||||
|
)
|
||||||
|
.addButton((button) =>
|
||||||
|
button.setButtonText("Clear").onClick(() => {
|
||||||
|
this.exportedRecoveryBundle = "";
|
||||||
|
exportBundleTextArea?.setValue("");
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
let importBundleTextArea: TextAreaComponent | undefined;
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Import recovery bundle")
|
||||||
|
.setDesc("Paste a bundle from another device to register this device against the same encrypted vault.")
|
||||||
|
.addTextArea((textArea) => {
|
||||||
|
importBundleTextArea = textArea;
|
||||||
|
textArea.setValue(this.importedRecoveryBundle);
|
||||||
|
textArea.inputEl.rows = 8;
|
||||||
|
textArea.inputEl.cols = 40;
|
||||||
|
textArea.onChange((value) => {
|
||||||
|
this.importedRecoveryBundle = value;
|
||||||
|
});
|
||||||
|
})
|
||||||
|
.addButton((button) =>
|
||||||
|
button.setButtonText("Import").onClick(async () => {
|
||||||
|
await syncService.importRecoveryBundle(this.importedRecoveryBundle);
|
||||||
|
this.importedRecoveryBundle = "";
|
||||||
|
this.exportedRecoveryBundle = "";
|
||||||
|
importBundleTextArea?.setValue("");
|
||||||
|
new Notice("Recovery bundle imported. Run sync to register this device.");
|
||||||
|
this.display();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
containerEl.createEl("h3", { text: "Devices" });
|
||||||
|
const deviceListContainer = containerEl.createDiv();
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Connected devices")
|
||||||
|
.setDesc("Refresh the active device list and revoke devices that should no longer have access.")
|
||||||
|
.addButton((button) =>
|
||||||
|
button.setButtonText("Refresh").onClick(async () => {
|
||||||
|
await this.renderDeviceList(deviceListContainer);
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
void this.renderDeviceList(deviceListContainer);
|
||||||
|
|
||||||
|
containerEl.createEl("h3", { text: "Key rotation" });
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName("Rotate vault key")
|
||||||
|
.setDesc("Generate a new vault key, re-encrypt local content, and require your other devices to import a fresh recovery bundle.")
|
||||||
|
.addButton((button) =>
|
||||||
|
button.setButtonText("Rotate key").onClick(async () => {
|
||||||
|
const result = await syncService.rotateVaultKey();
|
||||||
|
this.exportedRecoveryBundle = result.recoveryBundle;
|
||||||
|
exportBundleTextArea?.setValue(result.recoveryBundle);
|
||||||
|
new Notice(
|
||||||
|
`Vault key rotated. Uploaded ${result.uploadedFiles} files. Share the fresh recovery bundle with your other devices.`
|
||||||
|
);
|
||||||
|
await this.renderDeviceList(deviceListContainer);
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async renderDeviceList(containerEl: HTMLElement): Promise<void> {
|
||||||
|
containerEl.empty();
|
||||||
|
|
||||||
|
if (!this.plugin.settings.deviceId || !this.plugin.settings.authToken) {
|
||||||
|
containerEl.createEl("p", {
|
||||||
|
text: "Run a sync or import a recovery bundle to register this device before managing devices."
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const response = await this.plugin.getSyncServiceOrThrow().listDevices();
|
||||||
|
const summaryParts = [`Active server key: ${response.activeKeyId ?? "not set"}`];
|
||||||
|
if (response.keyRotatedAt) {
|
||||||
|
summaryParts.push(`Last key rotation: ${response.keyRotatedAt}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
containerEl.createEl("p", {
|
||||||
|
text: summaryParts.join(" | ")
|
||||||
|
});
|
||||||
|
|
||||||
|
for (const device of response.devices) {
|
||||||
|
const isCurrentDevice = device.deviceId === this.plugin.settings.deviceId;
|
||||||
|
const description = [
|
||||||
|
`ID: ${device.deviceId}`,
|
||||||
|
`Registered: ${device.issuedAt}`,
|
||||||
|
device.revokedAt ? `Revoked: ${device.revokedAt}` : "Active"
|
||||||
|
].join(" | ");
|
||||||
|
|
||||||
|
new Setting(containerEl)
|
||||||
|
.setName(`${device.deviceName}${isCurrentDevice ? " (this device)" : ""}`)
|
||||||
|
.setDesc(description)
|
||||||
|
.addButton((button) => {
|
||||||
|
if (device.revokedAt) {
|
||||||
|
button.setButtonText("Revoked").setDisabled(true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
button.setButtonText(isCurrentDevice ? "Revoke this device" : "Revoke").onClick(async () => {
|
||||||
|
const selfRevoked = await this.plugin.getSyncServiceOrThrow().revokeDevice(device.deviceId);
|
||||||
|
new Notice(
|
||||||
|
selfRevoked
|
||||||
|
? "This device was revoked. Import a recovery bundle or run sync again to register a new device identity."
|
||||||
|
: "Device revoked."
|
||||||
|
);
|
||||||
|
await this.renderDeviceList(containerEl);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
containerEl.createEl("p", {
|
||||||
|
text: error instanceof Error ? `Failed to load devices: ${error.message}` : "Failed to load devices."
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
47
apps/obsidian-plugin/src/settings/settings.ts
Normal file
47
apps/obsidian-plugin/src/settings/settings.ts
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
import type { SyncFileKind } from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
export interface SyncedFileState {
|
||||||
|
kind: SyncFileKind;
|
||||||
|
revisionId: string;
|
||||||
|
contentHash: string;
|
||||||
|
lastSyncedContent?: string;
|
||||||
|
updatedAt: string;
|
||||||
|
pendingConflict?: boolean;
|
||||||
|
pendingConflictHash?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SyncState {
|
||||||
|
serverRevision: number;
|
||||||
|
files: Record<string, SyncedFileState>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ObsidianSyncSettings {
|
||||||
|
serverUrl: string;
|
||||||
|
vaultId: string;
|
||||||
|
deviceId: string;
|
||||||
|
deviceName: string;
|
||||||
|
authToken: string;
|
||||||
|
keyId: string;
|
||||||
|
exportedVaultKey: string;
|
||||||
|
syncIntervalMinutes: number;
|
||||||
|
pullBatchSize: number;
|
||||||
|
pushBatchSize: number;
|
||||||
|
syncState: SyncState;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DEFAULT_SETTINGS: ObsidianSyncSettings = {
|
||||||
|
serverUrl: "http://localhost:8787",
|
||||||
|
vaultId: "primary-vault",
|
||||||
|
deviceId: "",
|
||||||
|
deviceName: "",
|
||||||
|
authToken: "",
|
||||||
|
keyId: "",
|
||||||
|
exportedVaultKey: "",
|
||||||
|
syncIntervalMinutes: 5,
|
||||||
|
pullBatchSize: 50,
|
||||||
|
pushBatchSize: 50,
|
||||||
|
syncState: {
|
||||||
|
serverRevision: 0,
|
||||||
|
files: {}
|
||||||
|
}
|
||||||
|
};
|
||||||
91
apps/obsidian-plugin/src/sync/SyncRunLogger.ts
Normal file
91
apps/obsidian-plugin/src/sync/SyncRunLogger.ts
Normal file
@@ -0,0 +1,91 @@
|
|||||||
|
import type { ClientLogEntry, ClientLogUploadRequest } from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
type LogLevel = ClientLogEntry["level"];
|
||||||
|
|
||||||
|
function normalizeError(error: unknown): Record<string, unknown> {
|
||||||
|
if (error instanceof Error) {
|
||||||
|
return {
|
||||||
|
name: error.name,
|
||||||
|
message: error.message,
|
||||||
|
stack: error.stack
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
value: String(error)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export class SyncRunLogger {
|
||||||
|
private readonly runId = globalThis.crypto.randomUUID();
|
||||||
|
|
||||||
|
private readonly entries: ClientLogEntry[] = [];
|
||||||
|
|
||||||
|
private deviceId: string;
|
||||||
|
|
||||||
|
constructor(deviceId: string) {
|
||||||
|
this.deviceId = deviceId || "unregistered-device";
|
||||||
|
}
|
||||||
|
|
||||||
|
setDeviceId(deviceId: string): void {
|
||||||
|
if (deviceId) {
|
||||||
|
this.deviceId = deviceId;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
getRunId(): string {
|
||||||
|
return this.runId;
|
||||||
|
}
|
||||||
|
|
||||||
|
hasEntries(): boolean {
|
||||||
|
return this.entries.length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
toUploadRequest(vaultId: string): ClientLogUploadRequest {
|
||||||
|
return {
|
||||||
|
vaultId,
|
||||||
|
deviceId: this.deviceId,
|
||||||
|
runId: this.runId,
|
||||||
|
entries: [...this.entries]
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
debug(message: string, context?: Record<string, unknown>): void {
|
||||||
|
this.log("debug", message, context);
|
||||||
|
}
|
||||||
|
|
||||||
|
info(message: string, context?: Record<string, unknown>): void {
|
||||||
|
this.log("info", message, context);
|
||||||
|
}
|
||||||
|
|
||||||
|
warn(message: string, context?: Record<string, unknown>): void {
|
||||||
|
this.log("warn", message, context);
|
||||||
|
}
|
||||||
|
|
||||||
|
error(message: string, error?: unknown, context?: Record<string, unknown>): void {
|
||||||
|
this.log("error", message, {
|
||||||
|
...context,
|
||||||
|
...(error === undefined ? {} : { error: normalizeError(error) })
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private log(level: LogLevel, message: string, context?: Record<string, unknown>): void {
|
||||||
|
const entry: ClientLogEntry = {
|
||||||
|
level,
|
||||||
|
message,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
deviceId: this.deviceId,
|
||||||
|
context: {
|
||||||
|
runId: this.runId,
|
||||||
|
...(context ?? {})
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
this.entries.push(entry);
|
||||||
|
|
||||||
|
const consoleMethod =
|
||||||
|
level === "debug" ? console.debug : level === "info" ? console.info : level === "warn" ? console.warn : console.error;
|
||||||
|
|
||||||
|
consoleMethod(`[Obsidian Sync][${this.runId}] ${message}`, entry.context ?? {});
|
||||||
|
}
|
||||||
|
}
|
||||||
984
apps/obsidian-plugin/src/sync/SyncService.ts
Normal file
984
apps/obsidian-plugin/src/sync/SyncService.ts
Normal file
@@ -0,0 +1,984 @@
|
|||||||
|
import { App, Notice, TFile, normalizePath } from "obsidian";
|
||||||
|
|
||||||
|
import {
|
||||||
|
computeBinaryHash,
|
||||||
|
computeTextHash,
|
||||||
|
createConflictCopyPath,
|
||||||
|
decryptBytes,
|
||||||
|
decryptText,
|
||||||
|
encryptBytes,
|
||||||
|
encryptText,
|
||||||
|
generateVaultKey,
|
||||||
|
importVaultKey,
|
||||||
|
mergeTextRevisions,
|
||||||
|
type VaultKeyHandle
|
||||||
|
} from "@obsidian-sync/sync-engine";
|
||||||
|
import {
|
||||||
|
ClientLogUploadResponseSchema,
|
||||||
|
ListDevicesResponseSchema,
|
||||||
|
RecoveryBundleSchema,
|
||||||
|
RegisterDeviceResponseSchema,
|
||||||
|
RevokeDeviceResponseSchema,
|
||||||
|
RotateVaultKeyResponseSchema,
|
||||||
|
SyncPullResponseSchema,
|
||||||
|
SyncPushResponseSchema,
|
||||||
|
type ListDevicesResponse,
|
||||||
|
type RecoveryBundle,
|
||||||
|
type SyncConflict,
|
||||||
|
type SyncFileKind,
|
||||||
|
type SyncFileRecord,
|
||||||
|
type SyncPullResponse,
|
||||||
|
type Tombstone
|
||||||
|
} from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
import type { ObsidianSyncSettings, SyncedFileState } from "../settings/settings";
|
||||||
|
import { SyncRunLogger } from "./SyncRunLogger";
|
||||||
|
|
||||||
|
type SettingsReader = () => ObsidianSyncSettings;
|
||||||
|
type SettingsWriter = (settings: ObsidianSyncSettings) => Promise<void>;
|
||||||
|
type SyncMode = "manual" | "scheduled";
|
||||||
|
|
||||||
|
const DEFAULT_BATCH_SIZE = 50;
|
||||||
|
const IGNORED_PATH_PREFIXES = [".obsidian/"];
|
||||||
|
const TEXT_FILE_EXTENSIONS = new Set(["md", "txt", "markdown", "canvas", "json", "yaml", "yml", "csv"]);
|
||||||
|
|
||||||
|
interface CollectLocalChangeOptions {
|
||||||
|
forceAllFiles?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface LocalChangeSet {
|
||||||
|
files: SyncFileRecord[];
|
||||||
|
tombstones: Tombstone[];
|
||||||
|
statesByPath: Record<string, SyncedFileState>;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface LocalFileSnapshot {
|
||||||
|
kind: SyncFileKind;
|
||||||
|
contentHash: string;
|
||||||
|
updatedAt: string;
|
||||||
|
sizeBytes: number;
|
||||||
|
textContent?: string;
|
||||||
|
binaryContent?: ArrayBuffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PullSummary {
|
||||||
|
pulledChanges: number;
|
||||||
|
finalServerRevision: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PushSummary {
|
||||||
|
acceptedServerRevision: number;
|
||||||
|
acceptedFilePaths: string[];
|
||||||
|
acceptedTombstones: string[];
|
||||||
|
conflicts: SyncConflict[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RotateVaultKeyResult {
|
||||||
|
recoveryBundle: string;
|
||||||
|
activeKeyId: string;
|
||||||
|
rotatedAt: string;
|
||||||
|
uploadedFiles: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class SyncService {
|
||||||
|
constructor(
|
||||||
|
private readonly app: App,
|
||||||
|
private readonly readSettings: SettingsReader,
|
||||||
|
private readonly writeSettings: SettingsWriter
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async runManualSync(): Promise<void> {
|
||||||
|
await this.runSync("manual");
|
||||||
|
}
|
||||||
|
|
||||||
|
async runScheduledSync(): Promise<void> {
|
||||||
|
await this.runSync("scheduled");
|
||||||
|
}
|
||||||
|
|
||||||
|
async exportRecoveryBundle(): Promise<string> {
|
||||||
|
await this.ensureVaultKey();
|
||||||
|
const settings = this.readSettings();
|
||||||
|
return this.serializeRecoveryBundle(settings);
|
||||||
|
}
|
||||||
|
|
||||||
|
async importRecoveryBundle(serializedBundle: string): Promise<void> {
|
||||||
|
const parsed = RecoveryBundleSchema.parse(JSON.parse(serializedBundle) as RecoveryBundle);
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
|
||||||
|
nextSettings.serverUrl = parsed.serverUrl;
|
||||||
|
nextSettings.vaultId = parsed.vaultId;
|
||||||
|
nextSettings.keyId = parsed.keyId;
|
||||||
|
nextSettings.exportedVaultKey = parsed.exportedVaultKey;
|
||||||
|
nextSettings.deviceId = "";
|
||||||
|
nextSettings.authToken = "";
|
||||||
|
nextSettings.syncState = {
|
||||||
|
serverRevision: 0,
|
||||||
|
files: {}
|
||||||
|
};
|
||||||
|
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
}
|
||||||
|
|
||||||
|
async listDevices(): Promise<ListDevicesResponse> {
|
||||||
|
await this.ensureRegisteredDevice();
|
||||||
|
const settings = this.readSettings();
|
||||||
|
return this.postJson(
|
||||||
|
"/api/devices/list",
|
||||||
|
{
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
deviceId: settings.deviceId
|
||||||
|
},
|
||||||
|
ListDevicesResponseSchema,
|
||||||
|
settings.authToken
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async revokeDevice(targetDeviceId: string): Promise<boolean> {
|
||||||
|
await this.ensureRegisteredDevice();
|
||||||
|
const settings = this.readSettings();
|
||||||
|
|
||||||
|
await this.postJson(
|
||||||
|
"/api/devices/revoke",
|
||||||
|
{
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
deviceId: settings.deviceId,
|
||||||
|
targetDeviceId
|
||||||
|
},
|
||||||
|
RevokeDeviceResponseSchema,
|
||||||
|
settings.authToken
|
||||||
|
);
|
||||||
|
|
||||||
|
if (targetDeviceId === settings.deviceId) {
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
nextSettings.deviceId = "";
|
||||||
|
nextSettings.authToken = "";
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
async rotateVaultKey(): Promise<RotateVaultKeyResult> {
|
||||||
|
const diagnostics = new SyncRunLogger(this.readSettings().deviceId);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const currentVaultKey = await this.ensureVaultKey();
|
||||||
|
diagnostics.info("vault-key-ready", {
|
||||||
|
keyId: currentVaultKey.keyId
|
||||||
|
});
|
||||||
|
|
||||||
|
await this.ensureDeviceRegistration(diagnostics);
|
||||||
|
diagnostics.setDeviceId(this.readSettings().deviceId);
|
||||||
|
|
||||||
|
const generatedKey = await generateVaultKey();
|
||||||
|
const localChanges = await this.collectLocalChanges(
|
||||||
|
{
|
||||||
|
keyId: generatedKey.keyId,
|
||||||
|
key: generatedKey.key
|
||||||
|
},
|
||||||
|
diagnostics,
|
||||||
|
{
|
||||||
|
forceAllFiles: true
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
const pushSummary = await this.pushLocalChanges(localChanges, diagnostics);
|
||||||
|
const settings = this.readSettings();
|
||||||
|
const rotated = await this.postJson(
|
||||||
|
"/api/keys/rotate",
|
||||||
|
{
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
deviceId: settings.deviceId,
|
||||||
|
nextKeyId: generatedKey.keyId,
|
||||||
|
previousKeyId: settings.keyId || undefined
|
||||||
|
},
|
||||||
|
RotateVaultKeyResponseSchema,
|
||||||
|
settings.authToken
|
||||||
|
);
|
||||||
|
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
nextSettings.keyId = generatedKey.keyId;
|
||||||
|
nextSettings.exportedVaultKey = generatedKey.exportedKey;
|
||||||
|
nextSettings.syncState.serverRevision = Math.max(nextSettings.syncState.serverRevision, pushSummary.acceptedServerRevision);
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
|
||||||
|
diagnostics.info("vault-key-rotated", {
|
||||||
|
activeKeyId: rotated.activeKeyId,
|
||||||
|
rotatedAt: rotated.rotatedAt,
|
||||||
|
uploadedFiles: pushSummary.acceptedFilePaths.length
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
recoveryBundle: this.serializeRecoveryBundle(nextSettings),
|
||||||
|
activeKeyId: rotated.activeKeyId,
|
||||||
|
rotatedAt: rotated.rotatedAt,
|
||||||
|
uploadedFiles: pushSummary.acceptedFilePaths.length
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
diagnostics.error("vault-key-rotation-failed", error);
|
||||||
|
throw error;
|
||||||
|
} finally {
|
||||||
|
await this.uploadDiagnostics(diagnostics);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async runSync(mode: SyncMode): Promise<void> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
const diagnostics = new SyncRunLogger(settings.deviceId);
|
||||||
|
if (!settings.serverUrl || !settings.vaultId) {
|
||||||
|
if (mode === "manual") {
|
||||||
|
new Notice("Set a server URL and vault ID before syncing.");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
diagnostics.info("sync-started", {
|
||||||
|
mode,
|
||||||
|
serverRevision: settings.syncState.serverRevision
|
||||||
|
});
|
||||||
|
|
||||||
|
try {
|
||||||
|
const vaultKey = await this.ensureVaultKey();
|
||||||
|
diagnostics.info("vault-key-ready", {
|
||||||
|
keyId: vaultKey.keyId
|
||||||
|
});
|
||||||
|
|
||||||
|
await this.ensureDeviceRegistration(diagnostics);
|
||||||
|
diagnostics.setDeviceId(this.readSettings().deviceId);
|
||||||
|
|
||||||
|
const pullSummary = await this.pullAndApplyRemoteChanges(vaultKey, diagnostics);
|
||||||
|
const localChanges = await this.collectLocalChanges(vaultKey, diagnostics);
|
||||||
|
const pushSummary = await this.pushLocalChanges(localChanges, diagnostics);
|
||||||
|
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
nextSettings.syncState.serverRevision = Math.max(pullSummary.finalServerRevision, pushSummary.acceptedServerRevision);
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
|
||||||
|
if (mode === "manual") {
|
||||||
|
new Notice(
|
||||||
|
`Sync complete. Pulled ${pullSummary.pulledChanges} changes, uploaded ${pushSummary.acceptedFilePaths.length} files.`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (pushSummary.conflicts.length > 0) {
|
||||||
|
diagnostics.warn("server-conflicts-reported", {
|
||||||
|
count: pushSummary.conflicts.length,
|
||||||
|
paths: pushSummary.conflicts.map((conflict) => conflict.path)
|
||||||
|
});
|
||||||
|
new Notice(`${pushSummary.conflicts.length} server conflicts need review.`);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
diagnostics.error("sync-failed", error, {
|
||||||
|
mode
|
||||||
|
});
|
||||||
|
console.error("Obsidian Sync failed", error);
|
||||||
|
new Notice(
|
||||||
|
error instanceof Error
|
||||||
|
? `Sync failed: ${error.message}. Run ${diagnostics.getRunId()}.`
|
||||||
|
: `Sync failed. Run ${diagnostics.getRunId()}.`
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
await this.uploadDiagnostics(diagnostics);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private cloneSettings(): ObsidianSyncSettings {
|
||||||
|
const current = this.readSettings();
|
||||||
|
return {
|
||||||
|
...current,
|
||||||
|
syncState: {
|
||||||
|
...current.syncState,
|
||||||
|
files: { ...current.syncState.files }
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private getBaseUrl(): string {
|
||||||
|
return this.readSettings().serverUrl.replace(/\/$/, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
private getPullBatchSize(): number {
|
||||||
|
return Math.max(1, this.readSettings().pullBatchSize || DEFAULT_BATCH_SIZE);
|
||||||
|
}
|
||||||
|
|
||||||
|
private getPushBatchSize(): number {
|
||||||
|
return Math.max(1, this.readSettings().pushBatchSize || DEFAULT_BATCH_SIZE);
|
||||||
|
}
|
||||||
|
|
||||||
|
private getSyncableFiles(): TFile[] {
|
||||||
|
return this.app.vault.getFiles().filter((file) => !IGNORED_PATH_PREFIXES.some((prefix) => file.path.startsWith(prefix)));
|
||||||
|
}
|
||||||
|
|
||||||
|
private resolveFileKind(file: TFile, previous?: SyncedFileState): SyncFileKind {
|
||||||
|
if (previous?.kind) {
|
||||||
|
return previous.kind;
|
||||||
|
}
|
||||||
|
|
||||||
|
return TEXT_FILE_EXTENSIONS.has(file.extension.toLowerCase()) ? "text" : "binary";
|
||||||
|
}
|
||||||
|
|
||||||
|
private createSyncedState(
|
||||||
|
kind: SyncFileKind,
|
||||||
|
revisionId: string,
|
||||||
|
contentHash: string,
|
||||||
|
updatedAt: string,
|
||||||
|
lastSyncedContent?: string
|
||||||
|
): SyncedFileState {
|
||||||
|
return {
|
||||||
|
kind,
|
||||||
|
revisionId,
|
||||||
|
contentHash,
|
||||||
|
updatedAt,
|
||||||
|
...(kind === "text" && lastSyncedContent !== undefined ? { lastSyncedContent } : {})
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private buildRecoveryBundle(settings: ObsidianSyncSettings): RecoveryBundle {
|
||||||
|
return RecoveryBundleSchema.parse({
|
||||||
|
version: 1,
|
||||||
|
serverUrl: settings.serverUrl,
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
keyId: settings.keyId,
|
||||||
|
exportedVaultKey: settings.exportedVaultKey,
|
||||||
|
generatedAt: new Date().toISOString()
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private serializeRecoveryBundle(settings: ObsidianSyncSettings): string {
|
||||||
|
return JSON.stringify(this.buildRecoveryBundle(settings), null, 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async readLocalFileSnapshot(file: TFile, previous?: SyncedFileState): Promise<LocalFileSnapshot> {
|
||||||
|
const kind = this.resolveFileKind(file, previous);
|
||||||
|
|
||||||
|
if (kind === "text") {
|
||||||
|
const textContent = await this.app.vault.cachedRead(file);
|
||||||
|
return {
|
||||||
|
kind,
|
||||||
|
textContent,
|
||||||
|
contentHash: await computeTextHash(textContent),
|
||||||
|
updatedAt: new Date(file.stat.mtime).toISOString(),
|
||||||
|
sizeBytes: new TextEncoder().encode(textContent).byteLength
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const binaryContent = await this.app.vault.readBinary(file);
|
||||||
|
return {
|
||||||
|
kind,
|
||||||
|
binaryContent,
|
||||||
|
contentHash: await computeBinaryHash(binaryContent),
|
||||||
|
updatedAt: new Date(file.stat.mtime).toISOString(),
|
||||||
|
sizeBytes: file.stat.size
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private async ensureVaultKey(): Promise<VaultKeyHandle> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
if (settings.exportedVaultKey && settings.keyId) {
|
||||||
|
return importVaultKey(settings.exportedVaultKey, settings.keyId);
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
const generated = await generateVaultKey();
|
||||||
|
nextSettings.keyId = generated.keyId;
|
||||||
|
nextSettings.exportedVaultKey = generated.exportedKey;
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
|
||||||
|
return {
|
||||||
|
keyId: generated.keyId,
|
||||||
|
key: generated.key
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private async ensureRegisteredDevice(): Promise<void> {
|
||||||
|
await this.ensureDeviceRegistration(new SyncRunLogger(this.readSettings().deviceId));
|
||||||
|
}
|
||||||
|
|
||||||
|
private async ensureDeviceRegistration(diagnostics: SyncRunLogger): Promise<void> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
if (settings.deviceId && settings.authToken) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const deviceName = settings.deviceName || this.app.vault.getName();
|
||||||
|
const response = await this.postJson(
|
||||||
|
"/api/devices/register",
|
||||||
|
{
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
deviceName
|
||||||
|
},
|
||||||
|
RegisterDeviceResponseSchema
|
||||||
|
);
|
||||||
|
|
||||||
|
diagnostics.info("device-registered", {
|
||||||
|
deviceId: response.deviceId,
|
||||||
|
deviceName
|
||||||
|
});
|
||||||
|
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
nextSettings.deviceId = response.deviceId;
|
||||||
|
nextSettings.authToken = response.token;
|
||||||
|
nextSettings.deviceName = deviceName;
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
}
|
||||||
|
|
||||||
|
private ensureCompatibleVaultKey(pullResponse: SyncPullResponse): void {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
if (pullResponse.activeKeyId && settings.keyId && pullResponse.activeKeyId !== settings.keyId) {
|
||||||
|
throw new Error("The server is using a different vault key. Import the latest recovery bundle before syncing.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async pullRemoteChangesPage(sinceServerRevision: number): Promise<SyncPullResponse> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
return this.postJson(
|
||||||
|
"/api/sync/pull",
|
||||||
|
{
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
deviceId: settings.deviceId,
|
||||||
|
sinceServerRevision,
|
||||||
|
limit: this.getPullBatchSize()
|
||||||
|
},
|
||||||
|
SyncPullResponseSchema,
|
||||||
|
settings.authToken
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async pullAndApplyRemoteChanges(vaultKey: VaultKeyHandle, diagnostics: SyncRunLogger): Promise<PullSummary> {
|
||||||
|
let pulledChanges = 0;
|
||||||
|
let nextSinceServerRevision = this.readSettings().syncState.serverRevision;
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
const pullResponse = await this.pullRemoteChangesPage(nextSinceServerRevision);
|
||||||
|
this.ensureCompatibleVaultKey(pullResponse);
|
||||||
|
|
||||||
|
if (pullResponse.changes.length === 0) {
|
||||||
|
diagnostics.info("remote-pull-finished", {
|
||||||
|
changes: pulledChanges,
|
||||||
|
serverRevision: pullResponse.serverRevision,
|
||||||
|
hasMore: pullResponse.hasMore
|
||||||
|
});
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
await this.applyRemoteChanges(pullResponse, vaultKey, diagnostics);
|
||||||
|
pulledChanges += pullResponse.changes.length;
|
||||||
|
nextSinceServerRevision = pullResponse.nextSinceServerRevision;
|
||||||
|
|
||||||
|
diagnostics.info("remote-pull-page-applied", {
|
||||||
|
pageChanges: pullResponse.changes.length,
|
||||||
|
nextSinceServerRevision,
|
||||||
|
hasMore: pullResponse.hasMore
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!pullResponse.hasMore) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
pulledChanges,
|
||||||
|
finalServerRevision: this.readSettings().syncState.serverRevision
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private async pushLocalChanges(localChanges: LocalChangeSet, diagnostics: SyncRunLogger): Promise<PushSummary> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
if (localChanges.files.length === 0 && localChanges.tombstones.length === 0) {
|
||||||
|
return {
|
||||||
|
acceptedServerRevision: settings.syncState.serverRevision,
|
||||||
|
acceptedFilePaths: [],
|
||||||
|
acceptedTombstones: [],
|
||||||
|
conflicts: []
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const acceptedFilePaths: string[] = [];
|
||||||
|
const acceptedTombstones: string[] = [];
|
||||||
|
const conflicts: SyncConflict[] = [];
|
||||||
|
let acceptedServerRevision = settings.syncState.serverRevision;
|
||||||
|
let batchNumber = 0;
|
||||||
|
const fileBatches = this.chunkEntries(localChanges.files, this.getPushBatchSize());
|
||||||
|
const tombstoneBatches = this.chunkEntries(localChanges.tombstones, this.getPushBatchSize());
|
||||||
|
|
||||||
|
for (const fileBatch of fileBatches) {
|
||||||
|
batchNumber += 1;
|
||||||
|
const response = await this.pushBatch(fileBatch, [], acceptedServerRevision);
|
||||||
|
await this.reconcileAcceptedLocalChanges(localChanges.statesByPath, response.acceptedFilePaths, response.acceptedTombstones);
|
||||||
|
acceptedServerRevision = Math.max(acceptedServerRevision, response.acceptedServerRevision);
|
||||||
|
acceptedFilePaths.push(...response.acceptedFilePaths);
|
||||||
|
acceptedTombstones.push(...response.acceptedTombstones);
|
||||||
|
conflicts.push(...response.conflicts);
|
||||||
|
await this.persistAcceptedServerRevision(acceptedServerRevision);
|
||||||
|
|
||||||
|
diagnostics.info("push-batch-finished", {
|
||||||
|
batchNumber,
|
||||||
|
batchKind: "files",
|
||||||
|
attempted: fileBatch.length,
|
||||||
|
accepted: response.acceptedFilePaths.length,
|
||||||
|
conflicts: response.conflicts.length
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const tombstoneBatch of tombstoneBatches) {
|
||||||
|
batchNumber += 1;
|
||||||
|
const response = await this.pushBatch([], tombstoneBatch, acceptedServerRevision);
|
||||||
|
await this.reconcileAcceptedLocalChanges(localChanges.statesByPath, response.acceptedFilePaths, response.acceptedTombstones);
|
||||||
|
acceptedServerRevision = Math.max(acceptedServerRevision, response.acceptedServerRevision);
|
||||||
|
acceptedFilePaths.push(...response.acceptedFilePaths);
|
||||||
|
acceptedTombstones.push(...response.acceptedTombstones);
|
||||||
|
conflicts.push(...response.conflicts);
|
||||||
|
await this.persistAcceptedServerRevision(acceptedServerRevision);
|
||||||
|
|
||||||
|
diagnostics.info("push-batch-finished", {
|
||||||
|
batchNumber,
|
||||||
|
batchKind: "tombstones",
|
||||||
|
attempted: tombstoneBatch.length,
|
||||||
|
accepted: response.acceptedTombstones.length,
|
||||||
|
conflicts: response.conflicts.length
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
diagnostics.info("remote-push-finished", {
|
||||||
|
acceptedFiles: acceptedFilePaths.length,
|
||||||
|
acceptedDeletes: acceptedTombstones.length,
|
||||||
|
conflicts: conflicts.length
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
acceptedServerRevision,
|
||||||
|
acceptedFilePaths,
|
||||||
|
acceptedTombstones,
|
||||||
|
conflicts
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private async pushBatch(
|
||||||
|
files: SyncFileRecord[],
|
||||||
|
tombstones: Tombstone[],
|
||||||
|
knownServerRevision: number
|
||||||
|
): Promise<ReturnType<typeof SyncPushResponseSchema.parse>> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
return this.postJson(
|
||||||
|
"/api/sync/push",
|
||||||
|
{
|
||||||
|
vaultId: settings.vaultId,
|
||||||
|
deviceId: settings.deviceId,
|
||||||
|
knownServerRevision,
|
||||||
|
files,
|
||||||
|
tombstones
|
||||||
|
},
|
||||||
|
SyncPushResponseSchema,
|
||||||
|
settings.authToken
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private chunkEntries<TValue>(entries: TValue[], batchSize: number): TValue[][] {
|
||||||
|
if (entries.length === 0) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
const batches: TValue[][] = [];
|
||||||
|
for (let index = 0; index < entries.length; index += batchSize) {
|
||||||
|
batches.push(entries.slice(index, index + batchSize));
|
||||||
|
}
|
||||||
|
|
||||||
|
return batches;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async collectLocalChanges(
|
||||||
|
vaultKey: VaultKeyHandle,
|
||||||
|
diagnostics: SyncRunLogger,
|
||||||
|
options: CollectLocalChangeOptions = {}
|
||||||
|
): Promise<LocalChangeSet> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
const files: SyncFileRecord[] = [];
|
||||||
|
const tombstones: Tombstone[] = [];
|
||||||
|
const statesByPath: Record<string, SyncedFileState> = {};
|
||||||
|
const seenPaths = new Set<string>();
|
||||||
|
|
||||||
|
for (const file of this.getSyncableFiles()) {
|
||||||
|
const previous = settings.syncState.files[file.path];
|
||||||
|
const snapshot = await this.readLocalFileSnapshot(file, previous);
|
||||||
|
seenPaths.add(file.path);
|
||||||
|
|
||||||
|
if (previous?.pendingConflict && previous.pendingConflictHash === snapshot.contentHash) {
|
||||||
|
diagnostics.debug("skipped-pending-conflict-file", {
|
||||||
|
path: file.path,
|
||||||
|
kind: snapshot.kind
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!options.forceAllFiles && previous && previous.contentHash === snapshot.contentHash) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const envelope =
|
||||||
|
snapshot.kind === "text"
|
||||||
|
? await encryptText(snapshot.textContent ?? "", vaultKey)
|
||||||
|
: await encryptBytes(snapshot.binaryContent ?? new ArrayBuffer(0), vaultKey);
|
||||||
|
const revisionId = globalThis.crypto.randomUUID();
|
||||||
|
|
||||||
|
files.push({
|
||||||
|
manifest: {
|
||||||
|
path: file.path,
|
||||||
|
kind: snapshot.kind,
|
||||||
|
contentHash: snapshot.contentHash,
|
||||||
|
revisionId,
|
||||||
|
baseRevisionId: previous?.revisionId,
|
||||||
|
updatedAt: snapshot.updatedAt,
|
||||||
|
sizeBytes: snapshot.sizeBytes,
|
||||||
|
deviceId: settings.deviceId
|
||||||
|
},
|
||||||
|
envelope
|
||||||
|
});
|
||||||
|
|
||||||
|
statesByPath[file.path] = {
|
||||||
|
kind: snapshot.kind,
|
||||||
|
revisionId,
|
||||||
|
contentHash: snapshot.contentHash,
|
||||||
|
updatedAt: snapshot.updatedAt,
|
||||||
|
...(snapshot.kind === "text" ? { lastSyncedContent: snapshot.textContent } : {})
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const [path, state] of Object.entries(settings.syncState.files)) {
|
||||||
|
if (seenPaths.has(path)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
tombstones.push({
|
||||||
|
path,
|
||||||
|
revisionId: globalThis.crypto.randomUUID(),
|
||||||
|
baseRevisionId: state.revisionId,
|
||||||
|
deletedAt: new Date().toISOString(),
|
||||||
|
deviceId: settings.deviceId
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
diagnostics.info("local-scan-finished", {
|
||||||
|
stagedFiles: files.length,
|
||||||
|
stagedDeletes: tombstones.length,
|
||||||
|
forceAllFiles: Boolean(options.forceAllFiles)
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
files,
|
||||||
|
tombstones,
|
||||||
|
statesByPath
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private async persistAcceptedServerRevision(serverRevision: number): Promise<void> {
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
nextSettings.syncState.serverRevision = Math.max(nextSettings.syncState.serverRevision, serverRevision);
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async reconcileAcceptedLocalChanges(
|
||||||
|
statesByPath: Record<string, SyncedFileState>,
|
||||||
|
acceptedFilePaths: string[],
|
||||||
|
acceptedTombstones: string[]
|
||||||
|
): Promise<void> {
|
||||||
|
if (acceptedFilePaths.length === 0 && acceptedTombstones.length === 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
|
||||||
|
for (const path of acceptedFilePaths) {
|
||||||
|
const state = statesByPath[path];
|
||||||
|
if (state) {
|
||||||
|
nextSettings.syncState.files[path] = state;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const path of acceptedTombstones) {
|
||||||
|
delete nextSettings.syncState.files[path];
|
||||||
|
}
|
||||||
|
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async applyRemoteChanges(
|
||||||
|
pullResponse: SyncPullResponse,
|
||||||
|
vaultKey: VaultKeyHandle,
|
||||||
|
diagnostics: SyncRunLogger
|
||||||
|
): Promise<void> {
|
||||||
|
if (pullResponse.changes.length === 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextSettings = this.cloneSettings();
|
||||||
|
|
||||||
|
for (const change of pullResponse.changes) {
|
||||||
|
if (change.file) {
|
||||||
|
const remotePath = normalizePath(change.file.manifest.path);
|
||||||
|
const localAbstract = this.app.vault.getAbstractFileByPath(remotePath);
|
||||||
|
const previous = nextSettings.syncState.files[remotePath];
|
||||||
|
|
||||||
|
if (change.file.manifest.kind === "text") {
|
||||||
|
const remoteContent = await decryptText(change.file.envelope, vaultKey);
|
||||||
|
|
||||||
|
if (!(localAbstract instanceof TFile)) {
|
||||||
|
await this.ensureParentFolders(remotePath);
|
||||||
|
await this.app.vault.create(remotePath, remoteContent);
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"text",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt,
|
||||||
|
remoteContent
|
||||||
|
);
|
||||||
|
diagnostics.info("remote-text-created", {
|
||||||
|
path: remotePath
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const localContent = await this.app.vault.cachedRead(localAbstract);
|
||||||
|
const localHash = await computeTextHash(localContent);
|
||||||
|
|
||||||
|
if (localHash === change.file.manifest.contentHash) {
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"text",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt,
|
||||||
|
remoteContent
|
||||||
|
);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!previous || localHash === previous.contentHash || localContent === previous.lastSyncedContent) {
|
||||||
|
await this.app.vault.modify(localAbstract, remoteContent);
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"text",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt,
|
||||||
|
remoteContent
|
||||||
|
);
|
||||||
|
diagnostics.info("remote-text-applied", {
|
||||||
|
path: remotePath
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const merged = mergeTextRevisions({
|
||||||
|
base: previous.lastSyncedContent,
|
||||||
|
local: localContent,
|
||||||
|
remote: remoteContent
|
||||||
|
});
|
||||||
|
|
||||||
|
await this.app.vault.modify(localAbstract, merged.content);
|
||||||
|
|
||||||
|
if (merged.status === "conflict") {
|
||||||
|
const conflictCopyPath = createConflictCopyPath(remotePath);
|
||||||
|
await this.ensureParentFolders(conflictCopyPath);
|
||||||
|
await this.writeTextFile(conflictCopyPath, remoteContent);
|
||||||
|
|
||||||
|
nextSettings.syncState.files[remotePath] = {
|
||||||
|
...this.createSyncedState(
|
||||||
|
"text",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt,
|
||||||
|
remoteContent
|
||||||
|
),
|
||||||
|
pendingConflict: true,
|
||||||
|
pendingConflictHash: await computeTextHash(merged.content)
|
||||||
|
};
|
||||||
|
|
||||||
|
diagnostics.warn("remote-text-conflict", {
|
||||||
|
path: remotePath,
|
||||||
|
conflictCopyPath
|
||||||
|
});
|
||||||
|
new Notice(`Conflict markers added to ${remotePath}. Review the file before the next sync.`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"text",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt,
|
||||||
|
remoteContent
|
||||||
|
);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const remoteBinary = await decryptBytes(change.file.envelope, vaultKey);
|
||||||
|
|
||||||
|
if (!(localAbstract instanceof TFile)) {
|
||||||
|
await this.ensureParentFolders(remotePath);
|
||||||
|
await this.app.vault.createBinary(remotePath, remoteBinary);
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"binary",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt
|
||||||
|
);
|
||||||
|
diagnostics.info("remote-binary-created", {
|
||||||
|
path: remotePath
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const localBinary = await this.app.vault.readBinary(localAbstract);
|
||||||
|
const localHash = await computeBinaryHash(localBinary);
|
||||||
|
|
||||||
|
if (localHash === change.file.manifest.contentHash) {
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"binary",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt
|
||||||
|
);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!previous || localHash === previous.contentHash) {
|
||||||
|
await this.app.vault.modifyBinary(localAbstract, remoteBinary);
|
||||||
|
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||||
|
"binary",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt
|
||||||
|
);
|
||||||
|
diagnostics.info("remote-binary-applied", {
|
||||||
|
path: remotePath
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const conflictCopyPath = createConflictCopyPath(remotePath);
|
||||||
|
await this.ensureParentFolders(conflictCopyPath);
|
||||||
|
await this.writeBinaryFile(conflictCopyPath, remoteBinary);
|
||||||
|
|
||||||
|
nextSettings.syncState.files[remotePath] = {
|
||||||
|
...this.createSyncedState(
|
||||||
|
"binary",
|
||||||
|
change.file.manifest.revisionId,
|
||||||
|
change.file.manifest.contentHash,
|
||||||
|
change.file.manifest.updatedAt
|
||||||
|
),
|
||||||
|
pendingConflict: true,
|
||||||
|
pendingConflictHash: localHash
|
||||||
|
};
|
||||||
|
|
||||||
|
diagnostics.warn("remote-binary-conflict", {
|
||||||
|
path: remotePath,
|
||||||
|
conflictCopyPath
|
||||||
|
});
|
||||||
|
new Notice(`Attachment conflict detected for ${remotePath}. Review the remote conflict copy before the next sync.`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (change.tombstone) {
|
||||||
|
const localAbstract = this.app.vault.getAbstractFileByPath(change.tombstone.path);
|
||||||
|
const previous = nextSettings.syncState.files[change.tombstone.path];
|
||||||
|
|
||||||
|
if (!(localAbstract instanceof TFile)) {
|
||||||
|
delete nextSettings.syncState.files[change.tombstone.path];
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const snapshot = await this.readLocalFileSnapshot(localAbstract, previous);
|
||||||
|
if (!previous || snapshot.contentHash === previous.contentHash) {
|
||||||
|
await this.app.vault.delete(localAbstract, true);
|
||||||
|
delete nextSettings.syncState.files[change.tombstone.path];
|
||||||
|
diagnostics.info("remote-delete-applied", {
|
||||||
|
path: change.tombstone.path
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
diagnostics.warn("remote-delete-skipped", {
|
||||||
|
path: change.tombstone.path
|
||||||
|
});
|
||||||
|
new Notice(`Skipped remote delete for ${change.tombstone.path} because local changes still exist.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
nextSettings.syncState.serverRevision = pullResponse.nextSinceServerRevision;
|
||||||
|
await this.writeSettings(nextSettings);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async ensureParentFolders(path: string): Promise<void> {
|
||||||
|
const segments = normalizePath(path).split("/");
|
||||||
|
segments.pop();
|
||||||
|
|
||||||
|
let currentPath = "";
|
||||||
|
for (const segment of segments) {
|
||||||
|
currentPath = currentPath ? `${currentPath}/${segment}` : segment;
|
||||||
|
if (!this.app.vault.getAbstractFileByPath(currentPath)) {
|
||||||
|
await this.app.vault.createFolder(currentPath);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async writeTextFile(path: string, content: string): Promise<void> {
|
||||||
|
const existing = this.app.vault.getAbstractFileByPath(path);
|
||||||
|
if (existing instanceof TFile) {
|
||||||
|
await this.app.vault.modify(existing, content);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await this.app.vault.create(path, content);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async writeBinaryFile(path: string, content: ArrayBuffer): Promise<void> {
|
||||||
|
const existing = this.app.vault.getAbstractFileByPath(path);
|
||||||
|
if (existing instanceof TFile) {
|
||||||
|
await this.app.vault.modifyBinary(existing, content);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await this.app.vault.createBinary(path, content);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async uploadDiagnostics(diagnostics: SyncRunLogger): Promise<void> {
|
||||||
|
const settings = this.readSettings();
|
||||||
|
diagnostics.setDeviceId(settings.deviceId);
|
||||||
|
|
||||||
|
if (!diagnostics.hasEntries() || !settings.deviceId || !settings.authToken) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await this.postJson(
|
||||||
|
"/api/logs",
|
||||||
|
diagnostics.toUploadRequest(settings.vaultId),
|
||||||
|
ClientLogUploadResponseSchema,
|
||||||
|
settings.authToken
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
console.warn("Failed to upload sync diagnostics", error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async postJson<TSchema>(
|
||||||
|
path: string,
|
||||||
|
body: unknown,
|
||||||
|
schema: { parse: (value: unknown) => TSchema },
|
||||||
|
authToken?: string
|
||||||
|
): Promise<TSchema> {
|
||||||
|
const response = await fetch(`${this.getBaseUrl()}${path}`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
"content-type": "application/json",
|
||||||
|
...(authToken ? { authorization: `Bearer ${authToken}` } : {})
|
||||||
|
},
|
||||||
|
body: JSON.stringify(body)
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`${response.status} ${response.statusText}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const json = (await response.json()) as unknown;
|
||||||
|
return schema.parse(json);
|
||||||
|
}
|
||||||
|
}
|
||||||
21
apps/obsidian-plugin/tsconfig.json
Normal file
21
apps/obsidian-plugin/tsconfig.json
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"extends": "../../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"composite": true,
|
||||||
|
"declaration": true,
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": "src",
|
||||||
|
"module": "CommonJS"
|
||||||
|
},
|
||||||
|
"references": [
|
||||||
|
{
|
||||||
|
"path": "../../packages/sync-protocol"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": "../../packages/sync-engine"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"include": [
|
||||||
|
"src/**/*.ts"
|
||||||
|
]
|
||||||
|
}
|
||||||
23
apps/sync-server/package.json
Normal file
23
apps/sync-server/package.json
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "@obsidian-sync/sync-server",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"types": "dist/index.d.ts",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc -b",
|
||||||
|
"dev": "tsx watch src/index.ts",
|
||||||
|
"start": "node dist/index.js",
|
||||||
|
"typecheck": "tsc -b --pretty false"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@obsidian-sync/sync-protocol": "0.1.0",
|
||||||
|
"cors": "^2.8.5",
|
||||||
|
"express": "^4.21.2"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/cors": "^2.8.17",
|
||||||
|
"@types/express": "^4.17.21",
|
||||||
|
"tsx": "^4.19.3"
|
||||||
|
}
|
||||||
|
}
|
||||||
38
apps/sync-server/src/index.ts
Normal file
38
apps/sync-server/src/index.ts
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
import path from "node:path";
|
||||||
|
|
||||||
|
import cors from "cors";
|
||||||
|
import express from "express";
|
||||||
|
|
||||||
|
import { createJsonFileLogger } from "./logging/jsonFileLogger";
|
||||||
|
import { createRequestLoggingMiddleware } from "./middleware/requestLoggingMiddleware";
|
||||||
|
import { createSyncRouter } from "./routes/createSyncRouter";
|
||||||
|
import { createFileSyncStore } from "./store/fileSyncStore";
|
||||||
|
|
||||||
|
const port = Number(process.env.PORT ?? 8787);
|
||||||
|
const dataDirectory = process.env.SYNC_DATA_DIR
|
||||||
|
? path.resolve(process.env.SYNC_DATA_DIR)
|
||||||
|
: path.resolve(process.cwd(), "data");
|
||||||
|
const app = express();
|
||||||
|
const logger = createJsonFileLogger(path.join(dataDirectory, "logs", "server.jsonl"));
|
||||||
|
const store = createFileSyncStore(path.join(dataDirectory, "sync-state.json"));
|
||||||
|
|
||||||
|
app.use(cors());
|
||||||
|
app.use(express.json({ limit: "10mb" }));
|
||||||
|
app.use(createRequestLoggingMiddleware(logger));
|
||||||
|
|
||||||
|
app.get("/health", (_request, response) => {
|
||||||
|
response.json({
|
||||||
|
status: "ok",
|
||||||
|
dataDirectory
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
app.use("/api", createSyncRouter(store, logger));
|
||||||
|
|
||||||
|
app.listen(port, () => {
|
||||||
|
logger.info("server-started", {
|
||||||
|
port,
|
||||||
|
dataDirectory
|
||||||
|
});
|
||||||
|
console.log(`Obsidian Sync server listening on http://localhost:${port}`);
|
||||||
|
});
|
||||||
45
apps/sync-server/src/logging/jsonFileLogger.ts
Normal file
45
apps/sync-server/src/logging/jsonFileLogger.ts
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
import { appendFileSync, mkdirSync } from "node:fs";
|
||||||
|
import { dirname } from "node:path";
|
||||||
|
|
||||||
|
export type LogLevel = "debug" | "info" | "warn" | "error";
|
||||||
|
|
||||||
|
export interface StructuredLogger {
|
||||||
|
log(level: LogLevel, message: string, context?: Record<string, unknown>): void;
|
||||||
|
debug(message: string, context?: Record<string, unknown>): void;
|
||||||
|
info(message: string, context?: Record<string, unknown>): void;
|
||||||
|
warn(message: string, context?: Record<string, unknown>): void;
|
||||||
|
error(message: string, context?: Record<string, unknown>): void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createJsonFileLogger(logFilePath: string): StructuredLogger {
|
||||||
|
mkdirSync(dirname(logFilePath), { recursive: true });
|
||||||
|
|
||||||
|
const write = (level: LogLevel, message: string, context?: Record<string, unknown>) => {
|
||||||
|
const payload = {
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
level,
|
||||||
|
message,
|
||||||
|
context: context ?? {}
|
||||||
|
};
|
||||||
|
|
||||||
|
appendFileSync(logFilePath, `${JSON.stringify(payload)}\n`, "utf8");
|
||||||
|
};
|
||||||
|
|
||||||
|
return {
|
||||||
|
log(level, message, context) {
|
||||||
|
write(level, message, context);
|
||||||
|
},
|
||||||
|
debug(message, context) {
|
||||||
|
write("debug", message, context);
|
||||||
|
},
|
||||||
|
info(message, context) {
|
||||||
|
write("info", message, context);
|
||||||
|
},
|
||||||
|
warn(message, context) {
|
||||||
|
write("warn", message, context);
|
||||||
|
},
|
||||||
|
error(message, context) {
|
||||||
|
write("error", message, context);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
26
apps/sync-server/src/middleware/requestLoggingMiddleware.ts
Normal file
26
apps/sync-server/src/middleware/requestLoggingMiddleware.ts
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
import type { Request, Response } from "express";
|
||||||
|
import type { NextFunction } from "express";
|
||||||
|
|
||||||
|
import type { StructuredLogger } from "../logging/jsonFileLogger";
|
||||||
|
|
||||||
|
export function createRequestLoggingMiddleware(logger: StructuredLogger) {
|
||||||
|
return (request: Request, response: Response, next: NextFunction) => {
|
||||||
|
const requestId = globalThis.crypto.randomUUID();
|
||||||
|
const startedAt = Date.now();
|
||||||
|
|
||||||
|
response.locals.requestId = requestId;
|
||||||
|
response.setHeader("x-request-id", requestId);
|
||||||
|
|
||||||
|
response.on("finish", () => {
|
||||||
|
logger.info("http-request", {
|
||||||
|
requestId,
|
||||||
|
method: request.method,
|
||||||
|
path: request.originalUrl,
|
||||||
|
statusCode: response.statusCode,
|
||||||
|
durationMs: Date.now() - startedAt
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
next();
|
||||||
|
};
|
||||||
|
}
|
||||||
206
apps/sync-server/src/routes/createSyncRouter.ts
Normal file
206
apps/sync-server/src/routes/createSyncRouter.ts
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
import type { Request, Response } from "express";
|
||||||
|
import express from "express";
|
||||||
|
|
||||||
|
import {
|
||||||
|
type ClientLogUploadRequest,
|
||||||
|
type ListDevicesRequest,
|
||||||
|
type RegisterDeviceRequest,
|
||||||
|
type RevokeDeviceRequest,
|
||||||
|
type RotateVaultKeyRequest,
|
||||||
|
type SyncPullRequest,
|
||||||
|
type SyncPushRequest,
|
||||||
|
ClientLogUploadRequestSchema,
|
||||||
|
ClientLogUploadResponseSchema,
|
||||||
|
ListDevicesRequestSchema,
|
||||||
|
ListDevicesResponseSchema,
|
||||||
|
RegisterDeviceRequestSchema,
|
||||||
|
RevokeDeviceRequestSchema,
|
||||||
|
RevokeDeviceResponseSchema,
|
||||||
|
RotateVaultKeyRequestSchema,
|
||||||
|
RotateVaultKeyResponseSchema,
|
||||||
|
SyncPullRequestSchema,
|
||||||
|
SyncPushRequestSchema,
|
||||||
|
parseWithSchema
|
||||||
|
} from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
import type { StructuredLogger } from "../logging/jsonFileLogger";
|
||||||
|
import type { SyncStore } from "../store/syncStore";
|
||||||
|
|
||||||
|
function readBearerToken(request: Request): string | undefined {
|
||||||
|
const authorization = request.header("authorization");
|
||||||
|
return authorization?.replace(/^Bearer\s+/i, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
function getRequestId(response: Response): string {
|
||||||
|
return String(response.locals.requestId ?? "unknown-request");
|
||||||
|
}
|
||||||
|
|
||||||
|
function unauthorized(request: Request, response: Response, logger: StructuredLogger): void {
|
||||||
|
logger.warn("request-unauthorized", {
|
||||||
|
requestId: getRequestId(response),
|
||||||
|
method: request.method,
|
||||||
|
path: request.originalUrl
|
||||||
|
});
|
||||||
|
|
||||||
|
response.status(401).json({
|
||||||
|
error: "Unauthorized"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function badRequest(request: Request, response: Response, logger: StructuredLogger, error: unknown): void {
|
||||||
|
logger.warn("request-invalid", {
|
||||||
|
requestId: getRequestId(response),
|
||||||
|
method: request.method,
|
||||||
|
path: request.originalUrl,
|
||||||
|
error: error instanceof Error ? error.message : "Invalid request"
|
||||||
|
});
|
||||||
|
|
||||||
|
response.status(400).json({
|
||||||
|
error: error instanceof Error ? error.message : "Invalid request"
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createSyncRouter(store: SyncStore, logger: StructuredLogger): express.Router {
|
||||||
|
const router = express.Router();
|
||||||
|
|
||||||
|
router.post("/devices/register", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: RegisterDeviceRequest = parseWithSchema(RegisterDeviceRequestSchema, request.body);
|
||||||
|
const registered = store.registerDevice(body);
|
||||||
|
logger.info("device-registered", {
|
||||||
|
requestId: getRequestId(response),
|
||||||
|
vaultId: registered.vaultId,
|
||||||
|
deviceId: registered.deviceId,
|
||||||
|
deviceName: body.deviceName
|
||||||
|
});
|
||||||
|
response.status(201).json(registered);
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.post("/sync/pull", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: SyncPullRequest = parseWithSchema(SyncPullRequestSchema, request.body);
|
||||||
|
const token = readBearerToken(request);
|
||||||
|
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||||
|
unauthorized(request, response, logger);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
response.json(store.pull(body.vaultId, body.sinceServerRevision, body.limit));
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.post("/sync/push", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: SyncPushRequest = parseWithSchema(SyncPushRequestSchema, request.body);
|
||||||
|
const token = readBearerToken(request);
|
||||||
|
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||||
|
unauthorized(request, response, logger);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
response.json(store.push(body));
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.post("/devices/list", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: ListDevicesRequest = parseWithSchema(ListDevicesRequestSchema, request.body);
|
||||||
|
const token = readBearerToken(request);
|
||||||
|
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||||
|
unauthorized(request, response, logger);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
response.json(ListDevicesResponseSchema.parse(store.listDevices(body.vaultId)));
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.post("/devices/revoke", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: RevokeDeviceRequest = parseWithSchema(RevokeDeviceRequestSchema, request.body);
|
||||||
|
const token = readBearerToken(request);
|
||||||
|
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||||
|
unauthorized(request, response, logger);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const revoked = store.revokeDevice(body.vaultId, body.targetDeviceId);
|
||||||
|
logger.warn("device-revoked", {
|
||||||
|
requestId: getRequestId(response),
|
||||||
|
vaultId: body.vaultId,
|
||||||
|
requestedBy: body.deviceId,
|
||||||
|
targetDeviceId: body.targetDeviceId,
|
||||||
|
revokedAt: revoked.revokedAt
|
||||||
|
});
|
||||||
|
response.json(RevokeDeviceResponseSchema.parse(revoked));
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.post("/keys/rotate", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: RotateVaultKeyRequest = parseWithSchema(RotateVaultKeyRequestSchema, request.body);
|
||||||
|
const token = readBearerToken(request);
|
||||||
|
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||||
|
unauthorized(request, response, logger);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const rotated = store.rotateVaultKey(body.vaultId, body.nextKeyId, body.previousKeyId);
|
||||||
|
logger.info("vault-key-rotated", {
|
||||||
|
requestId: getRequestId(response),
|
||||||
|
vaultId: body.vaultId,
|
||||||
|
requestedBy: body.deviceId,
|
||||||
|
activeKeyId: rotated.activeKeyId,
|
||||||
|
rotatedAt: rotated.rotatedAt
|
||||||
|
});
|
||||||
|
response.json(RotateVaultKeyResponseSchema.parse(rotated));
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
router.post("/logs", (request: Request, response: Response) => {
|
||||||
|
try {
|
||||||
|
const body: ClientLogUploadRequest = parseWithSchema(ClientLogUploadRequestSchema, request.body);
|
||||||
|
const token = readBearerToken(request);
|
||||||
|
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||||
|
unauthorized(request, response, logger);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const entry of body.entries) {
|
||||||
|
logger.log(entry.level, "client-sync-log", {
|
||||||
|
requestId: getRequestId(response),
|
||||||
|
vaultId: body.vaultId,
|
||||||
|
deviceId: body.deviceId,
|
||||||
|
runId: body.runId,
|
||||||
|
clientTimestamp: entry.timestamp,
|
||||||
|
clientMessage: entry.message,
|
||||||
|
context: entry.context ?? {}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
response.status(202).json(
|
||||||
|
ClientLogUploadResponseSchema.parse({
|
||||||
|
accepted: body.entries.length,
|
||||||
|
requestId: getRequestId(response)
|
||||||
|
})
|
||||||
|
);
|
||||||
|
} catch (error) {
|
||||||
|
badRequest(request, response, logger, error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
return router;
|
||||||
|
}
|
||||||
341
apps/sync-server/src/store/fileSyncStore.ts
Normal file
341
apps/sync-server/src/store/fileSyncStore.ts
Normal file
@@ -0,0 +1,341 @@
|
|||||||
|
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
|
||||||
|
import { dirname } from "node:path";
|
||||||
|
|
||||||
|
import type {
|
||||||
|
ListDevicesResponse,
|
||||||
|
RegisterDeviceRequest,
|
||||||
|
RegisterDeviceResponse,
|
||||||
|
RevokeDeviceResponse,
|
||||||
|
RotateVaultKeyResponse,
|
||||||
|
SyncChange,
|
||||||
|
SyncConflict,
|
||||||
|
SyncFileRecord,
|
||||||
|
SyncPullResponse,
|
||||||
|
SyncPushRequest,
|
||||||
|
SyncPushResponse,
|
||||||
|
Tombstone
|
||||||
|
} from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
import type { SyncStore } from "./syncStore";
|
||||||
|
|
||||||
|
interface VaultDevice {
|
||||||
|
deviceId: string;
|
||||||
|
deviceName: string;
|
||||||
|
token: string;
|
||||||
|
issuedAt: string;
|
||||||
|
revokedAt?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface VaultState {
|
||||||
|
serverRevision: number;
|
||||||
|
devices: Map<string, VaultDevice>;
|
||||||
|
files: Map<string, SyncFileRecord>;
|
||||||
|
tombstones: Map<string, Tombstone>;
|
||||||
|
changes: SyncChange[];
|
||||||
|
activeKeyId?: string;
|
||||||
|
keyRotatedAt?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PersistedVaultState {
|
||||||
|
serverRevision: number;
|
||||||
|
devices: Record<string, VaultDevice>;
|
||||||
|
files: Record<string, SyncFileRecord>;
|
||||||
|
tombstones: Record<string, Tombstone>;
|
||||||
|
changes: SyncChange[];
|
||||||
|
activeKeyId?: string;
|
||||||
|
keyRotatedAt?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PersistedStore {
|
||||||
|
vaults: Record<string, PersistedVaultState>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function createVaultState(): VaultState {
|
||||||
|
return {
|
||||||
|
serverRevision: 0,
|
||||||
|
devices: new Map(),
|
||||||
|
files: new Map(),
|
||||||
|
tombstones: new Map(),
|
||||||
|
changes: []
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function mapFromRecord<TValue>(record: Record<string, TValue> | undefined): Map<string, TValue> {
|
||||||
|
return new Map(Object.entries(record ?? {}));
|
||||||
|
}
|
||||||
|
|
||||||
|
function recordFromMap<TValue>(map: Map<string, TValue>): Record<string, TValue> {
|
||||||
|
return Object.fromEntries(map.entries());
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadVaults(filePath: string): Map<string, VaultState> {
|
||||||
|
if (!existsSync(filePath)) {
|
||||||
|
return new Map();
|
||||||
|
}
|
||||||
|
|
||||||
|
const raw = readFileSync(filePath, "utf8");
|
||||||
|
if (!raw.trim()) {
|
||||||
|
return new Map();
|
||||||
|
}
|
||||||
|
|
||||||
|
const parsed = JSON.parse(raw) as Partial<PersistedStore>;
|
||||||
|
const vaults = new Map<string, VaultState>();
|
||||||
|
|
||||||
|
for (const [vaultId, vault] of Object.entries(parsed.vaults ?? {})) {
|
||||||
|
vaults.set(vaultId, {
|
||||||
|
serverRevision: vault.serverRevision ?? 0,
|
||||||
|
devices: mapFromRecord(vault.devices),
|
||||||
|
files: mapFromRecord(vault.files),
|
||||||
|
tombstones: mapFromRecord(vault.tombstones),
|
||||||
|
changes: [...(vault.changes ?? [])],
|
||||||
|
activeKeyId: vault.activeKeyId,
|
||||||
|
keyRotatedAt: vault.keyRotatedAt
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return vaults;
|
||||||
|
}
|
||||||
|
|
||||||
|
function persistVaults(filePath: string, vaults: Map<string, VaultState>): void {
|
||||||
|
mkdirSync(dirname(filePath), { recursive: true });
|
||||||
|
|
||||||
|
const payload: PersistedStore = {
|
||||||
|
vaults: Object.fromEntries(
|
||||||
|
[...vaults.entries()].map(([vaultId, vault]) => [
|
||||||
|
vaultId,
|
||||||
|
{
|
||||||
|
serverRevision: vault.serverRevision,
|
||||||
|
devices: recordFromMap(vault.devices),
|
||||||
|
files: recordFromMap(vault.files),
|
||||||
|
tombstones: recordFromMap(vault.tombstones),
|
||||||
|
changes: vault.changes,
|
||||||
|
activeKeyId: vault.activeKeyId,
|
||||||
|
keyRotatedAt: vault.keyRotatedAt
|
||||||
|
}
|
||||||
|
])
|
||||||
|
)
|
||||||
|
};
|
||||||
|
|
||||||
|
const temporaryPath = `${filePath}.tmp`;
|
||||||
|
writeFileSync(temporaryPath, JSON.stringify(payload, null, 2), "utf8");
|
||||||
|
renameSync(temporaryPath, filePath);
|
||||||
|
}
|
||||||
|
|
||||||
|
function appendFileChange(vault: VaultState, file: SyncFileRecord): void {
|
||||||
|
vault.serverRevision += 1;
|
||||||
|
vault.changes.push({
|
||||||
|
serverRevision: vault.serverRevision,
|
||||||
|
file
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function appendTombstoneChange(vault: VaultState, tombstone: Tombstone): void {
|
||||||
|
vault.serverRevision += 1;
|
||||||
|
vault.changes.push({
|
||||||
|
serverRevision: vault.serverRevision,
|
||||||
|
tombstone
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function detectFileConflict(current: SyncFileRecord | undefined, incoming: SyncFileRecord): SyncConflict | undefined {
|
||||||
|
if (!current) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (current.manifest.contentHash === incoming.manifest.contentHash) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!incoming.manifest.baseRevisionId || incoming.manifest.baseRevisionId !== current.manifest.revisionId) {
|
||||||
|
return {
|
||||||
|
path: incoming.manifest.path,
|
||||||
|
serverRevisionId: current.manifest.revisionId,
|
||||||
|
clientRevisionId: incoming.manifest.revisionId,
|
||||||
|
reason: "revision-mismatch"
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function detectTombstoneConflict(current: SyncFileRecord | undefined, incoming: Tombstone): SyncConflict | undefined {
|
||||||
|
if (!current) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!incoming.baseRevisionId || incoming.baseRevisionId !== current.manifest.revisionId) {
|
||||||
|
return {
|
||||||
|
path: incoming.path,
|
||||||
|
serverRevisionId: current.manifest.revisionId,
|
||||||
|
clientRevisionId: incoming.revisionId,
|
||||||
|
reason: "revision-mismatch"
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createFileSyncStore(filePath: string): SyncStore {
|
||||||
|
const vaults = loadVaults(filePath);
|
||||||
|
|
||||||
|
function getVault(vaultId: string): VaultState {
|
||||||
|
let vault = vaults.get(vaultId);
|
||||||
|
if (!vault) {
|
||||||
|
vault = createVaultState();
|
||||||
|
vaults.set(vaultId, vault);
|
||||||
|
}
|
||||||
|
|
||||||
|
return vault;
|
||||||
|
}
|
||||||
|
|
||||||
|
function persist(): void {
|
||||||
|
persistVaults(filePath, vaults);
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
registerDevice(request: RegisterDeviceRequest): RegisterDeviceResponse {
|
||||||
|
const vault = getVault(request.vaultId);
|
||||||
|
const deviceId = globalThis.crypto.randomUUID();
|
||||||
|
const token = globalThis.crypto.randomUUID();
|
||||||
|
const issuedAt = new Date().toISOString();
|
||||||
|
|
||||||
|
vault.devices.set(deviceId, {
|
||||||
|
deviceId,
|
||||||
|
deviceName: request.deviceName,
|
||||||
|
token,
|
||||||
|
issuedAt
|
||||||
|
});
|
||||||
|
|
||||||
|
persist();
|
||||||
|
|
||||||
|
return {
|
||||||
|
vaultId: request.vaultId,
|
||||||
|
deviceId,
|
||||||
|
token,
|
||||||
|
issuedAt
|
||||||
|
};
|
||||||
|
},
|
||||||
|
|
||||||
|
authenticate(vaultId: string, deviceId: string, token: string): boolean {
|
||||||
|
const vault = vaults.get(vaultId);
|
||||||
|
const device = vault?.devices.get(deviceId);
|
||||||
|
return Boolean(device && !device.revokedAt && device.token === token);
|
||||||
|
},
|
||||||
|
|
||||||
|
pull(vaultId: string, sinceServerRevision: number, limit?: number): SyncPullResponse {
|
||||||
|
const vault = getVault(vaultId);
|
||||||
|
const matchingChanges = vault.changes.filter((change) => change.serverRevision > sinceServerRevision);
|
||||||
|
const page = matchingChanges.slice(0, limit ?? matchingChanges.length);
|
||||||
|
const nextSinceServerRevision = page.at(-1)?.serverRevision ?? sinceServerRevision;
|
||||||
|
|
||||||
|
return {
|
||||||
|
serverRevision: vault.serverRevision,
|
||||||
|
changes: page,
|
||||||
|
hasMore: matchingChanges.length > page.length,
|
||||||
|
nextSinceServerRevision,
|
||||||
|
activeKeyId: vault.activeKeyId,
|
||||||
|
keyRotatedAt: vault.keyRotatedAt
|
||||||
|
};
|
||||||
|
},
|
||||||
|
|
||||||
|
push(request: SyncPushRequest): SyncPushResponse {
|
||||||
|
const vault = getVault(request.vaultId);
|
||||||
|
const acceptedFilePaths: string[] = [];
|
||||||
|
const acceptedTombstones: string[] = [];
|
||||||
|
const conflicts: SyncConflict[] = [];
|
||||||
|
let acceptedKeyId: string | undefined;
|
||||||
|
|
||||||
|
for (const file of request.files) {
|
||||||
|
const current = vault.files.get(file.manifest.path);
|
||||||
|
const conflict = detectFileConflict(current, file);
|
||||||
|
if (conflict) {
|
||||||
|
conflicts.push(conflict);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
vault.files.set(file.manifest.path, file);
|
||||||
|
vault.tombstones.delete(file.manifest.path);
|
||||||
|
appendFileChange(vault, file);
|
||||||
|
acceptedFilePaths.push(file.manifest.path);
|
||||||
|
acceptedKeyId = file.envelope.keyId;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const tombstone of request.tombstones) {
|
||||||
|
const current = vault.files.get(tombstone.path);
|
||||||
|
const conflict = detectTombstoneConflict(current, tombstone);
|
||||||
|
if (conflict) {
|
||||||
|
conflicts.push(conflict);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
vault.files.delete(tombstone.path);
|
||||||
|
vault.tombstones.set(tombstone.path, tombstone);
|
||||||
|
appendTombstoneChange(vault, tombstone);
|
||||||
|
acceptedTombstones.push(tombstone.path);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (acceptedKeyId && acceptedKeyId !== vault.activeKeyId) {
|
||||||
|
vault.activeKeyId = acceptedKeyId;
|
||||||
|
vault.keyRotatedAt = new Date().toISOString();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (acceptedFilePaths.length > 0 || acceptedTombstones.length > 0) {
|
||||||
|
persist();
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
acceptedServerRevision: vault.serverRevision,
|
||||||
|
acceptedFilePaths,
|
||||||
|
acceptedTombstones,
|
||||||
|
conflicts,
|
||||||
|
activeKeyId: vault.activeKeyId,
|
||||||
|
keyRotatedAt: vault.keyRotatedAt
|
||||||
|
};
|
||||||
|
},
|
||||||
|
|
||||||
|
listDevices(vaultId: string): ListDevicesResponse {
|
||||||
|
const vault = getVault(vaultId);
|
||||||
|
const devices = [...vault.devices.values()].sort((left, right) => left.issuedAt.localeCompare(right.issuedAt));
|
||||||
|
return {
|
||||||
|
devices,
|
||||||
|
activeKeyId: vault.activeKeyId,
|
||||||
|
keyRotatedAt: vault.keyRotatedAt
|
||||||
|
};
|
||||||
|
},
|
||||||
|
|
||||||
|
revokeDevice(vaultId: string, targetDeviceId: string): RevokeDeviceResponse {
|
||||||
|
const vault = getVault(vaultId);
|
||||||
|
const device = vault.devices.get(targetDeviceId);
|
||||||
|
if (!device) {
|
||||||
|
throw new Error("Device not found.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!device.revokedAt) {
|
||||||
|
device.revokedAt = new Date().toISOString();
|
||||||
|
vault.devices.set(targetDeviceId, device);
|
||||||
|
persist();
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
targetDeviceId,
|
||||||
|
revokedAt: device.revokedAt
|
||||||
|
};
|
||||||
|
},
|
||||||
|
|
||||||
|
rotateVaultKey(vaultId: string, nextKeyId: string, previousKeyId?: string): RotateVaultKeyResponse {
|
||||||
|
const vault = getVault(vaultId);
|
||||||
|
if (previousKeyId && vault.activeKeyId && previousKeyId !== vault.activeKeyId) {
|
||||||
|
throw new Error("The provided previous key does not match the current server key.");
|
||||||
|
}
|
||||||
|
|
||||||
|
vault.activeKeyId = nextKeyId;
|
||||||
|
vault.keyRotatedAt = new Date().toISOString();
|
||||||
|
persist();
|
||||||
|
|
||||||
|
return {
|
||||||
|
activeKeyId: nextKeyId,
|
||||||
|
rotatedAt: vault.keyRotatedAt
|
||||||
|
};
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
20
apps/sync-server/src/store/syncStore.ts
Normal file
20
apps/sync-server/src/store/syncStore.ts
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
import type {
|
||||||
|
ListDevicesResponse,
|
||||||
|
RegisterDeviceRequest,
|
||||||
|
RegisterDeviceResponse,
|
||||||
|
RevokeDeviceResponse,
|
||||||
|
RotateVaultKeyResponse,
|
||||||
|
SyncPullResponse,
|
||||||
|
SyncPushRequest,
|
||||||
|
SyncPushResponse
|
||||||
|
} from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
export interface SyncStore {
|
||||||
|
registerDevice(request: RegisterDeviceRequest): RegisterDeviceResponse;
|
||||||
|
authenticate(vaultId: string, deviceId: string, token: string): boolean;
|
||||||
|
pull(vaultId: string, sinceServerRevision: number, limit?: number): SyncPullResponse;
|
||||||
|
push(request: SyncPushRequest): SyncPushResponse;
|
||||||
|
listDevices(vaultId: string): ListDevicesResponse;
|
||||||
|
revokeDevice(vaultId: string, targetDeviceId: string): RevokeDeviceResponse;
|
||||||
|
rotateVaultKey(vaultId: string, nextKeyId: string, previousKeyId?: string): RotateVaultKeyResponse;
|
||||||
|
}
|
||||||
18
apps/sync-server/tsconfig.json
Normal file
18
apps/sync-server/tsconfig.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"extends": "../../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"composite": true,
|
||||||
|
"declaration": true,
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": "src",
|
||||||
|
"module": "CommonJS"
|
||||||
|
},
|
||||||
|
"references": [
|
||||||
|
{
|
||||||
|
"path": "../../packages/sync-protocol"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"include": [
|
||||||
|
"src/**/*.ts"
|
||||||
|
]
|
||||||
|
}
|
||||||
31
docs/architecture.md
Normal file
31
docs/architecture.md
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
- `apps/obsidian-plugin`: Obsidian client plugin for local vault scanning, sync orchestration, merge handling, and user notifications.
|
||||||
|
- `apps/sync-server`: Self-hosted HTTP API for device registration and encrypted revision exchange.
|
||||||
|
- `packages/sync-protocol`: Shared schemas and request and response types.
|
||||||
|
- `packages/sync-engine`: Shared merge, hashing, and crypto helpers.
|
||||||
|
|
||||||
|
## Current Data Flow
|
||||||
|
|
||||||
|
1. The plugin registers a device with the server and stores the returned device identity and token.
|
||||||
|
2. The plugin creates or imports a local vault key.
|
||||||
|
3. The plugin pulls encrypted remote changes and applies them locally for both text notes and binary attachments.
|
||||||
|
4. Pull operations are paged so the client can resume from the last applied server revision after an interruption.
|
||||||
|
5. The plugin scans syncable vault files, hashes changed content, encrypts it, and pushes encrypted revisions in batches.
|
||||||
|
6. The server stores ciphertext, revision metadata, tombstones, device state, and active key status in a file-backed state store.
|
||||||
|
7. Each sync run emits client-side diagnostics that can be uploaded to the server and correlated with request logs through request IDs.
|
||||||
|
8. Recovery bundles package the vault ID, server URL, and current client-side vault key so a new device can join the same encrypted vault.
|
||||||
|
|
||||||
|
## Conflict Policy
|
||||||
|
|
||||||
|
- If only one side changed since the last synced base, accept the changed side.
|
||||||
|
- If both sides changed, write conflict markers into the local file, create a separate remote conflict copy, and require user review before upload continues.
|
||||||
|
|
||||||
|
## Known Gaps
|
||||||
|
|
||||||
|
- The server storage backend is durable but still file-based rather than relational.
|
||||||
|
- Key distribution between devices still depends on manually sharing a recovery bundle.
|
||||||
|
- Full plugin integration tests across multiple Obsidian instances are not implemented yet.
|
||||||
|
|
||||||
50
docs/deployment.md
Normal file
50
docs/deployment.md
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
# Deployment
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Node.js 22 or newer
|
||||||
|
- npm 10 or newer
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
## Run The Sync Server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run dev:server
|
||||||
|
```
|
||||||
|
|
||||||
|
The server listens on `http://localhost:8787` by default.
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
- `PORT`: overrides the default HTTP port.
|
||||||
|
- `SYNC_DATA_DIR`: overrides the default data directory. If omitted, the server writes data under `./data` from the workspace root.
|
||||||
|
|
||||||
|
## Data Written By The Server
|
||||||
|
|
||||||
|
- `data/sync-state.json`: durable encrypted revision metadata, devices, tombstones, and sync history.
|
||||||
|
- `data/logs/server.jsonl`: structured JSON-line server and client-sync diagnostics.
|
||||||
|
|
||||||
|
## Management Capabilities
|
||||||
|
|
||||||
|
- `POST /api/devices/list`: lists known devices for the vault plus active key status.
|
||||||
|
- `POST /api/devices/revoke`: revokes a device so it can no longer authenticate.
|
||||||
|
- `POST /api/keys/rotate`: stores the currently active vault key identifier.
|
||||||
|
- `POST /api/sync/pull`: supports paged pulls through `sinceServerRevision` and `limit`.
|
||||||
|
|
||||||
|
## Operational Notes
|
||||||
|
|
||||||
|
- The server never needs plaintext note bodies. Clients encrypt before upload and decrypt after download.
|
||||||
|
- Request logs include a response header named `x-request-id` for correlating client and server failures.
|
||||||
|
- The current storage backend is file-based. It is durable for a single-node deployment but not yet designed for clustered or high-availability hosting.
|
||||||
|
- The plugin updates local sync state after each accepted push batch and each applied pull page, which lets interrupted sync runs resume from the last durable checkpoint.
|
||||||
1723
package-lock.json
generated
Normal file
1723
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
20
package.json
Normal file
20
package.json
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"name": "obsidian-sync",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"workspaces": [
|
||||||
|
"apps/*",
|
||||||
|
"packages/*"
|
||||||
|
],
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc -b tsconfig.json",
|
||||||
|
"test": "tsx --test tests/**/*.test.ts",
|
||||||
|
"typecheck": "tsc -b tsconfig.json --pretty false",
|
||||||
|
"dev:server": "npm run dev --workspace @obsidian-sync/sync-server"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/node": "^22.14.1",
|
||||||
|
"tsx": "^4.19.3",
|
||||||
|
"typescript": "^5.8.3"
|
||||||
|
}
|
||||||
|
}
|
||||||
14
packages/sync-engine/package.json
Normal file
14
packages/sync-engine/package.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"name": "@obsidian-sync/sync-engine",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"types": "dist/index.d.ts",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc -b",
|
||||||
|
"typecheck": "tsc -b --pretty false"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@obsidian-sync/sync-protocol": "0.1.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
182
packages/sync-engine/src/crypto.ts
Normal file
182
packages/sync-engine/src/crypto.ts
Normal file
@@ -0,0 +1,182 @@
|
|||||||
|
import type { EncryptionEnvelope } from "@obsidian-sync/sync-protocol";
|
||||||
|
|
||||||
|
export interface VaultKeyHandle {
|
||||||
|
keyId: string;
|
||||||
|
key: CryptoKey;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface GeneratedVaultKey extends VaultKeyHandle {
|
||||||
|
exportedKey: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const BASE64_ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
||||||
|
|
||||||
|
function getCryptoApi(): Crypto {
|
||||||
|
if (!globalThis.crypto?.subtle) {
|
||||||
|
throw new Error("Web Crypto API is unavailable in this environment.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return globalThis.crypto;
|
||||||
|
}
|
||||||
|
|
||||||
|
function bytesToBase64(bytes: Uint8Array): string {
|
||||||
|
let output = "";
|
||||||
|
|
||||||
|
for (let index = 0; index < bytes.length; index += 3) {
|
||||||
|
const first = bytes[index] ?? 0;
|
||||||
|
const second = bytes[index + 1] ?? 0;
|
||||||
|
const third = bytes[index + 2] ?? 0;
|
||||||
|
const combined = (first << 16) | (second << 8) | third;
|
||||||
|
|
||||||
|
output += BASE64_ALPHABET[(combined >> 18) & 63];
|
||||||
|
output += BASE64_ALPHABET[(combined >> 12) & 63];
|
||||||
|
output += index + 1 < bytes.length ? BASE64_ALPHABET[(combined >> 6) & 63] : "=";
|
||||||
|
output += index + 2 < bytes.length ? BASE64_ALPHABET[combined & 63] : "=";
|
||||||
|
}
|
||||||
|
|
||||||
|
return output;
|
||||||
|
}
|
||||||
|
|
||||||
|
function base64ToBytes(value: string): Uint8Array {
|
||||||
|
const normalized = value.replace(/\s+/g, "");
|
||||||
|
const padding = normalized.endsWith("==") ? 2 : normalized.endsWith("=") ? 1 : 0;
|
||||||
|
const outputLength = Math.floor((normalized.length * 3) / 4) - padding;
|
||||||
|
const bytes = new Uint8Array(outputLength);
|
||||||
|
|
||||||
|
let byteIndex = 0;
|
||||||
|
for (let index = 0; index < normalized.length; index += 4) {
|
||||||
|
const first = BASE64_ALPHABET.indexOf(normalized[index] ?? "A");
|
||||||
|
const second = BASE64_ALPHABET.indexOf(normalized[index + 1] ?? "A");
|
||||||
|
const thirdChar = normalized[index + 2] ?? "=";
|
||||||
|
const fourthChar = normalized[index + 3] ?? "=";
|
||||||
|
const third = thirdChar === "=" ? 0 : BASE64_ALPHABET.indexOf(thirdChar);
|
||||||
|
const fourth = fourthChar === "=" ? 0 : BASE64_ALPHABET.indexOf(fourthChar);
|
||||||
|
const combined = (first << 18) | (second << 12) | (third << 6) | fourth;
|
||||||
|
|
||||||
|
bytes[byteIndex] = (combined >> 16) & 255;
|
||||||
|
byteIndex += 1;
|
||||||
|
|
||||||
|
if (thirdChar !== "=" && byteIndex < outputLength + 1) {
|
||||||
|
bytes[byteIndex] = (combined >> 8) & 255;
|
||||||
|
byteIndex += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fourthChar !== "=" && byteIndex < outputLength + 1) {
|
||||||
|
bytes[byteIndex] = combined & 255;
|
||||||
|
byteIndex += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
function toOwnedBytes(bytes: Uint8Array): Uint8Array<ArrayBuffer> {
|
||||||
|
const buffer = new ArrayBuffer(bytes.byteLength);
|
||||||
|
const view = new Uint8Array(buffer);
|
||||||
|
view.set(bytes);
|
||||||
|
return view;
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeBytes(value: ArrayBuffer | Uint8Array): Uint8Array<ArrayBuffer> {
|
||||||
|
return value instanceof Uint8Array ? toOwnedBytes(value) : toOwnedBytes(new Uint8Array(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
function toArrayBuffer(bytes: Uint8Array): ArrayBuffer {
|
||||||
|
return toOwnedBytes(bytes).buffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function generateVaultKey(keyId = globalThis.crypto.randomUUID()): Promise<GeneratedVaultKey> {
|
||||||
|
const cryptoApi = getCryptoApi();
|
||||||
|
const key = await cryptoApi.subtle.generateKey(
|
||||||
|
{
|
||||||
|
name: "AES-GCM",
|
||||||
|
length: 256
|
||||||
|
},
|
||||||
|
true,
|
||||||
|
["encrypt", "decrypt"]
|
||||||
|
);
|
||||||
|
|
||||||
|
const rawKey = await cryptoApi.subtle.exportKey("raw", key);
|
||||||
|
|
||||||
|
return {
|
||||||
|
keyId,
|
||||||
|
key,
|
||||||
|
exportedKey: bytesToBase64(new Uint8Array(rawKey))
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function importVaultKey(exportedKey: string, keyId: string): Promise<VaultKeyHandle> {
|
||||||
|
const cryptoApi = getCryptoApi();
|
||||||
|
const rawKey = base64ToBytes(exportedKey);
|
||||||
|
const key = await cryptoApi.subtle.importKey(
|
||||||
|
"raw",
|
||||||
|
toArrayBuffer(rawKey),
|
||||||
|
{
|
||||||
|
name: "AES-GCM",
|
||||||
|
length: 256
|
||||||
|
},
|
||||||
|
true,
|
||||||
|
["encrypt", "decrypt"]
|
||||||
|
);
|
||||||
|
|
||||||
|
return {
|
||||||
|
keyId,
|
||||||
|
key
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function encryptBytes(plaintext: ArrayBuffer | Uint8Array, vaultKey: VaultKeyHandle): Promise<EncryptionEnvelope> {
|
||||||
|
const cryptoApi = getCryptoApi();
|
||||||
|
const iv = cryptoApi.getRandomValues(new Uint8Array(12));
|
||||||
|
const ciphertext = await cryptoApi.subtle.encrypt(
|
||||||
|
{
|
||||||
|
name: "AES-GCM",
|
||||||
|
iv
|
||||||
|
},
|
||||||
|
vaultKey.key,
|
||||||
|
normalizeBytes(plaintext)
|
||||||
|
);
|
||||||
|
|
||||||
|
return {
|
||||||
|
algorithm: "AES-GCM-256",
|
||||||
|
keyId: vaultKey.keyId,
|
||||||
|
iv: bytesToBase64(iv),
|
||||||
|
ciphertext: bytesToBase64(new Uint8Array(ciphertext))
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function decryptBytes(envelope: EncryptionEnvelope, vaultKey: VaultKeyHandle): Promise<ArrayBuffer> {
|
||||||
|
const cryptoApi = getCryptoApi();
|
||||||
|
const plaintext = await cryptoApi.subtle.decrypt(
|
||||||
|
{
|
||||||
|
name: "AES-GCM",
|
||||||
|
iv: toOwnedBytes(base64ToBytes(envelope.iv))
|
||||||
|
},
|
||||||
|
vaultKey.key,
|
||||||
|
toOwnedBytes(base64ToBytes(envelope.ciphertext))
|
||||||
|
);
|
||||||
|
|
||||||
|
return plaintext.slice(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function encryptText(plaintext: string, vaultKey: VaultKeyHandle): Promise<EncryptionEnvelope> {
|
||||||
|
return encryptBytes(new TextEncoder().encode(plaintext), vaultKey);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function decryptText(envelope: EncryptionEnvelope, vaultKey: VaultKeyHandle): Promise<string> {
|
||||||
|
return new TextDecoder().decode(await decryptBytes(envelope, vaultKey));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function computeHash(bytes: Uint8Array): Promise<string> {
|
||||||
|
const cryptoApi = getCryptoApi();
|
||||||
|
const digest = await cryptoApi.subtle.digest("SHA-256", toOwnedBytes(bytes));
|
||||||
|
return bytesToBase64(new Uint8Array(digest));
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function computeTextHash(value: string): Promise<string> {
|
||||||
|
return computeHash(new TextEncoder().encode(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function computeBinaryHash(value: ArrayBuffer | Uint8Array): Promise<string> {
|
||||||
|
return computeHash(normalizeBytes(value));
|
||||||
|
}
|
||||||
2
packages/sync-engine/src/index.ts
Normal file
2
packages/sync-engine/src/index.ts
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
export * from "./crypto";
|
||||||
|
export * from "./merge";
|
||||||
59
packages/sync-engine/src/merge.ts
Normal file
59
packages/sync-engine/src/merge.ts
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
export interface MergeTextInput {
|
||||||
|
base?: string;
|
||||||
|
local: string;
|
||||||
|
remote: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface MergeTextResult {
|
||||||
|
status: "merged" | "conflict";
|
||||||
|
content: string;
|
||||||
|
reason?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function mergeTextRevisions(input: MergeTextInput): MergeTextResult {
|
||||||
|
const base = input.base ?? "";
|
||||||
|
|
||||||
|
if (input.local === input.remote) {
|
||||||
|
return {
|
||||||
|
status: "merged",
|
||||||
|
content: input.local
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (base === input.local) {
|
||||||
|
return {
|
||||||
|
status: "merged",
|
||||||
|
content: input.remote
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (base === input.remote) {
|
||||||
|
return {
|
||||||
|
status: "merged",
|
||||||
|
content: input.local
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
status: "conflict",
|
||||||
|
reason: "Both local and remote content changed since the last shared base.",
|
||||||
|
content: [
|
||||||
|
"<<<<<<< LOCAL",
|
||||||
|
input.local,
|
||||||
|
"=======",
|
||||||
|
input.remote,
|
||||||
|
">>>>>>> REMOTE"
|
||||||
|
].join("\n")
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createConflictCopyPath(path: string): string {
|
||||||
|
const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
|
||||||
|
const extensionIndex = path.lastIndexOf(".");
|
||||||
|
|
||||||
|
if (extensionIndex <= 0) {
|
||||||
|
return `${path}.remote-conflict.${timestamp}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
return `${path.slice(0, extensionIndex)}.remote-conflict.${timestamp}${path.slice(extensionIndex)}`;
|
||||||
|
}
|
||||||
18
packages/sync-engine/tsconfig.json
Normal file
18
packages/sync-engine/tsconfig.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"extends": "../../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"composite": true,
|
||||||
|
"declaration": true,
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": "src",
|
||||||
|
"module": "CommonJS"
|
||||||
|
},
|
||||||
|
"references": [
|
||||||
|
{
|
||||||
|
"path": "../sync-protocol"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"include": [
|
||||||
|
"src/**/*.ts"
|
||||||
|
]
|
||||||
|
}
|
||||||
14
packages/sync-protocol/package.json
Normal file
14
packages/sync-protocol/package.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"name": "@obsidian-sync/sync-protocol",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.1.0",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"types": "dist/index.d.ts",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc -b",
|
||||||
|
"typecheck": "tsc -b --pretty false"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"zod": "^3.24.2"
|
||||||
|
}
|
||||||
|
}
|
||||||
194
packages/sync-protocol/src/index.ts
Normal file
194
packages/sync-protocol/src/index.ts
Normal file
@@ -0,0 +1,194 @@
|
|||||||
|
import { z } from "zod";
|
||||||
|
|
||||||
|
export const SyncFileKindSchema = z.enum(["text", "binary"]);
|
||||||
|
export type SyncFileKind = z.infer<typeof SyncFileKindSchema>;
|
||||||
|
|
||||||
|
export const EncryptionEnvelopeSchema = z.object({
|
||||||
|
algorithm: z.literal("AES-GCM-256"),
|
||||||
|
keyId: z.string().min(1),
|
||||||
|
iv: z.string().min(1),
|
||||||
|
ciphertext: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type EncryptionEnvelope = z.infer<typeof EncryptionEnvelopeSchema>;
|
||||||
|
|
||||||
|
export const FileManifestSchema = z.object({
|
||||||
|
path: z.string().min(1),
|
||||||
|
kind: SyncFileKindSchema,
|
||||||
|
contentHash: z.string().min(1),
|
||||||
|
revisionId: z.string().min(1),
|
||||||
|
baseRevisionId: z.string().min(1).optional(),
|
||||||
|
updatedAt: z.string().datetime(),
|
||||||
|
sizeBytes: z.number().int().nonnegative(),
|
||||||
|
deviceId: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type FileManifest = z.infer<typeof FileManifestSchema>;
|
||||||
|
|
||||||
|
export const SyncFileRecordSchema = z.object({
|
||||||
|
manifest: FileManifestSchema,
|
||||||
|
envelope: EncryptionEnvelopeSchema
|
||||||
|
});
|
||||||
|
export type SyncFileRecord = z.infer<typeof SyncFileRecordSchema>;
|
||||||
|
|
||||||
|
export const TombstoneSchema = z.object({
|
||||||
|
path: z.string().min(1),
|
||||||
|
revisionId: z.string().min(1),
|
||||||
|
baseRevisionId: z.string().min(1).optional(),
|
||||||
|
deletedAt: z.string().datetime(),
|
||||||
|
deviceId: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type Tombstone = z.infer<typeof TombstoneSchema>;
|
||||||
|
|
||||||
|
export const SyncConflictSchema = z.object({
|
||||||
|
path: z.string().min(1),
|
||||||
|
serverRevisionId: z.string().min(1),
|
||||||
|
clientRevisionId: z.string().min(1),
|
||||||
|
reason: z.enum(["revision-mismatch"])
|
||||||
|
});
|
||||||
|
export type SyncConflict = z.infer<typeof SyncConflictSchema>;
|
||||||
|
|
||||||
|
export const SyncChangeSchema = z
|
||||||
|
.object({
|
||||||
|
serverRevision: z.number().int().nonnegative(),
|
||||||
|
file: SyncFileRecordSchema.optional(),
|
||||||
|
tombstone: TombstoneSchema.optional()
|
||||||
|
})
|
||||||
|
.refine((value) => Number(Boolean(value.file)) + Number(Boolean(value.tombstone)) === 1, {
|
||||||
|
message: "A change must include either a file or a tombstone."
|
||||||
|
});
|
||||||
|
export type SyncChange = z.infer<typeof SyncChangeSchema>;
|
||||||
|
|
||||||
|
export const RegisterDeviceRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceName: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type RegisterDeviceRequest = z.infer<typeof RegisterDeviceRequestSchema>;
|
||||||
|
|
||||||
|
export const RegisterDeviceResponseSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
token: z.string().min(1),
|
||||||
|
issuedAt: z.string().datetime()
|
||||||
|
});
|
||||||
|
export type RegisterDeviceResponse = z.infer<typeof RegisterDeviceResponseSchema>;
|
||||||
|
|
||||||
|
export const RecoveryBundleSchema = z.object({
|
||||||
|
version: z.literal(1),
|
||||||
|
serverUrl: z.string().min(1),
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
keyId: z.string().min(1),
|
||||||
|
exportedVaultKey: z.string().min(1),
|
||||||
|
generatedAt: z.string().datetime()
|
||||||
|
});
|
||||||
|
export type RecoveryBundle = z.infer<typeof RecoveryBundleSchema>;
|
||||||
|
|
||||||
|
export const DeviceRecordSchema = z.object({
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
deviceName: z.string().min(1),
|
||||||
|
issuedAt: z.string().datetime(),
|
||||||
|
revokedAt: z.string().datetime().optional()
|
||||||
|
});
|
||||||
|
export type DeviceRecord = z.infer<typeof DeviceRecordSchema>;
|
||||||
|
|
||||||
|
export const ListDevicesRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type ListDevicesRequest = z.infer<typeof ListDevicesRequestSchema>;
|
||||||
|
|
||||||
|
export const ListDevicesResponseSchema = z.object({
|
||||||
|
devices: z.array(DeviceRecordSchema),
|
||||||
|
activeKeyId: z.string().min(1).optional(),
|
||||||
|
keyRotatedAt: z.string().datetime().optional()
|
||||||
|
});
|
||||||
|
export type ListDevicesResponse = z.infer<typeof ListDevicesResponseSchema>;
|
||||||
|
|
||||||
|
export const RevokeDeviceRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
targetDeviceId: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type RevokeDeviceRequest = z.infer<typeof RevokeDeviceRequestSchema>;
|
||||||
|
|
||||||
|
export const RevokeDeviceResponseSchema = z.object({
|
||||||
|
targetDeviceId: z.string().min(1),
|
||||||
|
revokedAt: z.string().datetime()
|
||||||
|
});
|
||||||
|
export type RevokeDeviceResponse = z.infer<typeof RevokeDeviceResponseSchema>;
|
||||||
|
|
||||||
|
export const RotateVaultKeyRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
nextKeyId: z.string().min(1),
|
||||||
|
previousKeyId: z.string().min(1).optional()
|
||||||
|
});
|
||||||
|
export type RotateVaultKeyRequest = z.infer<typeof RotateVaultKeyRequestSchema>;
|
||||||
|
|
||||||
|
export const RotateVaultKeyResponseSchema = z.object({
|
||||||
|
activeKeyId: z.string().min(1),
|
||||||
|
rotatedAt: z.string().datetime()
|
||||||
|
});
|
||||||
|
export type RotateVaultKeyResponse = z.infer<typeof RotateVaultKeyResponseSchema>;
|
||||||
|
|
||||||
|
export const SyncPullRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
sinceServerRevision: z.number().int().nonnegative(),
|
||||||
|
limit: z.number().int().positive().max(500).optional()
|
||||||
|
});
|
||||||
|
export type SyncPullRequest = z.infer<typeof SyncPullRequestSchema>;
|
||||||
|
|
||||||
|
export const SyncPullResponseSchema = z.object({
|
||||||
|
serverRevision: z.number().int().nonnegative(),
|
||||||
|
changes: z.array(SyncChangeSchema),
|
||||||
|
hasMore: z.boolean(),
|
||||||
|
nextSinceServerRevision: z.number().int().nonnegative(),
|
||||||
|
activeKeyId: z.string().min(1).optional(),
|
||||||
|
keyRotatedAt: z.string().datetime().optional()
|
||||||
|
});
|
||||||
|
export type SyncPullResponse = z.infer<typeof SyncPullResponseSchema>;
|
||||||
|
|
||||||
|
export const SyncPushRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
knownServerRevision: z.number().int().nonnegative(),
|
||||||
|
files: z.array(SyncFileRecordSchema),
|
||||||
|
tombstones: z.array(TombstoneSchema)
|
||||||
|
});
|
||||||
|
export type SyncPushRequest = z.infer<typeof SyncPushRequestSchema>;
|
||||||
|
|
||||||
|
export const SyncPushResponseSchema = z.object({
|
||||||
|
acceptedServerRevision: z.number().int().nonnegative(),
|
||||||
|
acceptedFilePaths: z.array(z.string()),
|
||||||
|
acceptedTombstones: z.array(z.string()),
|
||||||
|
conflicts: z.array(SyncConflictSchema),
|
||||||
|
activeKeyId: z.string().min(1).optional(),
|
||||||
|
keyRotatedAt: z.string().datetime().optional()
|
||||||
|
});
|
||||||
|
export type SyncPushResponse = z.infer<typeof SyncPushResponseSchema>;
|
||||||
|
|
||||||
|
export const ClientLogEntrySchema = z.object({
|
||||||
|
level: z.enum(["debug", "info", "warn", "error"]),
|
||||||
|
message: z.string().min(1),
|
||||||
|
timestamp: z.string().datetime(),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
context: z.record(z.string(), z.unknown()).optional()
|
||||||
|
});
|
||||||
|
export type ClientLogEntry = z.infer<typeof ClientLogEntrySchema>;
|
||||||
|
|
||||||
|
export const ClientLogUploadRequestSchema = z.object({
|
||||||
|
vaultId: z.string().min(1),
|
||||||
|
deviceId: z.string().min(1),
|
||||||
|
runId: z.string().min(1),
|
||||||
|
entries: z.array(ClientLogEntrySchema)
|
||||||
|
});
|
||||||
|
export type ClientLogUploadRequest = z.infer<typeof ClientLogUploadRequestSchema>;
|
||||||
|
|
||||||
|
export const ClientLogUploadResponseSchema = z.object({
|
||||||
|
accepted: z.number().int().nonnegative(),
|
||||||
|
requestId: z.string().min(1)
|
||||||
|
});
|
||||||
|
export type ClientLogUploadResponse = z.infer<typeof ClientLogUploadResponseSchema>;
|
||||||
|
|
||||||
|
export function parseWithSchema<T>(schema: z.ZodSchema<T>, value: unknown): T {
|
||||||
|
return schema.parse(value);
|
||||||
|
}
|
||||||
13
packages/sync-protocol/tsconfig.json
Normal file
13
packages/sync-protocol/tsconfig.json
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"extends": "../../tsconfig.base.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"composite": true,
|
||||||
|
"declaration": true,
|
||||||
|
"outDir": "dist",
|
||||||
|
"rootDir": "src",
|
||||||
|
"module": "CommonJS"
|
||||||
|
},
|
||||||
|
"include": [
|
||||||
|
"src/**/*.ts"
|
||||||
|
]
|
||||||
|
}
|
||||||
39
tests/contract/sync-engine.test.ts
Normal file
39
tests/contract/sync-engine.test.ts
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import test from "node:test";
|
||||||
|
|
||||||
|
import {
|
||||||
|
computeBinaryHash,
|
||||||
|
decryptBytes,
|
||||||
|
encryptBytes,
|
||||||
|
generateVaultKey,
|
||||||
|
mergeTextRevisions
|
||||||
|
} from "../../packages/sync-engine/src";
|
||||||
|
|
||||||
|
test("encryptBytes round-trips binary payloads", async () => {
|
||||||
|
const vaultKey = await generateVaultKey("test-key");
|
||||||
|
const original = new Uint8Array([1, 2, 3, 4, 5]).buffer;
|
||||||
|
|
||||||
|
const encrypted = await encryptBytes(original, vaultKey);
|
||||||
|
const decrypted = await decryptBytes(encrypted, vaultKey);
|
||||||
|
|
||||||
|
assert.deepEqual([...new Uint8Array(decrypted)], [1, 2, 3, 4, 5]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("computeBinaryHash is stable for identical binary payloads", async () => {
|
||||||
|
const first = await computeBinaryHash(new Uint8Array([9, 8, 7, 6]).buffer);
|
||||||
|
const second = await computeBinaryHash(new Uint8Array([9, 8, 7, 6]).buffer);
|
||||||
|
|
||||||
|
assert.equal(first, second);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("mergeTextRevisions reports conflicts when both sides diverge", () => {
|
||||||
|
const result = mergeTextRevisions({
|
||||||
|
base: "alpha",
|
||||||
|
local: "alpha local",
|
||||||
|
remote: "alpha remote"
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.equal(result.status, "conflict");
|
||||||
|
assert.match(result.content, /<<<<<<< LOCAL/);
|
||||||
|
assert.match(result.content, />>>>>>> REMOTE/);
|
||||||
|
});
|
||||||
174
tests/e2e/multi-device-store.test.ts
Normal file
174
tests/e2e/multi-device-store.test.ts
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
import test from "node:test";
|
||||||
|
|
||||||
|
import { createFileSyncStore } from "../../apps/sync-server/src/store/fileSyncStore";
|
||||||
|
|
||||||
|
test("file-backed store survives restart and serves another device", () => {
|
||||||
|
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-"));
|
||||||
|
const storePath = path.join(tempDirectory, "sync-state.json");
|
||||||
|
|
||||||
|
try {
|
||||||
|
const serverA = createFileSyncStore(storePath);
|
||||||
|
const deviceA = serverA.registerDevice({
|
||||||
|
vaultId: "vault-1",
|
||||||
|
deviceName: "Desktop"
|
||||||
|
});
|
||||||
|
const deviceB = serverA.registerDevice({
|
||||||
|
vaultId: "vault-1",
|
||||||
|
deviceName: "Phone"
|
||||||
|
});
|
||||||
|
|
||||||
|
const pushResponse = serverA.push({
|
||||||
|
vaultId: "vault-1",
|
||||||
|
deviceId: deviceA.deviceId,
|
||||||
|
knownServerRevision: 0,
|
||||||
|
files: [
|
||||||
|
{
|
||||||
|
manifest: {
|
||||||
|
path: "notes/hello.md",
|
||||||
|
kind: "text",
|
||||||
|
contentHash: "hash-1",
|
||||||
|
revisionId: "rev-1",
|
||||||
|
updatedAt: "2026-04-08T00:00:00.000Z",
|
||||||
|
sizeBytes: 12,
|
||||||
|
deviceId: deviceA.deviceId
|
||||||
|
},
|
||||||
|
envelope: {
|
||||||
|
algorithm: "AES-GCM-256",
|
||||||
|
keyId: "key-1",
|
||||||
|
iv: "iv-1",
|
||||||
|
ciphertext: "cipher-1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
tombstones: []
|
||||||
|
});
|
||||||
|
|
||||||
|
assert.deepEqual(pushResponse.acceptedFilePaths, ["notes/hello.md"]);
|
||||||
|
|
||||||
|
const serverB = createFileSyncStore(storePath);
|
||||||
|
assert.equal(serverB.authenticate("vault-1", deviceB.deviceId, deviceB.token), true);
|
||||||
|
|
||||||
|
const pullResponse = serverB.pull("vault-1", 0);
|
||||||
|
assert.equal(pullResponse.changes.length, 1);
|
||||||
|
assert.equal(pullResponse.changes[0].file?.manifest.path, "notes/hello.md");
|
||||||
|
assert.equal(pullResponse.serverRevision, 1);
|
||||||
|
assert.equal(pullResponse.hasMore, false);
|
||||||
|
assert.equal(pullResponse.nextSinceServerRevision, 1);
|
||||||
|
} finally {
|
||||||
|
rmSync(tempDirectory, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("file-backed store paginates pull results", () => {
|
||||||
|
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-page-"));
|
||||||
|
const storePath = path.join(tempDirectory, "sync-state.json");
|
||||||
|
|
||||||
|
try {
|
||||||
|
const store = createFileSyncStore(storePath);
|
||||||
|
const device = store.registerDevice({
|
||||||
|
vaultId: "vault-2",
|
||||||
|
deviceName: "Desktop"
|
||||||
|
});
|
||||||
|
|
||||||
|
store.push({
|
||||||
|
vaultId: "vault-2",
|
||||||
|
deviceId: device.deviceId,
|
||||||
|
knownServerRevision: 0,
|
||||||
|
files: [
|
||||||
|
{
|
||||||
|
manifest: {
|
||||||
|
path: "a.md",
|
||||||
|
kind: "text",
|
||||||
|
contentHash: "hash-a",
|
||||||
|
revisionId: "rev-a",
|
||||||
|
updatedAt: "2026-04-08T00:00:00.000Z",
|
||||||
|
sizeBytes: 1,
|
||||||
|
deviceId: device.deviceId
|
||||||
|
},
|
||||||
|
envelope: {
|
||||||
|
algorithm: "AES-GCM-256",
|
||||||
|
keyId: "key-a",
|
||||||
|
iv: "iv-a",
|
||||||
|
ciphertext: "cipher-a"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
tombstones: []
|
||||||
|
});
|
||||||
|
|
||||||
|
store.push({
|
||||||
|
vaultId: "vault-2",
|
||||||
|
deviceId: device.deviceId,
|
||||||
|
knownServerRevision: 1,
|
||||||
|
files: [
|
||||||
|
{
|
||||||
|
manifest: {
|
||||||
|
path: "b.md",
|
||||||
|
kind: "text",
|
||||||
|
contentHash: "hash-b",
|
||||||
|
revisionId: "rev-b",
|
||||||
|
updatedAt: "2026-04-08T00:01:00.000Z",
|
||||||
|
sizeBytes: 1,
|
||||||
|
deviceId: device.deviceId
|
||||||
|
},
|
||||||
|
envelope: {
|
||||||
|
algorithm: "AES-GCM-256",
|
||||||
|
keyId: "key-a",
|
||||||
|
iv: "iv-b",
|
||||||
|
ciphertext: "cipher-b"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
tombstones: []
|
||||||
|
});
|
||||||
|
|
||||||
|
const firstPage = store.pull("vault-2", 0, 1);
|
||||||
|
assert.equal(firstPage.changes.length, 1);
|
||||||
|
assert.equal(firstPage.hasMore, true);
|
||||||
|
assert.equal(firstPage.nextSinceServerRevision, 1);
|
||||||
|
|
||||||
|
const secondPage = store.pull("vault-2", firstPage.nextSinceServerRevision, 1);
|
||||||
|
assert.equal(secondPage.changes.length, 1);
|
||||||
|
assert.equal(secondPage.hasMore, false);
|
||||||
|
assert.equal(secondPage.nextSinceServerRevision, 2);
|
||||||
|
assert.equal(secondPage.changes[0].file?.manifest.path, "b.md");
|
||||||
|
} finally {
|
||||||
|
rmSync(tempDirectory, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("device revocation and key rotation state are durable", () => {
|
||||||
|
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-device-"));
|
||||||
|
const storePath = path.join(tempDirectory, "sync-state.json");
|
||||||
|
|
||||||
|
try {
|
||||||
|
const store = createFileSyncStore(storePath);
|
||||||
|
const desktop = store.registerDevice({
|
||||||
|
vaultId: "vault-3",
|
||||||
|
deviceName: "Desktop"
|
||||||
|
});
|
||||||
|
const phone = store.registerDevice({
|
||||||
|
vaultId: "vault-3",
|
||||||
|
deviceName: "Phone"
|
||||||
|
});
|
||||||
|
|
||||||
|
const rotation = store.rotateVaultKey("vault-3", "key-rotated");
|
||||||
|
assert.equal(rotation.activeKeyId, "key-rotated");
|
||||||
|
|
||||||
|
const revoked = store.revokeDevice("vault-3", phone.deviceId);
|
||||||
|
assert.equal(revoked.targetDeviceId, phone.deviceId);
|
||||||
|
assert.equal(store.authenticate("vault-3", phone.deviceId, phone.token), false);
|
||||||
|
|
||||||
|
const restartedStore = createFileSyncStore(storePath);
|
||||||
|
const devices = restartedStore.listDevices("vault-3");
|
||||||
|
assert.equal(devices.activeKeyId, "key-rotated");
|
||||||
|
assert.equal(devices.devices.find((device) => device.deviceId === phone.deviceId)?.revokedAt !== undefined, true);
|
||||||
|
assert.equal(restartedStore.authenticate("vault-3", desktop.deviceId, desktop.token), true);
|
||||||
|
} finally {
|
||||||
|
rmSync(tempDirectory, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
25
tsconfig.base.json
Normal file
25
tsconfig.base.json
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"lib": [
|
||||||
|
"ES2022",
|
||||||
|
"DOM"
|
||||||
|
],
|
||||||
|
"strict": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"baseUrl": ".",
|
||||||
|
"paths": {
|
||||||
|
"@obsidian-sync/sync-protocol": [
|
||||||
|
"packages/sync-protocol/src"
|
||||||
|
],
|
||||||
|
"@obsidian-sync/sync-engine": [
|
||||||
|
"packages/sync-engine/src"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"moduleResolution": "Node",
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"noEmitOnError": true
|
||||||
|
}
|
||||||
|
}
|
||||||
17
tsconfig.json
Normal file
17
tsconfig.json
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"files": [],
|
||||||
|
"references": [
|
||||||
|
{
|
||||||
|
"path": "./packages/sync-protocol"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": "./packages/sync-engine"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": "./apps/sync-server"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": "./apps/obsidian-plugin"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user