Initial commit
This commit is contained in:
3
.gitignore
vendored
Normal file
3
.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
node_modules/
|
||||
dist/
|
||||
*.tsbuildinfo
|
||||
44
Plan.md
Normal file
44
Plan.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Obsidian Sync Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Build a self-hosted sync system for an Obsidian vault that works across Windows, Linux, and Android. The system will use an Obsidian plugin for clients, a central sync server, and end-to-end encrypted note transfer. Conflicts should try automatic merge first, then fall back to user-visible conflict handling.
|
||||
|
||||
## Current Implementation Slice
|
||||
|
||||
1. Create a TypeScript monorepo for the plugin, server, and shared libraries.
|
||||
2. Define shared sync contracts for device registration, pull, push, tombstones, and conflict responses.
|
||||
3. Add a shared sync engine with AES-GCM encryption, hashing, and safe text merge helpers.
|
||||
4. Build a durable server that registers devices, stores encrypted note revisions on disk, and writes structured request and client-sync logs.
|
||||
5. Build an Obsidian plugin with settings, manual sync, scheduled sync, remote pull, local push, attachment sync, conflict marker handling, and sync-run diagnostics upload.
|
||||
6. Validate the scaffold with a workspace build and focused tests.
|
||||
|
||||
## Delivery Phases
|
||||
|
||||
### Phase 1
|
||||
|
||||
- Monorepo setup
|
||||
- Protocol definitions
|
||||
- Shared crypto and merge helpers
|
||||
- Minimal server API
|
||||
- Plugin settings and sync loop
|
||||
|
||||
### Phase 2
|
||||
|
||||
- Durable storage backend for the server
|
||||
- Attachment sync
|
||||
- Better device onboarding and recovery flow
|
||||
- Structured logs and user-visible sync diagnostics
|
||||
|
||||
### Phase 3
|
||||
|
||||
- Hardening for large vaults and flaky networks
|
||||
- Packaging and deployment docs
|
||||
- Cross-device integration testing
|
||||
|
||||
## Notes
|
||||
|
||||
- The server now persists encrypted sync state to a local data directory and keeps JSON-line diagnostics logs.
|
||||
- Text conflicts are handled on the client and binary conflicts produce a remote conflict copy.
|
||||
- Recovery bundle onboarding, device revocation, key rotation, and batched sync are implemented in the current repo.
|
||||
- The next highest-priority gap is broader multi-instance Obsidian integration testing and a production-grade database-backed server store.
|
||||
97
README.md
Normal file
97
README.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Obsidian Sync
|
||||
|
||||
Obsidian Sync is a self-hosted, end-to-end encrypted vault sync system for Obsidian across Windows, Linux, and Android. It consists of an Obsidian plugin, a sync server, and shared protocol and crypto packages inside a TypeScript monorepo.
|
||||
|
||||
## Features
|
||||
|
||||
- End-to-end encrypted text note and binary attachment sync
|
||||
- Automatic text merge with conflict markers when both sides change
|
||||
- Binary conflict copies when attachments diverge
|
||||
- Durable server state stored on disk
|
||||
- Structured JSON-line diagnostics and client sync log upload
|
||||
- Recovery bundle export and import for onboarding new devices
|
||||
- Device listing and revocation
|
||||
- Vault key rotation with recovery-bundle refresh
|
||||
- Paged pull and batched push behavior so interrupted sync runs can resume safely
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- `apps/obsidian-plugin`: Obsidian plugin entry point, settings UI, and sync orchestration
|
||||
- `apps/sync-server`: Express server for encrypted revision exchange and device management
|
||||
- `packages/sync-protocol`: Shared request and response schemas
|
||||
- `packages/sync-engine`: Shared crypto, hashing, and merge helpers
|
||||
- `docs`: Architecture and deployment notes
|
||||
- `tests`: Focused contract and store-level integration tests
|
||||
|
||||
## Requirements
|
||||
|
||||
- Node.js 22 or newer
|
||||
- npm 10 or newer
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
## Run The Server
|
||||
|
||||
```bash
|
||||
npm run dev:server
|
||||
```
|
||||
|
||||
The default server URL is `http://localhost:8787`.
|
||||
|
||||
## First-Time Setup
|
||||
|
||||
1. Start the sync server.
|
||||
2. Load the plugin into an Obsidian vault.
|
||||
3. Set the server URL, vault ID, and device name in the plugin settings.
|
||||
4. Run a manual sync to register the first device and generate the initial vault key.
|
||||
5. Export a recovery bundle from the settings tab and store it securely.
|
||||
|
||||
## Add Another Device
|
||||
|
||||
1. Install the plugin on the second device.
|
||||
2. Open the plugin settings and paste the recovery bundle into the import field.
|
||||
3. Run a manual sync to register that device and pull the encrypted vault contents.
|
||||
|
||||
## Rotate The Vault Key
|
||||
|
||||
1. Open the plugin settings and use `Rotate key`.
|
||||
2. Wait for the client to re-encrypt and upload the current vault contents.
|
||||
3. Export the fresh recovery bundle.
|
||||
4. Import that new bundle on every other device before the next sync.
|
||||
|
||||
## Revoke A Device
|
||||
|
||||
Use the plugin settings device list to revoke a device that should no longer have access. A revoked device can no longer authenticate to the server.
|
||||
|
||||
## Server Data
|
||||
|
||||
- `data/sync-state.json`: durable revision, device, tombstone, and key-status state
|
||||
- `data/logs/server.jsonl`: structured server and client-sync diagnostics
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- The server storage backend is file-based rather than SQLite or PostgreSQL.
|
||||
- Sync validation is focused on shared libraries and the server store; there is not yet a full Obsidian plugin integration harness.
|
||||
- Device onboarding is driven by a copy-and-paste recovery bundle instead of a richer pairing flow.
|
||||
|
||||
## Documentation
|
||||
|
||||
- `docs/architecture.md`
|
||||
- `docs/deployment.md`
|
||||
- `Plan.md`
|
||||
2
Specsheet.md
Normal file
2
Specsheet.md
Normal file
@@ -0,0 +1,2 @@
|
||||
# Description
|
||||
I need a system that syncs my obisian vault between different devices and is connected by a server. For this application, I will need a client side plugin for obisdian which works on both windows, linux, and android obsidian. I will also need a server application that will allow me to sync my notes to it and distribute it out to other devices connected. This system will require a version control system that allow the user to edit it on different devices and sort out the different edits. If there is an error create a log and alert the user.
|
||||
9
apps/obsidian-plugin/manifest.json
Normal file
9
apps/obsidian-plugin/manifest.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"id": "obsidian-sync",
|
||||
"name": "Obsidian Sync",
|
||||
"version": "0.1.0",
|
||||
"minAppVersion": "1.5.0",
|
||||
"description": "Sync a vault through a self-hosted encrypted sync server.",
|
||||
"author": "Local",
|
||||
"isDesktopOnly": false
|
||||
}
|
||||
18
apps/obsidian-plugin/package.json
Normal file
18
apps/obsidian-plugin/package.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"name": "@obsidian-sync/obsidian-plugin",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"main": "dist/main.js",
|
||||
"types": "dist/main.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsc -b",
|
||||
"typecheck": "tsc -b --pretty false"
|
||||
},
|
||||
"dependencies": {
|
||||
"@obsidian-sync/sync-engine": "0.1.0",
|
||||
"@obsidian-sync/sync-protocol": "0.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"obsidian": "^1.8.7"
|
||||
}
|
||||
}
|
||||
96
apps/obsidian-plugin/src/main.ts
Normal file
96
apps/obsidian-plugin/src/main.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
import { Notice, Plugin } from "obsidian";
|
||||
|
||||
import { SyncSettingsTab } from "./settings/SyncSettingsTab";
|
||||
import { DEFAULT_SETTINGS, type ObsidianSyncSettings } from "./settings/settings";
|
||||
import { SyncService } from "./sync/SyncService";
|
||||
|
||||
export default class ObsidianSyncPlugin extends Plugin {
|
||||
settings: ObsidianSyncSettings = DEFAULT_SETTINGS;
|
||||
|
||||
private syncService?: SyncService;
|
||||
private scheduledSyncHandle?: number;
|
||||
|
||||
async onload(): Promise<void> {
|
||||
await this.loadSettings();
|
||||
|
||||
this.syncService = new SyncService(
|
||||
this.app,
|
||||
() => this.settings,
|
||||
async (settings) => {
|
||||
this.settings = settings;
|
||||
await this.saveSettings();
|
||||
}
|
||||
);
|
||||
|
||||
this.addSettingTab(new SyncSettingsTab(this.app, this, () => this.configureSyncSchedule()));
|
||||
|
||||
this.addCommand({
|
||||
id: "obsidian-sync-run-manual",
|
||||
name: "Run encrypted sync now",
|
||||
callback: async () => {
|
||||
await this.syncService?.runManualSync();
|
||||
}
|
||||
});
|
||||
|
||||
this.addCommand({
|
||||
id: "obsidian-sync-rotate-vault-key",
|
||||
name: "Rotate vault key",
|
||||
callback: async () => {
|
||||
const result = await this.getSyncServiceOrThrow().rotateVaultKey();
|
||||
new Notice(
|
||||
`Vault key rotated. Uploaded ${result.uploadedFiles} files. Export the fresh recovery bundle to update other devices.`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
this.configureSyncSchedule();
|
||||
new Notice("Obsidian Sync loaded.");
|
||||
}
|
||||
|
||||
onunload(): void {
|
||||
if (this.scheduledSyncHandle !== undefined) {
|
||||
window.clearInterval(this.scheduledSyncHandle);
|
||||
}
|
||||
}
|
||||
|
||||
async loadSettings(): Promise<void> {
|
||||
const loaded = (await this.loadData()) as Partial<ObsidianSyncSettings> | null;
|
||||
this.settings = {
|
||||
...DEFAULT_SETTINGS,
|
||||
...loaded,
|
||||
deviceName: loaded?.deviceName || this.app.vault.getName(),
|
||||
syncState: {
|
||||
...DEFAULT_SETTINGS.syncState,
|
||||
...(loaded?.syncState ?? {}),
|
||||
files: {
|
||||
...DEFAULT_SETTINGS.syncState.files,
|
||||
...(loaded?.syncState?.files ?? {})
|
||||
}
|
||||
}
|
||||
};
|
||||
await this.saveSettings();
|
||||
}
|
||||
|
||||
async saveSettings(): Promise<void> {
|
||||
await this.saveData(this.settings);
|
||||
}
|
||||
|
||||
getSyncServiceOrThrow(): SyncService {
|
||||
if (!this.syncService) {
|
||||
throw new Error("Sync service is not ready yet.");
|
||||
}
|
||||
|
||||
return this.syncService;
|
||||
}
|
||||
|
||||
private configureSyncSchedule(): void {
|
||||
if (this.scheduledSyncHandle !== undefined) {
|
||||
window.clearInterval(this.scheduledSyncHandle);
|
||||
}
|
||||
|
||||
const intervalMinutes = Math.max(1, this.settings.syncIntervalMinutes);
|
||||
this.scheduledSyncHandle = window.setInterval(() => {
|
||||
void this.syncService?.runScheduledSync();
|
||||
}, intervalMinutes * 60_000);
|
||||
}
|
||||
}
|
||||
228
apps/obsidian-plugin/src/settings/SyncSettingsTab.ts
Normal file
228
apps/obsidian-plugin/src/settings/SyncSettingsTab.ts
Normal file
@@ -0,0 +1,228 @@
|
||||
import { Notice, PluginSettingTab, Setting, TextAreaComponent } from "obsidian";
|
||||
|
||||
import type ObsidianSyncPlugin from "../main";
|
||||
|
||||
export class SyncSettingsTab extends PluginSettingTab {
|
||||
private exportedRecoveryBundle = "";
|
||||
private importedRecoveryBundle = "";
|
||||
|
||||
constructor(
|
||||
app: ObsidianSyncPlugin["app"],
|
||||
private readonly plugin: ObsidianSyncPlugin,
|
||||
private readonly onSettingsUpdated: () => void
|
||||
) {
|
||||
super(app, plugin);
|
||||
}
|
||||
|
||||
display(): void {
|
||||
const { containerEl } = this;
|
||||
containerEl.empty();
|
||||
|
||||
containerEl.createEl("h2", { text: "Obsidian Sync" });
|
||||
const syncService = this.plugin.getSyncServiceOrThrow();
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Server URL")
|
||||
.setDesc("Base URL of the sync server.")
|
||||
.addText((text) =>
|
||||
text
|
||||
.setPlaceholder("http://localhost:8787")
|
||||
.setValue(this.plugin.settings.serverUrl)
|
||||
.onChange(async (value) => {
|
||||
this.plugin.settings.serverUrl = value.trim();
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Vault ID")
|
||||
.setDesc("Stable identifier used to group devices for the same vault.")
|
||||
.addText((text) =>
|
||||
text.setValue(this.plugin.settings.vaultId).onChange(async (value) => {
|
||||
this.plugin.settings.vaultId = value.trim();
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Device name")
|
||||
.setDesc("Human-readable name shown on the server.")
|
||||
.addText((text) =>
|
||||
text.setValue(this.plugin.settings.deviceName).onChange(async (value) => {
|
||||
this.plugin.settings.deviceName = value.trim();
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Sync interval")
|
||||
.setDesc("Minutes between scheduled sync runs.")
|
||||
.addText((text) =>
|
||||
text.setValue(String(this.plugin.settings.syncIntervalMinutes)).onChange(async (value) => {
|
||||
const minutes = Number(value);
|
||||
this.plugin.settings.syncIntervalMinutes = Number.isFinite(minutes) && minutes > 0 ? minutes : 5;
|
||||
await this.plugin.saveSettings();
|
||||
this.onSettingsUpdated();
|
||||
})
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Pull batch size")
|
||||
.setDesc("Maximum remote changes to request in a single pull page.")
|
||||
.addText((text) =>
|
||||
text.setValue(String(this.plugin.settings.pullBatchSize)).onChange(async (value) => {
|
||||
const batchSize = Number(value);
|
||||
this.plugin.settings.pullBatchSize = Number.isFinite(batchSize) && batchSize > 0 ? Math.floor(batchSize) : 50;
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Push batch size")
|
||||
.setDesc("Maximum local changes to upload in a single push request.")
|
||||
.addText((text) =>
|
||||
text.setValue(String(this.plugin.settings.pushBatchSize)).onChange(async (value) => {
|
||||
const batchSize = Number(value);
|
||||
this.plugin.settings.pushBatchSize = Number.isFinite(batchSize) && batchSize > 0 ? Math.floor(batchSize) : 50;
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
|
||||
containerEl.createEl("h3", { text: "Recovery" });
|
||||
|
||||
let exportBundleTextArea: TextAreaComponent | undefined;
|
||||
new Setting(containerEl)
|
||||
.setName("Export recovery bundle")
|
||||
.setDesc("Generate a bundle you can paste into another device to recover the same encrypted vault.")
|
||||
.addTextArea((textArea) => {
|
||||
exportBundleTextArea = textArea;
|
||||
textArea.setValue(this.exportedRecoveryBundle);
|
||||
textArea.inputEl.rows = 8;
|
||||
textArea.inputEl.cols = 40;
|
||||
})
|
||||
.addButton((button) =>
|
||||
button.setButtonText("Generate").onClick(async () => {
|
||||
this.exportedRecoveryBundle = await syncService.exportRecoveryBundle();
|
||||
exportBundleTextArea?.setValue(this.exportedRecoveryBundle);
|
||||
new Notice("Recovery bundle generated.");
|
||||
})
|
||||
)
|
||||
.addButton((button) =>
|
||||
button.setButtonText("Clear").onClick(() => {
|
||||
this.exportedRecoveryBundle = "";
|
||||
exportBundleTextArea?.setValue("");
|
||||
})
|
||||
);
|
||||
|
||||
let importBundleTextArea: TextAreaComponent | undefined;
|
||||
new Setting(containerEl)
|
||||
.setName("Import recovery bundle")
|
||||
.setDesc("Paste a bundle from another device to register this device against the same encrypted vault.")
|
||||
.addTextArea((textArea) => {
|
||||
importBundleTextArea = textArea;
|
||||
textArea.setValue(this.importedRecoveryBundle);
|
||||
textArea.inputEl.rows = 8;
|
||||
textArea.inputEl.cols = 40;
|
||||
textArea.onChange((value) => {
|
||||
this.importedRecoveryBundle = value;
|
||||
});
|
||||
})
|
||||
.addButton((button) =>
|
||||
button.setButtonText("Import").onClick(async () => {
|
||||
await syncService.importRecoveryBundle(this.importedRecoveryBundle);
|
||||
this.importedRecoveryBundle = "";
|
||||
this.exportedRecoveryBundle = "";
|
||||
importBundleTextArea?.setValue("");
|
||||
new Notice("Recovery bundle imported. Run sync to register this device.");
|
||||
this.display();
|
||||
})
|
||||
);
|
||||
|
||||
containerEl.createEl("h3", { text: "Devices" });
|
||||
const deviceListContainer = containerEl.createDiv();
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Connected devices")
|
||||
.setDesc("Refresh the active device list and revoke devices that should no longer have access.")
|
||||
.addButton((button) =>
|
||||
button.setButtonText("Refresh").onClick(async () => {
|
||||
await this.renderDeviceList(deviceListContainer);
|
||||
})
|
||||
);
|
||||
|
||||
void this.renderDeviceList(deviceListContainer);
|
||||
|
||||
containerEl.createEl("h3", { text: "Key rotation" });
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Rotate vault key")
|
||||
.setDesc("Generate a new vault key, re-encrypt local content, and require your other devices to import a fresh recovery bundle.")
|
||||
.addButton((button) =>
|
||||
button.setButtonText("Rotate key").onClick(async () => {
|
||||
const result = await syncService.rotateVaultKey();
|
||||
this.exportedRecoveryBundle = result.recoveryBundle;
|
||||
exportBundleTextArea?.setValue(result.recoveryBundle);
|
||||
new Notice(
|
||||
`Vault key rotated. Uploaded ${result.uploadedFiles} files. Share the fresh recovery bundle with your other devices.`
|
||||
);
|
||||
await this.renderDeviceList(deviceListContainer);
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
private async renderDeviceList(containerEl: HTMLElement): Promise<void> {
|
||||
containerEl.empty();
|
||||
|
||||
if (!this.plugin.settings.deviceId || !this.plugin.settings.authToken) {
|
||||
containerEl.createEl("p", {
|
||||
text: "Run a sync or import a recovery bundle to register this device before managing devices."
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const response = await this.plugin.getSyncServiceOrThrow().listDevices();
|
||||
const summaryParts = [`Active server key: ${response.activeKeyId ?? "not set"}`];
|
||||
if (response.keyRotatedAt) {
|
||||
summaryParts.push(`Last key rotation: ${response.keyRotatedAt}`);
|
||||
}
|
||||
|
||||
containerEl.createEl("p", {
|
||||
text: summaryParts.join(" | ")
|
||||
});
|
||||
|
||||
for (const device of response.devices) {
|
||||
const isCurrentDevice = device.deviceId === this.plugin.settings.deviceId;
|
||||
const description = [
|
||||
`ID: ${device.deviceId}`,
|
||||
`Registered: ${device.issuedAt}`,
|
||||
device.revokedAt ? `Revoked: ${device.revokedAt}` : "Active"
|
||||
].join(" | ");
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName(`${device.deviceName}${isCurrentDevice ? " (this device)" : ""}`)
|
||||
.setDesc(description)
|
||||
.addButton((button) => {
|
||||
if (device.revokedAt) {
|
||||
button.setButtonText("Revoked").setDisabled(true);
|
||||
return;
|
||||
}
|
||||
|
||||
button.setButtonText(isCurrentDevice ? "Revoke this device" : "Revoke").onClick(async () => {
|
||||
const selfRevoked = await this.plugin.getSyncServiceOrThrow().revokeDevice(device.deviceId);
|
||||
new Notice(
|
||||
selfRevoked
|
||||
? "This device was revoked. Import a recovery bundle or run sync again to register a new device identity."
|
||||
: "Device revoked."
|
||||
);
|
||||
await this.renderDeviceList(containerEl);
|
||||
});
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
containerEl.createEl("p", {
|
||||
text: error instanceof Error ? `Failed to load devices: ${error.message}` : "Failed to load devices."
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
47
apps/obsidian-plugin/src/settings/settings.ts
Normal file
47
apps/obsidian-plugin/src/settings/settings.ts
Normal file
@@ -0,0 +1,47 @@
|
||||
import type { SyncFileKind } from "@obsidian-sync/sync-protocol";
|
||||
|
||||
export interface SyncedFileState {
|
||||
kind: SyncFileKind;
|
||||
revisionId: string;
|
||||
contentHash: string;
|
||||
lastSyncedContent?: string;
|
||||
updatedAt: string;
|
||||
pendingConflict?: boolean;
|
||||
pendingConflictHash?: string;
|
||||
}
|
||||
|
||||
export interface SyncState {
|
||||
serverRevision: number;
|
||||
files: Record<string, SyncedFileState>;
|
||||
}
|
||||
|
||||
export interface ObsidianSyncSettings {
|
||||
serverUrl: string;
|
||||
vaultId: string;
|
||||
deviceId: string;
|
||||
deviceName: string;
|
||||
authToken: string;
|
||||
keyId: string;
|
||||
exportedVaultKey: string;
|
||||
syncIntervalMinutes: number;
|
||||
pullBatchSize: number;
|
||||
pushBatchSize: number;
|
||||
syncState: SyncState;
|
||||
}
|
||||
|
||||
export const DEFAULT_SETTINGS: ObsidianSyncSettings = {
|
||||
serverUrl: "http://localhost:8787",
|
||||
vaultId: "primary-vault",
|
||||
deviceId: "",
|
||||
deviceName: "",
|
||||
authToken: "",
|
||||
keyId: "",
|
||||
exportedVaultKey: "",
|
||||
syncIntervalMinutes: 5,
|
||||
pullBatchSize: 50,
|
||||
pushBatchSize: 50,
|
||||
syncState: {
|
||||
serverRevision: 0,
|
||||
files: {}
|
||||
}
|
||||
};
|
||||
91
apps/obsidian-plugin/src/sync/SyncRunLogger.ts
Normal file
91
apps/obsidian-plugin/src/sync/SyncRunLogger.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
import type { ClientLogEntry, ClientLogUploadRequest } from "@obsidian-sync/sync-protocol";
|
||||
|
||||
type LogLevel = ClientLogEntry["level"];
|
||||
|
||||
function normalizeError(error: unknown): Record<string, unknown> {
|
||||
if (error instanceof Error) {
|
||||
return {
|
||||
name: error.name,
|
||||
message: error.message,
|
||||
stack: error.stack
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
value: String(error)
|
||||
};
|
||||
}
|
||||
|
||||
export class SyncRunLogger {
|
||||
private readonly runId = globalThis.crypto.randomUUID();
|
||||
|
||||
private readonly entries: ClientLogEntry[] = [];
|
||||
|
||||
private deviceId: string;
|
||||
|
||||
constructor(deviceId: string) {
|
||||
this.deviceId = deviceId || "unregistered-device";
|
||||
}
|
||||
|
||||
setDeviceId(deviceId: string): void {
|
||||
if (deviceId) {
|
||||
this.deviceId = deviceId;
|
||||
}
|
||||
}
|
||||
|
||||
getRunId(): string {
|
||||
return this.runId;
|
||||
}
|
||||
|
||||
hasEntries(): boolean {
|
||||
return this.entries.length > 0;
|
||||
}
|
||||
|
||||
toUploadRequest(vaultId: string): ClientLogUploadRequest {
|
||||
return {
|
||||
vaultId,
|
||||
deviceId: this.deviceId,
|
||||
runId: this.runId,
|
||||
entries: [...this.entries]
|
||||
};
|
||||
}
|
||||
|
||||
debug(message: string, context?: Record<string, unknown>): void {
|
||||
this.log("debug", message, context);
|
||||
}
|
||||
|
||||
info(message: string, context?: Record<string, unknown>): void {
|
||||
this.log("info", message, context);
|
||||
}
|
||||
|
||||
warn(message: string, context?: Record<string, unknown>): void {
|
||||
this.log("warn", message, context);
|
||||
}
|
||||
|
||||
error(message: string, error?: unknown, context?: Record<string, unknown>): void {
|
||||
this.log("error", message, {
|
||||
...context,
|
||||
...(error === undefined ? {} : { error: normalizeError(error) })
|
||||
});
|
||||
}
|
||||
|
||||
private log(level: LogLevel, message: string, context?: Record<string, unknown>): void {
|
||||
const entry: ClientLogEntry = {
|
||||
level,
|
||||
message,
|
||||
timestamp: new Date().toISOString(),
|
||||
deviceId: this.deviceId,
|
||||
context: {
|
||||
runId: this.runId,
|
||||
...(context ?? {})
|
||||
}
|
||||
};
|
||||
|
||||
this.entries.push(entry);
|
||||
|
||||
const consoleMethod =
|
||||
level === "debug" ? console.debug : level === "info" ? console.info : level === "warn" ? console.warn : console.error;
|
||||
|
||||
consoleMethod(`[Obsidian Sync][${this.runId}] ${message}`, entry.context ?? {});
|
||||
}
|
||||
}
|
||||
984
apps/obsidian-plugin/src/sync/SyncService.ts
Normal file
984
apps/obsidian-plugin/src/sync/SyncService.ts
Normal file
@@ -0,0 +1,984 @@
|
||||
import { App, Notice, TFile, normalizePath } from "obsidian";
|
||||
|
||||
import {
|
||||
computeBinaryHash,
|
||||
computeTextHash,
|
||||
createConflictCopyPath,
|
||||
decryptBytes,
|
||||
decryptText,
|
||||
encryptBytes,
|
||||
encryptText,
|
||||
generateVaultKey,
|
||||
importVaultKey,
|
||||
mergeTextRevisions,
|
||||
type VaultKeyHandle
|
||||
} from "@obsidian-sync/sync-engine";
|
||||
import {
|
||||
ClientLogUploadResponseSchema,
|
||||
ListDevicesResponseSchema,
|
||||
RecoveryBundleSchema,
|
||||
RegisterDeviceResponseSchema,
|
||||
RevokeDeviceResponseSchema,
|
||||
RotateVaultKeyResponseSchema,
|
||||
SyncPullResponseSchema,
|
||||
SyncPushResponseSchema,
|
||||
type ListDevicesResponse,
|
||||
type RecoveryBundle,
|
||||
type SyncConflict,
|
||||
type SyncFileKind,
|
||||
type SyncFileRecord,
|
||||
type SyncPullResponse,
|
||||
type Tombstone
|
||||
} from "@obsidian-sync/sync-protocol";
|
||||
|
||||
import type { ObsidianSyncSettings, SyncedFileState } from "../settings/settings";
|
||||
import { SyncRunLogger } from "./SyncRunLogger";
|
||||
|
||||
type SettingsReader = () => ObsidianSyncSettings;
|
||||
type SettingsWriter = (settings: ObsidianSyncSettings) => Promise<void>;
|
||||
type SyncMode = "manual" | "scheduled";
|
||||
|
||||
const DEFAULT_BATCH_SIZE = 50;
|
||||
const IGNORED_PATH_PREFIXES = [".obsidian/"];
|
||||
const TEXT_FILE_EXTENSIONS = new Set(["md", "txt", "markdown", "canvas", "json", "yaml", "yml", "csv"]);
|
||||
|
||||
interface CollectLocalChangeOptions {
|
||||
forceAllFiles?: boolean;
|
||||
}
|
||||
|
||||
interface LocalChangeSet {
|
||||
files: SyncFileRecord[];
|
||||
tombstones: Tombstone[];
|
||||
statesByPath: Record<string, SyncedFileState>;
|
||||
}
|
||||
|
||||
interface LocalFileSnapshot {
|
||||
kind: SyncFileKind;
|
||||
contentHash: string;
|
||||
updatedAt: string;
|
||||
sizeBytes: number;
|
||||
textContent?: string;
|
||||
binaryContent?: ArrayBuffer;
|
||||
}
|
||||
|
||||
interface PullSummary {
|
||||
pulledChanges: number;
|
||||
finalServerRevision: number;
|
||||
}
|
||||
|
||||
interface PushSummary {
|
||||
acceptedServerRevision: number;
|
||||
acceptedFilePaths: string[];
|
||||
acceptedTombstones: string[];
|
||||
conflicts: SyncConflict[];
|
||||
}
|
||||
|
||||
export interface RotateVaultKeyResult {
|
||||
recoveryBundle: string;
|
||||
activeKeyId: string;
|
||||
rotatedAt: string;
|
||||
uploadedFiles: number;
|
||||
}
|
||||
|
||||
export class SyncService {
|
||||
constructor(
|
||||
private readonly app: App,
|
||||
private readonly readSettings: SettingsReader,
|
||||
private readonly writeSettings: SettingsWriter
|
||||
) {}
|
||||
|
||||
async runManualSync(): Promise<void> {
|
||||
await this.runSync("manual");
|
||||
}
|
||||
|
||||
async runScheduledSync(): Promise<void> {
|
||||
await this.runSync("scheduled");
|
||||
}
|
||||
|
||||
async exportRecoveryBundle(): Promise<string> {
|
||||
await this.ensureVaultKey();
|
||||
const settings = this.readSettings();
|
||||
return this.serializeRecoveryBundle(settings);
|
||||
}
|
||||
|
||||
async importRecoveryBundle(serializedBundle: string): Promise<void> {
|
||||
const parsed = RecoveryBundleSchema.parse(JSON.parse(serializedBundle) as RecoveryBundle);
|
||||
const nextSettings = this.cloneSettings();
|
||||
|
||||
nextSettings.serverUrl = parsed.serverUrl;
|
||||
nextSettings.vaultId = parsed.vaultId;
|
||||
nextSettings.keyId = parsed.keyId;
|
||||
nextSettings.exportedVaultKey = parsed.exportedVaultKey;
|
||||
nextSettings.deviceId = "";
|
||||
nextSettings.authToken = "";
|
||||
nextSettings.syncState = {
|
||||
serverRevision: 0,
|
||||
files: {}
|
||||
};
|
||||
|
||||
await this.writeSettings(nextSettings);
|
||||
}
|
||||
|
||||
async listDevices(): Promise<ListDevicesResponse> {
|
||||
await this.ensureRegisteredDevice();
|
||||
const settings = this.readSettings();
|
||||
return this.postJson(
|
||||
"/api/devices/list",
|
||||
{
|
||||
vaultId: settings.vaultId,
|
||||
deviceId: settings.deviceId
|
||||
},
|
||||
ListDevicesResponseSchema,
|
||||
settings.authToken
|
||||
);
|
||||
}
|
||||
|
||||
async revokeDevice(targetDeviceId: string): Promise<boolean> {
|
||||
await this.ensureRegisteredDevice();
|
||||
const settings = this.readSettings();
|
||||
|
||||
await this.postJson(
|
||||
"/api/devices/revoke",
|
||||
{
|
||||
vaultId: settings.vaultId,
|
||||
deviceId: settings.deviceId,
|
||||
targetDeviceId
|
||||
},
|
||||
RevokeDeviceResponseSchema,
|
||||
settings.authToken
|
||||
);
|
||||
|
||||
if (targetDeviceId === settings.deviceId) {
|
||||
const nextSettings = this.cloneSettings();
|
||||
nextSettings.deviceId = "";
|
||||
nextSettings.authToken = "";
|
||||
await this.writeSettings(nextSettings);
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
async rotateVaultKey(): Promise<RotateVaultKeyResult> {
|
||||
const diagnostics = new SyncRunLogger(this.readSettings().deviceId);
|
||||
|
||||
try {
|
||||
const currentVaultKey = await this.ensureVaultKey();
|
||||
diagnostics.info("vault-key-ready", {
|
||||
keyId: currentVaultKey.keyId
|
||||
});
|
||||
|
||||
await this.ensureDeviceRegistration(diagnostics);
|
||||
diagnostics.setDeviceId(this.readSettings().deviceId);
|
||||
|
||||
const generatedKey = await generateVaultKey();
|
||||
const localChanges = await this.collectLocalChanges(
|
||||
{
|
||||
keyId: generatedKey.keyId,
|
||||
key: generatedKey.key
|
||||
},
|
||||
diagnostics,
|
||||
{
|
||||
forceAllFiles: true
|
||||
}
|
||||
);
|
||||
|
||||
const pushSummary = await this.pushLocalChanges(localChanges, diagnostics);
|
||||
const settings = this.readSettings();
|
||||
const rotated = await this.postJson(
|
||||
"/api/keys/rotate",
|
||||
{
|
||||
vaultId: settings.vaultId,
|
||||
deviceId: settings.deviceId,
|
||||
nextKeyId: generatedKey.keyId,
|
||||
previousKeyId: settings.keyId || undefined
|
||||
},
|
||||
RotateVaultKeyResponseSchema,
|
||||
settings.authToken
|
||||
);
|
||||
|
||||
const nextSettings = this.cloneSettings();
|
||||
nextSettings.keyId = generatedKey.keyId;
|
||||
nextSettings.exportedVaultKey = generatedKey.exportedKey;
|
||||
nextSettings.syncState.serverRevision = Math.max(nextSettings.syncState.serverRevision, pushSummary.acceptedServerRevision);
|
||||
await this.writeSettings(nextSettings);
|
||||
|
||||
diagnostics.info("vault-key-rotated", {
|
||||
activeKeyId: rotated.activeKeyId,
|
||||
rotatedAt: rotated.rotatedAt,
|
||||
uploadedFiles: pushSummary.acceptedFilePaths.length
|
||||
});
|
||||
|
||||
return {
|
||||
recoveryBundle: this.serializeRecoveryBundle(nextSettings),
|
||||
activeKeyId: rotated.activeKeyId,
|
||||
rotatedAt: rotated.rotatedAt,
|
||||
uploadedFiles: pushSummary.acceptedFilePaths.length
|
||||
};
|
||||
} catch (error) {
|
||||
diagnostics.error("vault-key-rotation-failed", error);
|
||||
throw error;
|
||||
} finally {
|
||||
await this.uploadDiagnostics(diagnostics);
|
||||
}
|
||||
}
|
||||
|
||||
private async runSync(mode: SyncMode): Promise<void> {
|
||||
const settings = this.readSettings();
|
||||
const diagnostics = new SyncRunLogger(settings.deviceId);
|
||||
if (!settings.serverUrl || !settings.vaultId) {
|
||||
if (mode === "manual") {
|
||||
new Notice("Set a server URL and vault ID before syncing.");
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
diagnostics.info("sync-started", {
|
||||
mode,
|
||||
serverRevision: settings.syncState.serverRevision
|
||||
});
|
||||
|
||||
try {
|
||||
const vaultKey = await this.ensureVaultKey();
|
||||
diagnostics.info("vault-key-ready", {
|
||||
keyId: vaultKey.keyId
|
||||
});
|
||||
|
||||
await this.ensureDeviceRegistration(diagnostics);
|
||||
diagnostics.setDeviceId(this.readSettings().deviceId);
|
||||
|
||||
const pullSummary = await this.pullAndApplyRemoteChanges(vaultKey, diagnostics);
|
||||
const localChanges = await this.collectLocalChanges(vaultKey, diagnostics);
|
||||
const pushSummary = await this.pushLocalChanges(localChanges, diagnostics);
|
||||
|
||||
const nextSettings = this.cloneSettings();
|
||||
nextSettings.syncState.serverRevision = Math.max(pullSummary.finalServerRevision, pushSummary.acceptedServerRevision);
|
||||
await this.writeSettings(nextSettings);
|
||||
|
||||
if (mode === "manual") {
|
||||
new Notice(
|
||||
`Sync complete. Pulled ${pullSummary.pulledChanges} changes, uploaded ${pushSummary.acceptedFilePaths.length} files.`
|
||||
);
|
||||
}
|
||||
|
||||
if (pushSummary.conflicts.length > 0) {
|
||||
diagnostics.warn("server-conflicts-reported", {
|
||||
count: pushSummary.conflicts.length,
|
||||
paths: pushSummary.conflicts.map((conflict) => conflict.path)
|
||||
});
|
||||
new Notice(`${pushSummary.conflicts.length} server conflicts need review.`);
|
||||
}
|
||||
} catch (error) {
|
||||
diagnostics.error("sync-failed", error, {
|
||||
mode
|
||||
});
|
||||
console.error("Obsidian Sync failed", error);
|
||||
new Notice(
|
||||
error instanceof Error
|
||||
? `Sync failed: ${error.message}. Run ${diagnostics.getRunId()}.`
|
||||
: `Sync failed. Run ${diagnostics.getRunId()}.`
|
||||
);
|
||||
} finally {
|
||||
await this.uploadDiagnostics(diagnostics);
|
||||
}
|
||||
}
|
||||
|
||||
private cloneSettings(): ObsidianSyncSettings {
|
||||
const current = this.readSettings();
|
||||
return {
|
||||
...current,
|
||||
syncState: {
|
||||
...current.syncState,
|
||||
files: { ...current.syncState.files }
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
private getBaseUrl(): string {
|
||||
return this.readSettings().serverUrl.replace(/\/$/, "");
|
||||
}
|
||||
|
||||
private getPullBatchSize(): number {
|
||||
return Math.max(1, this.readSettings().pullBatchSize || DEFAULT_BATCH_SIZE);
|
||||
}
|
||||
|
||||
private getPushBatchSize(): number {
|
||||
return Math.max(1, this.readSettings().pushBatchSize || DEFAULT_BATCH_SIZE);
|
||||
}
|
||||
|
||||
private getSyncableFiles(): TFile[] {
|
||||
return this.app.vault.getFiles().filter((file) => !IGNORED_PATH_PREFIXES.some((prefix) => file.path.startsWith(prefix)));
|
||||
}
|
||||
|
||||
private resolveFileKind(file: TFile, previous?: SyncedFileState): SyncFileKind {
|
||||
if (previous?.kind) {
|
||||
return previous.kind;
|
||||
}
|
||||
|
||||
return TEXT_FILE_EXTENSIONS.has(file.extension.toLowerCase()) ? "text" : "binary";
|
||||
}
|
||||
|
||||
private createSyncedState(
|
||||
kind: SyncFileKind,
|
||||
revisionId: string,
|
||||
contentHash: string,
|
||||
updatedAt: string,
|
||||
lastSyncedContent?: string
|
||||
): SyncedFileState {
|
||||
return {
|
||||
kind,
|
||||
revisionId,
|
||||
contentHash,
|
||||
updatedAt,
|
||||
...(kind === "text" && lastSyncedContent !== undefined ? { lastSyncedContent } : {})
|
||||
};
|
||||
}
|
||||
|
||||
private buildRecoveryBundle(settings: ObsidianSyncSettings): RecoveryBundle {
|
||||
return RecoveryBundleSchema.parse({
|
||||
version: 1,
|
||||
serverUrl: settings.serverUrl,
|
||||
vaultId: settings.vaultId,
|
||||
keyId: settings.keyId,
|
||||
exportedVaultKey: settings.exportedVaultKey,
|
||||
generatedAt: new Date().toISOString()
|
||||
});
|
||||
}
|
||||
|
||||
private serializeRecoveryBundle(settings: ObsidianSyncSettings): string {
|
||||
return JSON.stringify(this.buildRecoveryBundle(settings), null, 2);
|
||||
}
|
||||
|
||||
private async readLocalFileSnapshot(file: TFile, previous?: SyncedFileState): Promise<LocalFileSnapshot> {
|
||||
const kind = this.resolveFileKind(file, previous);
|
||||
|
||||
if (kind === "text") {
|
||||
const textContent = await this.app.vault.cachedRead(file);
|
||||
return {
|
||||
kind,
|
||||
textContent,
|
||||
contentHash: await computeTextHash(textContent),
|
||||
updatedAt: new Date(file.stat.mtime).toISOString(),
|
||||
sizeBytes: new TextEncoder().encode(textContent).byteLength
|
||||
};
|
||||
}
|
||||
|
||||
const binaryContent = await this.app.vault.readBinary(file);
|
||||
return {
|
||||
kind,
|
||||
binaryContent,
|
||||
contentHash: await computeBinaryHash(binaryContent),
|
||||
updatedAt: new Date(file.stat.mtime).toISOString(),
|
||||
sizeBytes: file.stat.size
|
||||
};
|
||||
}
|
||||
|
||||
private async ensureVaultKey(): Promise<VaultKeyHandle> {
|
||||
const settings = this.readSettings();
|
||||
if (settings.exportedVaultKey && settings.keyId) {
|
||||
return importVaultKey(settings.exportedVaultKey, settings.keyId);
|
||||
}
|
||||
|
||||
const nextSettings = this.cloneSettings();
|
||||
const generated = await generateVaultKey();
|
||||
nextSettings.keyId = generated.keyId;
|
||||
nextSettings.exportedVaultKey = generated.exportedKey;
|
||||
await this.writeSettings(nextSettings);
|
||||
|
||||
return {
|
||||
keyId: generated.keyId,
|
||||
key: generated.key
|
||||
};
|
||||
}
|
||||
|
||||
private async ensureRegisteredDevice(): Promise<void> {
|
||||
await this.ensureDeviceRegistration(new SyncRunLogger(this.readSettings().deviceId));
|
||||
}
|
||||
|
||||
private async ensureDeviceRegistration(diagnostics: SyncRunLogger): Promise<void> {
|
||||
const settings = this.readSettings();
|
||||
if (settings.deviceId && settings.authToken) {
|
||||
return;
|
||||
}
|
||||
|
||||
const deviceName = settings.deviceName || this.app.vault.getName();
|
||||
const response = await this.postJson(
|
||||
"/api/devices/register",
|
||||
{
|
||||
vaultId: settings.vaultId,
|
||||
deviceName
|
||||
},
|
||||
RegisterDeviceResponseSchema
|
||||
);
|
||||
|
||||
diagnostics.info("device-registered", {
|
||||
deviceId: response.deviceId,
|
||||
deviceName
|
||||
});
|
||||
|
||||
const nextSettings = this.cloneSettings();
|
||||
nextSettings.deviceId = response.deviceId;
|
||||
nextSettings.authToken = response.token;
|
||||
nextSettings.deviceName = deviceName;
|
||||
await this.writeSettings(nextSettings);
|
||||
}
|
||||
|
||||
private ensureCompatibleVaultKey(pullResponse: SyncPullResponse): void {
|
||||
const settings = this.readSettings();
|
||||
if (pullResponse.activeKeyId && settings.keyId && pullResponse.activeKeyId !== settings.keyId) {
|
||||
throw new Error("The server is using a different vault key. Import the latest recovery bundle before syncing.");
|
||||
}
|
||||
}
|
||||
|
||||
private async pullRemoteChangesPage(sinceServerRevision: number): Promise<SyncPullResponse> {
|
||||
const settings = this.readSettings();
|
||||
return this.postJson(
|
||||
"/api/sync/pull",
|
||||
{
|
||||
vaultId: settings.vaultId,
|
||||
deviceId: settings.deviceId,
|
||||
sinceServerRevision,
|
||||
limit: this.getPullBatchSize()
|
||||
},
|
||||
SyncPullResponseSchema,
|
||||
settings.authToken
|
||||
);
|
||||
}
|
||||
|
||||
private async pullAndApplyRemoteChanges(vaultKey: VaultKeyHandle, diagnostics: SyncRunLogger): Promise<PullSummary> {
|
||||
let pulledChanges = 0;
|
||||
let nextSinceServerRevision = this.readSettings().syncState.serverRevision;
|
||||
|
||||
while (true) {
|
||||
const pullResponse = await this.pullRemoteChangesPage(nextSinceServerRevision);
|
||||
this.ensureCompatibleVaultKey(pullResponse);
|
||||
|
||||
if (pullResponse.changes.length === 0) {
|
||||
diagnostics.info("remote-pull-finished", {
|
||||
changes: pulledChanges,
|
||||
serverRevision: pullResponse.serverRevision,
|
||||
hasMore: pullResponse.hasMore
|
||||
});
|
||||
break;
|
||||
}
|
||||
|
||||
await this.applyRemoteChanges(pullResponse, vaultKey, diagnostics);
|
||||
pulledChanges += pullResponse.changes.length;
|
||||
nextSinceServerRevision = pullResponse.nextSinceServerRevision;
|
||||
|
||||
diagnostics.info("remote-pull-page-applied", {
|
||||
pageChanges: pullResponse.changes.length,
|
||||
nextSinceServerRevision,
|
||||
hasMore: pullResponse.hasMore
|
||||
});
|
||||
|
||||
if (!pullResponse.hasMore) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
pulledChanges,
|
||||
finalServerRevision: this.readSettings().syncState.serverRevision
|
||||
};
|
||||
}
|
||||
|
||||
private async pushLocalChanges(localChanges: LocalChangeSet, diagnostics: SyncRunLogger): Promise<PushSummary> {
|
||||
const settings = this.readSettings();
|
||||
if (localChanges.files.length === 0 && localChanges.tombstones.length === 0) {
|
||||
return {
|
||||
acceptedServerRevision: settings.syncState.serverRevision,
|
||||
acceptedFilePaths: [],
|
||||
acceptedTombstones: [],
|
||||
conflicts: []
|
||||
};
|
||||
}
|
||||
|
||||
const acceptedFilePaths: string[] = [];
|
||||
const acceptedTombstones: string[] = [];
|
||||
const conflicts: SyncConflict[] = [];
|
||||
let acceptedServerRevision = settings.syncState.serverRevision;
|
||||
let batchNumber = 0;
|
||||
const fileBatches = this.chunkEntries(localChanges.files, this.getPushBatchSize());
|
||||
const tombstoneBatches = this.chunkEntries(localChanges.tombstones, this.getPushBatchSize());
|
||||
|
||||
for (const fileBatch of fileBatches) {
|
||||
batchNumber += 1;
|
||||
const response = await this.pushBatch(fileBatch, [], acceptedServerRevision);
|
||||
await this.reconcileAcceptedLocalChanges(localChanges.statesByPath, response.acceptedFilePaths, response.acceptedTombstones);
|
||||
acceptedServerRevision = Math.max(acceptedServerRevision, response.acceptedServerRevision);
|
||||
acceptedFilePaths.push(...response.acceptedFilePaths);
|
||||
acceptedTombstones.push(...response.acceptedTombstones);
|
||||
conflicts.push(...response.conflicts);
|
||||
await this.persistAcceptedServerRevision(acceptedServerRevision);
|
||||
|
||||
diagnostics.info("push-batch-finished", {
|
||||
batchNumber,
|
||||
batchKind: "files",
|
||||
attempted: fileBatch.length,
|
||||
accepted: response.acceptedFilePaths.length,
|
||||
conflicts: response.conflicts.length
|
||||
});
|
||||
}
|
||||
|
||||
for (const tombstoneBatch of tombstoneBatches) {
|
||||
batchNumber += 1;
|
||||
const response = await this.pushBatch([], tombstoneBatch, acceptedServerRevision);
|
||||
await this.reconcileAcceptedLocalChanges(localChanges.statesByPath, response.acceptedFilePaths, response.acceptedTombstones);
|
||||
acceptedServerRevision = Math.max(acceptedServerRevision, response.acceptedServerRevision);
|
||||
acceptedFilePaths.push(...response.acceptedFilePaths);
|
||||
acceptedTombstones.push(...response.acceptedTombstones);
|
||||
conflicts.push(...response.conflicts);
|
||||
await this.persistAcceptedServerRevision(acceptedServerRevision);
|
||||
|
||||
diagnostics.info("push-batch-finished", {
|
||||
batchNumber,
|
||||
batchKind: "tombstones",
|
||||
attempted: tombstoneBatch.length,
|
||||
accepted: response.acceptedTombstones.length,
|
||||
conflicts: response.conflicts.length
|
||||
});
|
||||
}
|
||||
|
||||
diagnostics.info("remote-push-finished", {
|
||||
acceptedFiles: acceptedFilePaths.length,
|
||||
acceptedDeletes: acceptedTombstones.length,
|
||||
conflicts: conflicts.length
|
||||
});
|
||||
|
||||
return {
|
||||
acceptedServerRevision,
|
||||
acceptedFilePaths,
|
||||
acceptedTombstones,
|
||||
conflicts
|
||||
};
|
||||
}
|
||||
|
||||
private async pushBatch(
|
||||
files: SyncFileRecord[],
|
||||
tombstones: Tombstone[],
|
||||
knownServerRevision: number
|
||||
): Promise<ReturnType<typeof SyncPushResponseSchema.parse>> {
|
||||
const settings = this.readSettings();
|
||||
return this.postJson(
|
||||
"/api/sync/push",
|
||||
{
|
||||
vaultId: settings.vaultId,
|
||||
deviceId: settings.deviceId,
|
||||
knownServerRevision,
|
||||
files,
|
||||
tombstones
|
||||
},
|
||||
SyncPushResponseSchema,
|
||||
settings.authToken
|
||||
);
|
||||
}
|
||||
|
||||
private chunkEntries<TValue>(entries: TValue[], batchSize: number): TValue[][] {
|
||||
if (entries.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const batches: TValue[][] = [];
|
||||
for (let index = 0; index < entries.length; index += batchSize) {
|
||||
batches.push(entries.slice(index, index + batchSize));
|
||||
}
|
||||
|
||||
return batches;
|
||||
}
|
||||
|
||||
private async collectLocalChanges(
|
||||
vaultKey: VaultKeyHandle,
|
||||
diagnostics: SyncRunLogger,
|
||||
options: CollectLocalChangeOptions = {}
|
||||
): Promise<LocalChangeSet> {
|
||||
const settings = this.readSettings();
|
||||
const files: SyncFileRecord[] = [];
|
||||
const tombstones: Tombstone[] = [];
|
||||
const statesByPath: Record<string, SyncedFileState> = {};
|
||||
const seenPaths = new Set<string>();
|
||||
|
||||
for (const file of this.getSyncableFiles()) {
|
||||
const previous = settings.syncState.files[file.path];
|
||||
const snapshot = await this.readLocalFileSnapshot(file, previous);
|
||||
seenPaths.add(file.path);
|
||||
|
||||
if (previous?.pendingConflict && previous.pendingConflictHash === snapshot.contentHash) {
|
||||
diagnostics.debug("skipped-pending-conflict-file", {
|
||||
path: file.path,
|
||||
kind: snapshot.kind
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!options.forceAllFiles && previous && previous.contentHash === snapshot.contentHash) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const envelope =
|
||||
snapshot.kind === "text"
|
||||
? await encryptText(snapshot.textContent ?? "", vaultKey)
|
||||
: await encryptBytes(snapshot.binaryContent ?? new ArrayBuffer(0), vaultKey);
|
||||
const revisionId = globalThis.crypto.randomUUID();
|
||||
|
||||
files.push({
|
||||
manifest: {
|
||||
path: file.path,
|
||||
kind: snapshot.kind,
|
||||
contentHash: snapshot.contentHash,
|
||||
revisionId,
|
||||
baseRevisionId: previous?.revisionId,
|
||||
updatedAt: snapshot.updatedAt,
|
||||
sizeBytes: snapshot.sizeBytes,
|
||||
deviceId: settings.deviceId
|
||||
},
|
||||
envelope
|
||||
});
|
||||
|
||||
statesByPath[file.path] = {
|
||||
kind: snapshot.kind,
|
||||
revisionId,
|
||||
contentHash: snapshot.contentHash,
|
||||
updatedAt: snapshot.updatedAt,
|
||||
...(snapshot.kind === "text" ? { lastSyncedContent: snapshot.textContent } : {})
|
||||
};
|
||||
}
|
||||
|
||||
for (const [path, state] of Object.entries(settings.syncState.files)) {
|
||||
if (seenPaths.has(path)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
tombstones.push({
|
||||
path,
|
||||
revisionId: globalThis.crypto.randomUUID(),
|
||||
baseRevisionId: state.revisionId,
|
||||
deletedAt: new Date().toISOString(),
|
||||
deviceId: settings.deviceId
|
||||
});
|
||||
}
|
||||
|
||||
diagnostics.info("local-scan-finished", {
|
||||
stagedFiles: files.length,
|
||||
stagedDeletes: tombstones.length,
|
||||
forceAllFiles: Boolean(options.forceAllFiles)
|
||||
});
|
||||
|
||||
return {
|
||||
files,
|
||||
tombstones,
|
||||
statesByPath
|
||||
};
|
||||
}
|
||||
|
||||
private async persistAcceptedServerRevision(serverRevision: number): Promise<void> {
|
||||
const nextSettings = this.cloneSettings();
|
||||
nextSettings.syncState.serverRevision = Math.max(nextSettings.syncState.serverRevision, serverRevision);
|
||||
await this.writeSettings(nextSettings);
|
||||
}
|
||||
|
||||
private async reconcileAcceptedLocalChanges(
|
||||
statesByPath: Record<string, SyncedFileState>,
|
||||
acceptedFilePaths: string[],
|
||||
acceptedTombstones: string[]
|
||||
): Promise<void> {
|
||||
if (acceptedFilePaths.length === 0 && acceptedTombstones.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const nextSettings = this.cloneSettings();
|
||||
|
||||
for (const path of acceptedFilePaths) {
|
||||
const state = statesByPath[path];
|
||||
if (state) {
|
||||
nextSettings.syncState.files[path] = state;
|
||||
}
|
||||
}
|
||||
|
||||
for (const path of acceptedTombstones) {
|
||||
delete nextSettings.syncState.files[path];
|
||||
}
|
||||
|
||||
await this.writeSettings(nextSettings);
|
||||
}
|
||||
|
||||
private async applyRemoteChanges(
|
||||
pullResponse: SyncPullResponse,
|
||||
vaultKey: VaultKeyHandle,
|
||||
diagnostics: SyncRunLogger
|
||||
): Promise<void> {
|
||||
if (pullResponse.changes.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const nextSettings = this.cloneSettings();
|
||||
|
||||
for (const change of pullResponse.changes) {
|
||||
if (change.file) {
|
||||
const remotePath = normalizePath(change.file.manifest.path);
|
||||
const localAbstract = this.app.vault.getAbstractFileByPath(remotePath);
|
||||
const previous = nextSettings.syncState.files[remotePath];
|
||||
|
||||
if (change.file.manifest.kind === "text") {
|
||||
const remoteContent = await decryptText(change.file.envelope, vaultKey);
|
||||
|
||||
if (!(localAbstract instanceof TFile)) {
|
||||
await this.ensureParentFolders(remotePath);
|
||||
await this.app.vault.create(remotePath, remoteContent);
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"text",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt,
|
||||
remoteContent
|
||||
);
|
||||
diagnostics.info("remote-text-created", {
|
||||
path: remotePath
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const localContent = await this.app.vault.cachedRead(localAbstract);
|
||||
const localHash = await computeTextHash(localContent);
|
||||
|
||||
if (localHash === change.file.manifest.contentHash) {
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"text",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt,
|
||||
remoteContent
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!previous || localHash === previous.contentHash || localContent === previous.lastSyncedContent) {
|
||||
await this.app.vault.modify(localAbstract, remoteContent);
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"text",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt,
|
||||
remoteContent
|
||||
);
|
||||
diagnostics.info("remote-text-applied", {
|
||||
path: remotePath
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const merged = mergeTextRevisions({
|
||||
base: previous.lastSyncedContent,
|
||||
local: localContent,
|
||||
remote: remoteContent
|
||||
});
|
||||
|
||||
await this.app.vault.modify(localAbstract, merged.content);
|
||||
|
||||
if (merged.status === "conflict") {
|
||||
const conflictCopyPath = createConflictCopyPath(remotePath);
|
||||
await this.ensureParentFolders(conflictCopyPath);
|
||||
await this.writeTextFile(conflictCopyPath, remoteContent);
|
||||
|
||||
nextSettings.syncState.files[remotePath] = {
|
||||
...this.createSyncedState(
|
||||
"text",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt,
|
||||
remoteContent
|
||||
),
|
||||
pendingConflict: true,
|
||||
pendingConflictHash: await computeTextHash(merged.content)
|
||||
};
|
||||
|
||||
diagnostics.warn("remote-text-conflict", {
|
||||
path: remotePath,
|
||||
conflictCopyPath
|
||||
});
|
||||
new Notice(`Conflict markers added to ${remotePath}. Review the file before the next sync.`);
|
||||
continue;
|
||||
}
|
||||
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"text",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt,
|
||||
remoteContent
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const remoteBinary = await decryptBytes(change.file.envelope, vaultKey);
|
||||
|
||||
if (!(localAbstract instanceof TFile)) {
|
||||
await this.ensureParentFolders(remotePath);
|
||||
await this.app.vault.createBinary(remotePath, remoteBinary);
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"binary",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt
|
||||
);
|
||||
diagnostics.info("remote-binary-created", {
|
||||
path: remotePath
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const localBinary = await this.app.vault.readBinary(localAbstract);
|
||||
const localHash = await computeBinaryHash(localBinary);
|
||||
|
||||
if (localHash === change.file.manifest.contentHash) {
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"binary",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!previous || localHash === previous.contentHash) {
|
||||
await this.app.vault.modifyBinary(localAbstract, remoteBinary);
|
||||
nextSettings.syncState.files[remotePath] = this.createSyncedState(
|
||||
"binary",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt
|
||||
);
|
||||
diagnostics.info("remote-binary-applied", {
|
||||
path: remotePath
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const conflictCopyPath = createConflictCopyPath(remotePath);
|
||||
await this.ensureParentFolders(conflictCopyPath);
|
||||
await this.writeBinaryFile(conflictCopyPath, remoteBinary);
|
||||
|
||||
nextSettings.syncState.files[remotePath] = {
|
||||
...this.createSyncedState(
|
||||
"binary",
|
||||
change.file.manifest.revisionId,
|
||||
change.file.manifest.contentHash,
|
||||
change.file.manifest.updatedAt
|
||||
),
|
||||
pendingConflict: true,
|
||||
pendingConflictHash: localHash
|
||||
};
|
||||
|
||||
diagnostics.warn("remote-binary-conflict", {
|
||||
path: remotePath,
|
||||
conflictCopyPath
|
||||
});
|
||||
new Notice(`Attachment conflict detected for ${remotePath}. Review the remote conflict copy before the next sync.`);
|
||||
}
|
||||
|
||||
if (change.tombstone) {
|
||||
const localAbstract = this.app.vault.getAbstractFileByPath(change.tombstone.path);
|
||||
const previous = nextSettings.syncState.files[change.tombstone.path];
|
||||
|
||||
if (!(localAbstract instanceof TFile)) {
|
||||
delete nextSettings.syncState.files[change.tombstone.path];
|
||||
continue;
|
||||
}
|
||||
|
||||
const snapshot = await this.readLocalFileSnapshot(localAbstract, previous);
|
||||
if (!previous || snapshot.contentHash === previous.contentHash) {
|
||||
await this.app.vault.delete(localAbstract, true);
|
||||
delete nextSettings.syncState.files[change.tombstone.path];
|
||||
diagnostics.info("remote-delete-applied", {
|
||||
path: change.tombstone.path
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
diagnostics.warn("remote-delete-skipped", {
|
||||
path: change.tombstone.path
|
||||
});
|
||||
new Notice(`Skipped remote delete for ${change.tombstone.path} because local changes still exist.`);
|
||||
}
|
||||
}
|
||||
|
||||
nextSettings.syncState.serverRevision = pullResponse.nextSinceServerRevision;
|
||||
await this.writeSettings(nextSettings);
|
||||
}
|
||||
|
||||
private async ensureParentFolders(path: string): Promise<void> {
|
||||
const segments = normalizePath(path).split("/");
|
||||
segments.pop();
|
||||
|
||||
let currentPath = "";
|
||||
for (const segment of segments) {
|
||||
currentPath = currentPath ? `${currentPath}/${segment}` : segment;
|
||||
if (!this.app.vault.getAbstractFileByPath(currentPath)) {
|
||||
await this.app.vault.createFolder(currentPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async writeTextFile(path: string, content: string): Promise<void> {
|
||||
const existing = this.app.vault.getAbstractFileByPath(path);
|
||||
if (existing instanceof TFile) {
|
||||
await this.app.vault.modify(existing, content);
|
||||
return;
|
||||
}
|
||||
|
||||
await this.app.vault.create(path, content);
|
||||
}
|
||||
|
||||
private async writeBinaryFile(path: string, content: ArrayBuffer): Promise<void> {
|
||||
const existing = this.app.vault.getAbstractFileByPath(path);
|
||||
if (existing instanceof TFile) {
|
||||
await this.app.vault.modifyBinary(existing, content);
|
||||
return;
|
||||
}
|
||||
|
||||
await this.app.vault.createBinary(path, content);
|
||||
}
|
||||
|
||||
private async uploadDiagnostics(diagnostics: SyncRunLogger): Promise<void> {
|
||||
const settings = this.readSettings();
|
||||
diagnostics.setDeviceId(settings.deviceId);
|
||||
|
||||
if (!diagnostics.hasEntries() || !settings.deviceId || !settings.authToken) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await this.postJson(
|
||||
"/api/logs",
|
||||
diagnostics.toUploadRequest(settings.vaultId),
|
||||
ClientLogUploadResponseSchema,
|
||||
settings.authToken
|
||||
);
|
||||
} catch (error) {
|
||||
console.warn("Failed to upload sync diagnostics", error);
|
||||
}
|
||||
}
|
||||
|
||||
private async postJson<TSchema>(
|
||||
path: string,
|
||||
body: unknown,
|
||||
schema: { parse: (value: unknown) => TSchema },
|
||||
authToken?: string
|
||||
): Promise<TSchema> {
|
||||
const response = await fetch(`${this.getBaseUrl()}${path}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
...(authToken ? { authorization: `Bearer ${authToken}` } : {})
|
||||
},
|
||||
body: JSON.stringify(body)
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const json = (await response.json()) as unknown;
|
||||
return schema.parse(json);
|
||||
}
|
||||
}
|
||||
21
apps/obsidian-plugin/tsconfig.json
Normal file
21
apps/obsidian-plugin/tsconfig.json
Normal file
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"composite": true,
|
||||
"declaration": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"module": "CommonJS"
|
||||
},
|
||||
"references": [
|
||||
{
|
||||
"path": "../../packages/sync-protocol"
|
||||
},
|
||||
{
|
||||
"path": "../../packages/sync-engine"
|
||||
}
|
||||
],
|
||||
"include": [
|
||||
"src/**/*.ts"
|
||||
]
|
||||
}
|
||||
23
apps/sync-server/package.json
Normal file
23
apps/sync-server/package.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"name": "@obsidian-sync/sync-server",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsc -b",
|
||||
"dev": "tsx watch src/index.ts",
|
||||
"start": "node dist/index.js",
|
||||
"typecheck": "tsc -b --pretty false"
|
||||
},
|
||||
"dependencies": {
|
||||
"@obsidian-sync/sync-protocol": "0.1.0",
|
||||
"cors": "^2.8.5",
|
||||
"express": "^4.21.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/cors": "^2.8.17",
|
||||
"@types/express": "^4.17.21",
|
||||
"tsx": "^4.19.3"
|
||||
}
|
||||
}
|
||||
38
apps/sync-server/src/index.ts
Normal file
38
apps/sync-server/src/index.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
import path from "node:path";
|
||||
|
||||
import cors from "cors";
|
||||
import express from "express";
|
||||
|
||||
import { createJsonFileLogger } from "./logging/jsonFileLogger";
|
||||
import { createRequestLoggingMiddleware } from "./middleware/requestLoggingMiddleware";
|
||||
import { createSyncRouter } from "./routes/createSyncRouter";
|
||||
import { createFileSyncStore } from "./store/fileSyncStore";
|
||||
|
||||
const port = Number(process.env.PORT ?? 8787);
|
||||
const dataDirectory = process.env.SYNC_DATA_DIR
|
||||
? path.resolve(process.env.SYNC_DATA_DIR)
|
||||
: path.resolve(process.cwd(), "data");
|
||||
const app = express();
|
||||
const logger = createJsonFileLogger(path.join(dataDirectory, "logs", "server.jsonl"));
|
||||
const store = createFileSyncStore(path.join(dataDirectory, "sync-state.json"));
|
||||
|
||||
app.use(cors());
|
||||
app.use(express.json({ limit: "10mb" }));
|
||||
app.use(createRequestLoggingMiddleware(logger));
|
||||
|
||||
app.get("/health", (_request, response) => {
|
||||
response.json({
|
||||
status: "ok",
|
||||
dataDirectory
|
||||
});
|
||||
});
|
||||
|
||||
app.use("/api", createSyncRouter(store, logger));
|
||||
|
||||
app.listen(port, () => {
|
||||
logger.info("server-started", {
|
||||
port,
|
||||
dataDirectory
|
||||
});
|
||||
console.log(`Obsidian Sync server listening on http://localhost:${port}`);
|
||||
});
|
||||
45
apps/sync-server/src/logging/jsonFileLogger.ts
Normal file
45
apps/sync-server/src/logging/jsonFileLogger.ts
Normal file
@@ -0,0 +1,45 @@
|
||||
import { appendFileSync, mkdirSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
export type LogLevel = "debug" | "info" | "warn" | "error";
|
||||
|
||||
export interface StructuredLogger {
|
||||
log(level: LogLevel, message: string, context?: Record<string, unknown>): void;
|
||||
debug(message: string, context?: Record<string, unknown>): void;
|
||||
info(message: string, context?: Record<string, unknown>): void;
|
||||
warn(message: string, context?: Record<string, unknown>): void;
|
||||
error(message: string, context?: Record<string, unknown>): void;
|
||||
}
|
||||
|
||||
export function createJsonFileLogger(logFilePath: string): StructuredLogger {
|
||||
mkdirSync(dirname(logFilePath), { recursive: true });
|
||||
|
||||
const write = (level: LogLevel, message: string, context?: Record<string, unknown>) => {
|
||||
const payload = {
|
||||
timestamp: new Date().toISOString(),
|
||||
level,
|
||||
message,
|
||||
context: context ?? {}
|
||||
};
|
||||
|
||||
appendFileSync(logFilePath, `${JSON.stringify(payload)}\n`, "utf8");
|
||||
};
|
||||
|
||||
return {
|
||||
log(level, message, context) {
|
||||
write(level, message, context);
|
||||
},
|
||||
debug(message, context) {
|
||||
write("debug", message, context);
|
||||
},
|
||||
info(message, context) {
|
||||
write("info", message, context);
|
||||
},
|
||||
warn(message, context) {
|
||||
write("warn", message, context);
|
||||
},
|
||||
error(message, context) {
|
||||
write("error", message, context);
|
||||
}
|
||||
};
|
||||
}
|
||||
26
apps/sync-server/src/middleware/requestLoggingMiddleware.ts
Normal file
26
apps/sync-server/src/middleware/requestLoggingMiddleware.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
import type { Request, Response } from "express";
|
||||
import type { NextFunction } from "express";
|
||||
|
||||
import type { StructuredLogger } from "../logging/jsonFileLogger";
|
||||
|
||||
export function createRequestLoggingMiddleware(logger: StructuredLogger) {
|
||||
return (request: Request, response: Response, next: NextFunction) => {
|
||||
const requestId = globalThis.crypto.randomUUID();
|
||||
const startedAt = Date.now();
|
||||
|
||||
response.locals.requestId = requestId;
|
||||
response.setHeader("x-request-id", requestId);
|
||||
|
||||
response.on("finish", () => {
|
||||
logger.info("http-request", {
|
||||
requestId,
|
||||
method: request.method,
|
||||
path: request.originalUrl,
|
||||
statusCode: response.statusCode,
|
||||
durationMs: Date.now() - startedAt
|
||||
});
|
||||
});
|
||||
|
||||
next();
|
||||
};
|
||||
}
|
||||
206
apps/sync-server/src/routes/createSyncRouter.ts
Normal file
206
apps/sync-server/src/routes/createSyncRouter.ts
Normal file
@@ -0,0 +1,206 @@
|
||||
import type { Request, Response } from "express";
|
||||
import express from "express";
|
||||
|
||||
import {
|
||||
type ClientLogUploadRequest,
|
||||
type ListDevicesRequest,
|
||||
type RegisterDeviceRequest,
|
||||
type RevokeDeviceRequest,
|
||||
type RotateVaultKeyRequest,
|
||||
type SyncPullRequest,
|
||||
type SyncPushRequest,
|
||||
ClientLogUploadRequestSchema,
|
||||
ClientLogUploadResponseSchema,
|
||||
ListDevicesRequestSchema,
|
||||
ListDevicesResponseSchema,
|
||||
RegisterDeviceRequestSchema,
|
||||
RevokeDeviceRequestSchema,
|
||||
RevokeDeviceResponseSchema,
|
||||
RotateVaultKeyRequestSchema,
|
||||
RotateVaultKeyResponseSchema,
|
||||
SyncPullRequestSchema,
|
||||
SyncPushRequestSchema,
|
||||
parseWithSchema
|
||||
} from "@obsidian-sync/sync-protocol";
|
||||
|
||||
import type { StructuredLogger } from "../logging/jsonFileLogger";
|
||||
import type { SyncStore } from "../store/syncStore";
|
||||
|
||||
function readBearerToken(request: Request): string | undefined {
|
||||
const authorization = request.header("authorization");
|
||||
return authorization?.replace(/^Bearer\s+/i, "");
|
||||
}
|
||||
|
||||
function getRequestId(response: Response): string {
|
||||
return String(response.locals.requestId ?? "unknown-request");
|
||||
}
|
||||
|
||||
function unauthorized(request: Request, response: Response, logger: StructuredLogger): void {
|
||||
logger.warn("request-unauthorized", {
|
||||
requestId: getRequestId(response),
|
||||
method: request.method,
|
||||
path: request.originalUrl
|
||||
});
|
||||
|
||||
response.status(401).json({
|
||||
error: "Unauthorized"
|
||||
});
|
||||
}
|
||||
|
||||
function badRequest(request: Request, response: Response, logger: StructuredLogger, error: unknown): void {
|
||||
logger.warn("request-invalid", {
|
||||
requestId: getRequestId(response),
|
||||
method: request.method,
|
||||
path: request.originalUrl,
|
||||
error: error instanceof Error ? error.message : "Invalid request"
|
||||
});
|
||||
|
||||
response.status(400).json({
|
||||
error: error instanceof Error ? error.message : "Invalid request"
|
||||
});
|
||||
}
|
||||
|
||||
export function createSyncRouter(store: SyncStore, logger: StructuredLogger): express.Router {
|
||||
const router = express.Router();
|
||||
|
||||
router.post("/devices/register", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: RegisterDeviceRequest = parseWithSchema(RegisterDeviceRequestSchema, request.body);
|
||||
const registered = store.registerDevice(body);
|
||||
logger.info("device-registered", {
|
||||
requestId: getRequestId(response),
|
||||
vaultId: registered.vaultId,
|
||||
deviceId: registered.deviceId,
|
||||
deviceName: body.deviceName
|
||||
});
|
||||
response.status(201).json(registered);
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
router.post("/sync/pull", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: SyncPullRequest = parseWithSchema(SyncPullRequestSchema, request.body);
|
||||
const token = readBearerToken(request);
|
||||
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||
unauthorized(request, response, logger);
|
||||
return;
|
||||
}
|
||||
|
||||
response.json(store.pull(body.vaultId, body.sinceServerRevision, body.limit));
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
router.post("/sync/push", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: SyncPushRequest = parseWithSchema(SyncPushRequestSchema, request.body);
|
||||
const token = readBearerToken(request);
|
||||
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||
unauthorized(request, response, logger);
|
||||
return;
|
||||
}
|
||||
|
||||
response.json(store.push(body));
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
router.post("/devices/list", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: ListDevicesRequest = parseWithSchema(ListDevicesRequestSchema, request.body);
|
||||
const token = readBearerToken(request);
|
||||
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||
unauthorized(request, response, logger);
|
||||
return;
|
||||
}
|
||||
|
||||
response.json(ListDevicesResponseSchema.parse(store.listDevices(body.vaultId)));
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
router.post("/devices/revoke", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: RevokeDeviceRequest = parseWithSchema(RevokeDeviceRequestSchema, request.body);
|
||||
const token = readBearerToken(request);
|
||||
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||
unauthorized(request, response, logger);
|
||||
return;
|
||||
}
|
||||
|
||||
const revoked = store.revokeDevice(body.vaultId, body.targetDeviceId);
|
||||
logger.warn("device-revoked", {
|
||||
requestId: getRequestId(response),
|
||||
vaultId: body.vaultId,
|
||||
requestedBy: body.deviceId,
|
||||
targetDeviceId: body.targetDeviceId,
|
||||
revokedAt: revoked.revokedAt
|
||||
});
|
||||
response.json(RevokeDeviceResponseSchema.parse(revoked));
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
router.post("/keys/rotate", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: RotateVaultKeyRequest = parseWithSchema(RotateVaultKeyRequestSchema, request.body);
|
||||
const token = readBearerToken(request);
|
||||
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||
unauthorized(request, response, logger);
|
||||
return;
|
||||
}
|
||||
|
||||
const rotated = store.rotateVaultKey(body.vaultId, body.nextKeyId, body.previousKeyId);
|
||||
logger.info("vault-key-rotated", {
|
||||
requestId: getRequestId(response),
|
||||
vaultId: body.vaultId,
|
||||
requestedBy: body.deviceId,
|
||||
activeKeyId: rotated.activeKeyId,
|
||||
rotatedAt: rotated.rotatedAt
|
||||
});
|
||||
response.json(RotateVaultKeyResponseSchema.parse(rotated));
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
router.post("/logs", (request: Request, response: Response) => {
|
||||
try {
|
||||
const body: ClientLogUploadRequest = parseWithSchema(ClientLogUploadRequestSchema, request.body);
|
||||
const token = readBearerToken(request);
|
||||
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
|
||||
unauthorized(request, response, logger);
|
||||
return;
|
||||
}
|
||||
|
||||
for (const entry of body.entries) {
|
||||
logger.log(entry.level, "client-sync-log", {
|
||||
requestId: getRequestId(response),
|
||||
vaultId: body.vaultId,
|
||||
deviceId: body.deviceId,
|
||||
runId: body.runId,
|
||||
clientTimestamp: entry.timestamp,
|
||||
clientMessage: entry.message,
|
||||
context: entry.context ?? {}
|
||||
});
|
||||
}
|
||||
|
||||
response.status(202).json(
|
||||
ClientLogUploadResponseSchema.parse({
|
||||
accepted: body.entries.length,
|
||||
requestId: getRequestId(response)
|
||||
})
|
||||
);
|
||||
} catch (error) {
|
||||
badRequest(request, response, logger, error);
|
||||
}
|
||||
});
|
||||
|
||||
return router;
|
||||
}
|
||||
341
apps/sync-server/src/store/fileSyncStore.ts
Normal file
341
apps/sync-server/src/store/fileSyncStore.ts
Normal file
@@ -0,0 +1,341 @@
|
||||
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
import type {
|
||||
ListDevicesResponse,
|
||||
RegisterDeviceRequest,
|
||||
RegisterDeviceResponse,
|
||||
RevokeDeviceResponse,
|
||||
RotateVaultKeyResponse,
|
||||
SyncChange,
|
||||
SyncConflict,
|
||||
SyncFileRecord,
|
||||
SyncPullResponse,
|
||||
SyncPushRequest,
|
||||
SyncPushResponse,
|
||||
Tombstone
|
||||
} from "@obsidian-sync/sync-protocol";
|
||||
|
||||
import type { SyncStore } from "./syncStore";
|
||||
|
||||
interface VaultDevice {
|
||||
deviceId: string;
|
||||
deviceName: string;
|
||||
token: string;
|
||||
issuedAt: string;
|
||||
revokedAt?: string;
|
||||
}
|
||||
|
||||
interface VaultState {
|
||||
serverRevision: number;
|
||||
devices: Map<string, VaultDevice>;
|
||||
files: Map<string, SyncFileRecord>;
|
||||
tombstones: Map<string, Tombstone>;
|
||||
changes: SyncChange[];
|
||||
activeKeyId?: string;
|
||||
keyRotatedAt?: string;
|
||||
}
|
||||
|
||||
interface PersistedVaultState {
|
||||
serverRevision: number;
|
||||
devices: Record<string, VaultDevice>;
|
||||
files: Record<string, SyncFileRecord>;
|
||||
tombstones: Record<string, Tombstone>;
|
||||
changes: SyncChange[];
|
||||
activeKeyId?: string;
|
||||
keyRotatedAt?: string;
|
||||
}
|
||||
|
||||
interface PersistedStore {
|
||||
vaults: Record<string, PersistedVaultState>;
|
||||
}
|
||||
|
||||
function createVaultState(): VaultState {
|
||||
return {
|
||||
serverRevision: 0,
|
||||
devices: new Map(),
|
||||
files: new Map(),
|
||||
tombstones: new Map(),
|
||||
changes: []
|
||||
};
|
||||
}
|
||||
|
||||
function mapFromRecord<TValue>(record: Record<string, TValue> | undefined): Map<string, TValue> {
|
||||
return new Map(Object.entries(record ?? {}));
|
||||
}
|
||||
|
||||
function recordFromMap<TValue>(map: Map<string, TValue>): Record<string, TValue> {
|
||||
return Object.fromEntries(map.entries());
|
||||
}
|
||||
|
||||
function loadVaults(filePath: string): Map<string, VaultState> {
|
||||
if (!existsSync(filePath)) {
|
||||
return new Map();
|
||||
}
|
||||
|
||||
const raw = readFileSync(filePath, "utf8");
|
||||
if (!raw.trim()) {
|
||||
return new Map();
|
||||
}
|
||||
|
||||
const parsed = JSON.parse(raw) as Partial<PersistedStore>;
|
||||
const vaults = new Map<string, VaultState>();
|
||||
|
||||
for (const [vaultId, vault] of Object.entries(parsed.vaults ?? {})) {
|
||||
vaults.set(vaultId, {
|
||||
serverRevision: vault.serverRevision ?? 0,
|
||||
devices: mapFromRecord(vault.devices),
|
||||
files: mapFromRecord(vault.files),
|
||||
tombstones: mapFromRecord(vault.tombstones),
|
||||
changes: [...(vault.changes ?? [])],
|
||||
activeKeyId: vault.activeKeyId,
|
||||
keyRotatedAt: vault.keyRotatedAt
|
||||
});
|
||||
}
|
||||
|
||||
return vaults;
|
||||
}
|
||||
|
||||
function persistVaults(filePath: string, vaults: Map<string, VaultState>): void {
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
|
||||
const payload: PersistedStore = {
|
||||
vaults: Object.fromEntries(
|
||||
[...vaults.entries()].map(([vaultId, vault]) => [
|
||||
vaultId,
|
||||
{
|
||||
serverRevision: vault.serverRevision,
|
||||
devices: recordFromMap(vault.devices),
|
||||
files: recordFromMap(vault.files),
|
||||
tombstones: recordFromMap(vault.tombstones),
|
||||
changes: vault.changes,
|
||||
activeKeyId: vault.activeKeyId,
|
||||
keyRotatedAt: vault.keyRotatedAt
|
||||
}
|
||||
])
|
||||
)
|
||||
};
|
||||
|
||||
const temporaryPath = `${filePath}.tmp`;
|
||||
writeFileSync(temporaryPath, JSON.stringify(payload, null, 2), "utf8");
|
||||
renameSync(temporaryPath, filePath);
|
||||
}
|
||||
|
||||
function appendFileChange(vault: VaultState, file: SyncFileRecord): void {
|
||||
vault.serverRevision += 1;
|
||||
vault.changes.push({
|
||||
serverRevision: vault.serverRevision,
|
||||
file
|
||||
});
|
||||
}
|
||||
|
||||
function appendTombstoneChange(vault: VaultState, tombstone: Tombstone): void {
|
||||
vault.serverRevision += 1;
|
||||
vault.changes.push({
|
||||
serverRevision: vault.serverRevision,
|
||||
tombstone
|
||||
});
|
||||
}
|
||||
|
||||
function detectFileConflict(current: SyncFileRecord | undefined, incoming: SyncFileRecord): SyncConflict | undefined {
|
||||
if (!current) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (current.manifest.contentHash === incoming.manifest.contentHash) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (!incoming.manifest.baseRevisionId || incoming.manifest.baseRevisionId !== current.manifest.revisionId) {
|
||||
return {
|
||||
path: incoming.manifest.path,
|
||||
serverRevisionId: current.manifest.revisionId,
|
||||
clientRevisionId: incoming.manifest.revisionId,
|
||||
reason: "revision-mismatch"
|
||||
};
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function detectTombstoneConflict(current: SyncFileRecord | undefined, incoming: Tombstone): SyncConflict | undefined {
|
||||
if (!current) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (!incoming.baseRevisionId || incoming.baseRevisionId !== current.manifest.revisionId) {
|
||||
return {
|
||||
path: incoming.path,
|
||||
serverRevisionId: current.manifest.revisionId,
|
||||
clientRevisionId: incoming.revisionId,
|
||||
reason: "revision-mismatch"
|
||||
};
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export function createFileSyncStore(filePath: string): SyncStore {
|
||||
const vaults = loadVaults(filePath);
|
||||
|
||||
function getVault(vaultId: string): VaultState {
|
||||
let vault = vaults.get(vaultId);
|
||||
if (!vault) {
|
||||
vault = createVaultState();
|
||||
vaults.set(vaultId, vault);
|
||||
}
|
||||
|
||||
return vault;
|
||||
}
|
||||
|
||||
function persist(): void {
|
||||
persistVaults(filePath, vaults);
|
||||
}
|
||||
|
||||
return {
|
||||
registerDevice(request: RegisterDeviceRequest): RegisterDeviceResponse {
|
||||
const vault = getVault(request.vaultId);
|
||||
const deviceId = globalThis.crypto.randomUUID();
|
||||
const token = globalThis.crypto.randomUUID();
|
||||
const issuedAt = new Date().toISOString();
|
||||
|
||||
vault.devices.set(deviceId, {
|
||||
deviceId,
|
||||
deviceName: request.deviceName,
|
||||
token,
|
||||
issuedAt
|
||||
});
|
||||
|
||||
persist();
|
||||
|
||||
return {
|
||||
vaultId: request.vaultId,
|
||||
deviceId,
|
||||
token,
|
||||
issuedAt
|
||||
};
|
||||
},
|
||||
|
||||
authenticate(vaultId: string, deviceId: string, token: string): boolean {
|
||||
const vault = vaults.get(vaultId);
|
||||
const device = vault?.devices.get(deviceId);
|
||||
return Boolean(device && !device.revokedAt && device.token === token);
|
||||
},
|
||||
|
||||
pull(vaultId: string, sinceServerRevision: number, limit?: number): SyncPullResponse {
|
||||
const vault = getVault(vaultId);
|
||||
const matchingChanges = vault.changes.filter((change) => change.serverRevision > sinceServerRevision);
|
||||
const page = matchingChanges.slice(0, limit ?? matchingChanges.length);
|
||||
const nextSinceServerRevision = page.at(-1)?.serverRevision ?? sinceServerRevision;
|
||||
|
||||
return {
|
||||
serverRevision: vault.serverRevision,
|
||||
changes: page,
|
||||
hasMore: matchingChanges.length > page.length,
|
||||
nextSinceServerRevision,
|
||||
activeKeyId: vault.activeKeyId,
|
||||
keyRotatedAt: vault.keyRotatedAt
|
||||
};
|
||||
},
|
||||
|
||||
push(request: SyncPushRequest): SyncPushResponse {
|
||||
const vault = getVault(request.vaultId);
|
||||
const acceptedFilePaths: string[] = [];
|
||||
const acceptedTombstones: string[] = [];
|
||||
const conflicts: SyncConflict[] = [];
|
||||
let acceptedKeyId: string | undefined;
|
||||
|
||||
for (const file of request.files) {
|
||||
const current = vault.files.get(file.manifest.path);
|
||||
const conflict = detectFileConflict(current, file);
|
||||
if (conflict) {
|
||||
conflicts.push(conflict);
|
||||
continue;
|
||||
}
|
||||
|
||||
vault.files.set(file.manifest.path, file);
|
||||
vault.tombstones.delete(file.manifest.path);
|
||||
appendFileChange(vault, file);
|
||||
acceptedFilePaths.push(file.manifest.path);
|
||||
acceptedKeyId = file.envelope.keyId;
|
||||
}
|
||||
|
||||
for (const tombstone of request.tombstones) {
|
||||
const current = vault.files.get(tombstone.path);
|
||||
const conflict = detectTombstoneConflict(current, tombstone);
|
||||
if (conflict) {
|
||||
conflicts.push(conflict);
|
||||
continue;
|
||||
}
|
||||
|
||||
vault.files.delete(tombstone.path);
|
||||
vault.tombstones.set(tombstone.path, tombstone);
|
||||
appendTombstoneChange(vault, tombstone);
|
||||
acceptedTombstones.push(tombstone.path);
|
||||
}
|
||||
|
||||
if (acceptedKeyId && acceptedKeyId !== vault.activeKeyId) {
|
||||
vault.activeKeyId = acceptedKeyId;
|
||||
vault.keyRotatedAt = new Date().toISOString();
|
||||
}
|
||||
|
||||
if (acceptedFilePaths.length > 0 || acceptedTombstones.length > 0) {
|
||||
persist();
|
||||
}
|
||||
|
||||
return {
|
||||
acceptedServerRevision: vault.serverRevision,
|
||||
acceptedFilePaths,
|
||||
acceptedTombstones,
|
||||
conflicts,
|
||||
activeKeyId: vault.activeKeyId,
|
||||
keyRotatedAt: vault.keyRotatedAt
|
||||
};
|
||||
},
|
||||
|
||||
listDevices(vaultId: string): ListDevicesResponse {
|
||||
const vault = getVault(vaultId);
|
||||
const devices = [...vault.devices.values()].sort((left, right) => left.issuedAt.localeCompare(right.issuedAt));
|
||||
return {
|
||||
devices,
|
||||
activeKeyId: vault.activeKeyId,
|
||||
keyRotatedAt: vault.keyRotatedAt
|
||||
};
|
||||
},
|
||||
|
||||
revokeDevice(vaultId: string, targetDeviceId: string): RevokeDeviceResponse {
|
||||
const vault = getVault(vaultId);
|
||||
const device = vault.devices.get(targetDeviceId);
|
||||
if (!device) {
|
||||
throw new Error("Device not found.");
|
||||
}
|
||||
|
||||
if (!device.revokedAt) {
|
||||
device.revokedAt = new Date().toISOString();
|
||||
vault.devices.set(targetDeviceId, device);
|
||||
persist();
|
||||
}
|
||||
|
||||
return {
|
||||
targetDeviceId,
|
||||
revokedAt: device.revokedAt
|
||||
};
|
||||
},
|
||||
|
||||
rotateVaultKey(vaultId: string, nextKeyId: string, previousKeyId?: string): RotateVaultKeyResponse {
|
||||
const vault = getVault(vaultId);
|
||||
if (previousKeyId && vault.activeKeyId && previousKeyId !== vault.activeKeyId) {
|
||||
throw new Error("The provided previous key does not match the current server key.");
|
||||
}
|
||||
|
||||
vault.activeKeyId = nextKeyId;
|
||||
vault.keyRotatedAt = new Date().toISOString();
|
||||
persist();
|
||||
|
||||
return {
|
||||
activeKeyId: nextKeyId,
|
||||
rotatedAt: vault.keyRotatedAt
|
||||
};
|
||||
}
|
||||
};
|
||||
}
|
||||
20
apps/sync-server/src/store/syncStore.ts
Normal file
20
apps/sync-server/src/store/syncStore.ts
Normal file
@@ -0,0 +1,20 @@
|
||||
import type {
|
||||
ListDevicesResponse,
|
||||
RegisterDeviceRequest,
|
||||
RegisterDeviceResponse,
|
||||
RevokeDeviceResponse,
|
||||
RotateVaultKeyResponse,
|
||||
SyncPullResponse,
|
||||
SyncPushRequest,
|
||||
SyncPushResponse
|
||||
} from "@obsidian-sync/sync-protocol";
|
||||
|
||||
export interface SyncStore {
|
||||
registerDevice(request: RegisterDeviceRequest): RegisterDeviceResponse;
|
||||
authenticate(vaultId: string, deviceId: string, token: string): boolean;
|
||||
pull(vaultId: string, sinceServerRevision: number, limit?: number): SyncPullResponse;
|
||||
push(request: SyncPushRequest): SyncPushResponse;
|
||||
listDevices(vaultId: string): ListDevicesResponse;
|
||||
revokeDevice(vaultId: string, targetDeviceId: string): RevokeDeviceResponse;
|
||||
rotateVaultKey(vaultId: string, nextKeyId: string, previousKeyId?: string): RotateVaultKeyResponse;
|
||||
}
|
||||
18
apps/sync-server/tsconfig.json
Normal file
18
apps/sync-server/tsconfig.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"composite": true,
|
||||
"declaration": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"module": "CommonJS"
|
||||
},
|
||||
"references": [
|
||||
{
|
||||
"path": "../../packages/sync-protocol"
|
||||
}
|
||||
],
|
||||
"include": [
|
||||
"src/**/*.ts"
|
||||
]
|
||||
}
|
||||
31
docs/architecture.md
Normal file
31
docs/architecture.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Architecture
|
||||
|
||||
## Components
|
||||
|
||||
- `apps/obsidian-plugin`: Obsidian client plugin for local vault scanning, sync orchestration, merge handling, and user notifications.
|
||||
- `apps/sync-server`: Self-hosted HTTP API for device registration and encrypted revision exchange.
|
||||
- `packages/sync-protocol`: Shared schemas and request and response types.
|
||||
- `packages/sync-engine`: Shared merge, hashing, and crypto helpers.
|
||||
|
||||
## Current Data Flow
|
||||
|
||||
1. The plugin registers a device with the server and stores the returned device identity and token.
|
||||
2. The plugin creates or imports a local vault key.
|
||||
3. The plugin pulls encrypted remote changes and applies them locally for both text notes and binary attachments.
|
||||
4. Pull operations are paged so the client can resume from the last applied server revision after an interruption.
|
||||
5. The plugin scans syncable vault files, hashes changed content, encrypts it, and pushes encrypted revisions in batches.
|
||||
6. The server stores ciphertext, revision metadata, tombstones, device state, and active key status in a file-backed state store.
|
||||
7. Each sync run emits client-side diagnostics that can be uploaded to the server and correlated with request logs through request IDs.
|
||||
8. Recovery bundles package the vault ID, server URL, and current client-side vault key so a new device can join the same encrypted vault.
|
||||
|
||||
## Conflict Policy
|
||||
|
||||
- If only one side changed since the last synced base, accept the changed side.
|
||||
- If both sides changed, write conflict markers into the local file, create a separate remote conflict copy, and require user review before upload continues.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
- The server storage backend is durable but still file-based rather than relational.
|
||||
- Key distribution between devices still depends on manually sharing a recovery bundle.
|
||||
- Full plugin integration tests across multiple Obsidian instances are not implemented yet.
|
||||
|
||||
50
docs/deployment.md
Normal file
50
docs/deployment.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Deployment
|
||||
|
||||
## Requirements
|
||||
|
||||
- Node.js 22 or newer
|
||||
- npm 10 or newer
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
## Run The Sync Server
|
||||
|
||||
```bash
|
||||
npm run dev:server
|
||||
```
|
||||
|
||||
The server listens on `http://localhost:8787` by default.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `PORT`: overrides the default HTTP port.
|
||||
- `SYNC_DATA_DIR`: overrides the default data directory. If omitted, the server writes data under `./data` from the workspace root.
|
||||
|
||||
## Data Written By The Server
|
||||
|
||||
- `data/sync-state.json`: durable encrypted revision metadata, devices, tombstones, and sync history.
|
||||
- `data/logs/server.jsonl`: structured JSON-line server and client-sync diagnostics.
|
||||
|
||||
## Management Capabilities
|
||||
|
||||
- `POST /api/devices/list`: lists known devices for the vault plus active key status.
|
||||
- `POST /api/devices/revoke`: revokes a device so it can no longer authenticate.
|
||||
- `POST /api/keys/rotate`: stores the currently active vault key identifier.
|
||||
- `POST /api/sync/pull`: supports paged pulls through `sinceServerRevision` and `limit`.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- The server never needs plaintext note bodies. Clients encrypt before upload and decrypt after download.
|
||||
- Request logs include a response header named `x-request-id` for correlating client and server failures.
|
||||
- The current storage backend is file-based. It is durable for a single-node deployment but not yet designed for clustered or high-availability hosting.
|
||||
- The plugin updates local sync state after each accepted push batch and each applied pull page, which lets interrupted sync runs resume from the last durable checkpoint.
|
||||
1723
package-lock.json
generated
Normal file
1723
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
20
package.json
Normal file
20
package.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"name": "obsidian-sync",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"workspaces": [
|
||||
"apps/*",
|
||||
"packages/*"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "tsc -b tsconfig.json",
|
||||
"test": "tsx --test tests/**/*.test.ts",
|
||||
"typecheck": "tsc -b tsconfig.json --pretty false",
|
||||
"dev:server": "npm run dev --workspace @obsidian-sync/sync-server"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.14.1",
|
||||
"tsx": "^4.19.3",
|
||||
"typescript": "^5.8.3"
|
||||
}
|
||||
}
|
||||
14
packages/sync-engine/package.json
Normal file
14
packages/sync-engine/package.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"name": "@obsidian-sync/sync-engine",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsc -b",
|
||||
"typecheck": "tsc -b --pretty false"
|
||||
},
|
||||
"dependencies": {
|
||||
"@obsidian-sync/sync-protocol": "0.1.0"
|
||||
}
|
||||
}
|
||||
182
packages/sync-engine/src/crypto.ts
Normal file
182
packages/sync-engine/src/crypto.ts
Normal file
@@ -0,0 +1,182 @@
|
||||
import type { EncryptionEnvelope } from "@obsidian-sync/sync-protocol";
|
||||
|
||||
export interface VaultKeyHandle {
|
||||
keyId: string;
|
||||
key: CryptoKey;
|
||||
}
|
||||
|
||||
export interface GeneratedVaultKey extends VaultKeyHandle {
|
||||
exportedKey: string;
|
||||
}
|
||||
|
||||
const BASE64_ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
||||
|
||||
function getCryptoApi(): Crypto {
|
||||
if (!globalThis.crypto?.subtle) {
|
||||
throw new Error("Web Crypto API is unavailable in this environment.");
|
||||
}
|
||||
|
||||
return globalThis.crypto;
|
||||
}
|
||||
|
||||
function bytesToBase64(bytes: Uint8Array): string {
|
||||
let output = "";
|
||||
|
||||
for (let index = 0; index < bytes.length; index += 3) {
|
||||
const first = bytes[index] ?? 0;
|
||||
const second = bytes[index + 1] ?? 0;
|
||||
const third = bytes[index + 2] ?? 0;
|
||||
const combined = (first << 16) | (second << 8) | third;
|
||||
|
||||
output += BASE64_ALPHABET[(combined >> 18) & 63];
|
||||
output += BASE64_ALPHABET[(combined >> 12) & 63];
|
||||
output += index + 1 < bytes.length ? BASE64_ALPHABET[(combined >> 6) & 63] : "=";
|
||||
output += index + 2 < bytes.length ? BASE64_ALPHABET[combined & 63] : "=";
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
|
||||
function base64ToBytes(value: string): Uint8Array {
|
||||
const normalized = value.replace(/\s+/g, "");
|
||||
const padding = normalized.endsWith("==") ? 2 : normalized.endsWith("=") ? 1 : 0;
|
||||
const outputLength = Math.floor((normalized.length * 3) / 4) - padding;
|
||||
const bytes = new Uint8Array(outputLength);
|
||||
|
||||
let byteIndex = 0;
|
||||
for (let index = 0; index < normalized.length; index += 4) {
|
||||
const first = BASE64_ALPHABET.indexOf(normalized[index] ?? "A");
|
||||
const second = BASE64_ALPHABET.indexOf(normalized[index + 1] ?? "A");
|
||||
const thirdChar = normalized[index + 2] ?? "=";
|
||||
const fourthChar = normalized[index + 3] ?? "=";
|
||||
const third = thirdChar === "=" ? 0 : BASE64_ALPHABET.indexOf(thirdChar);
|
||||
const fourth = fourthChar === "=" ? 0 : BASE64_ALPHABET.indexOf(fourthChar);
|
||||
const combined = (first << 18) | (second << 12) | (third << 6) | fourth;
|
||||
|
||||
bytes[byteIndex] = (combined >> 16) & 255;
|
||||
byteIndex += 1;
|
||||
|
||||
if (thirdChar !== "=" && byteIndex < outputLength + 1) {
|
||||
bytes[byteIndex] = (combined >> 8) & 255;
|
||||
byteIndex += 1;
|
||||
}
|
||||
|
||||
if (fourthChar !== "=" && byteIndex < outputLength + 1) {
|
||||
bytes[byteIndex] = combined & 255;
|
||||
byteIndex += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return bytes;
|
||||
}
|
||||
|
||||
function toOwnedBytes(bytes: Uint8Array): Uint8Array<ArrayBuffer> {
|
||||
const buffer = new ArrayBuffer(bytes.byteLength);
|
||||
const view = new Uint8Array(buffer);
|
||||
view.set(bytes);
|
||||
return view;
|
||||
}
|
||||
|
||||
function normalizeBytes(value: ArrayBuffer | Uint8Array): Uint8Array<ArrayBuffer> {
|
||||
return value instanceof Uint8Array ? toOwnedBytes(value) : toOwnedBytes(new Uint8Array(value));
|
||||
}
|
||||
|
||||
function toArrayBuffer(bytes: Uint8Array): ArrayBuffer {
|
||||
return toOwnedBytes(bytes).buffer;
|
||||
}
|
||||
|
||||
export async function generateVaultKey(keyId = globalThis.crypto.randomUUID()): Promise<GeneratedVaultKey> {
|
||||
const cryptoApi = getCryptoApi();
|
||||
const key = await cryptoApi.subtle.generateKey(
|
||||
{
|
||||
name: "AES-GCM",
|
||||
length: 256
|
||||
},
|
||||
true,
|
||||
["encrypt", "decrypt"]
|
||||
);
|
||||
|
||||
const rawKey = await cryptoApi.subtle.exportKey("raw", key);
|
||||
|
||||
return {
|
||||
keyId,
|
||||
key,
|
||||
exportedKey: bytesToBase64(new Uint8Array(rawKey))
|
||||
};
|
||||
}
|
||||
|
||||
export async function importVaultKey(exportedKey: string, keyId: string): Promise<VaultKeyHandle> {
|
||||
const cryptoApi = getCryptoApi();
|
||||
const rawKey = base64ToBytes(exportedKey);
|
||||
const key = await cryptoApi.subtle.importKey(
|
||||
"raw",
|
||||
toArrayBuffer(rawKey),
|
||||
{
|
||||
name: "AES-GCM",
|
||||
length: 256
|
||||
},
|
||||
true,
|
||||
["encrypt", "decrypt"]
|
||||
);
|
||||
|
||||
return {
|
||||
keyId,
|
||||
key
|
||||
};
|
||||
}
|
||||
|
||||
export async function encryptBytes(plaintext: ArrayBuffer | Uint8Array, vaultKey: VaultKeyHandle): Promise<EncryptionEnvelope> {
|
||||
const cryptoApi = getCryptoApi();
|
||||
const iv = cryptoApi.getRandomValues(new Uint8Array(12));
|
||||
const ciphertext = await cryptoApi.subtle.encrypt(
|
||||
{
|
||||
name: "AES-GCM",
|
||||
iv
|
||||
},
|
||||
vaultKey.key,
|
||||
normalizeBytes(plaintext)
|
||||
);
|
||||
|
||||
return {
|
||||
algorithm: "AES-GCM-256",
|
||||
keyId: vaultKey.keyId,
|
||||
iv: bytesToBase64(iv),
|
||||
ciphertext: bytesToBase64(new Uint8Array(ciphertext))
|
||||
};
|
||||
}
|
||||
|
||||
export async function decryptBytes(envelope: EncryptionEnvelope, vaultKey: VaultKeyHandle): Promise<ArrayBuffer> {
|
||||
const cryptoApi = getCryptoApi();
|
||||
const plaintext = await cryptoApi.subtle.decrypt(
|
||||
{
|
||||
name: "AES-GCM",
|
||||
iv: toOwnedBytes(base64ToBytes(envelope.iv))
|
||||
},
|
||||
vaultKey.key,
|
||||
toOwnedBytes(base64ToBytes(envelope.ciphertext))
|
||||
);
|
||||
|
||||
return plaintext.slice(0);
|
||||
}
|
||||
|
||||
export async function encryptText(plaintext: string, vaultKey: VaultKeyHandle): Promise<EncryptionEnvelope> {
|
||||
return encryptBytes(new TextEncoder().encode(plaintext), vaultKey);
|
||||
}
|
||||
|
||||
export async function decryptText(envelope: EncryptionEnvelope, vaultKey: VaultKeyHandle): Promise<string> {
|
||||
return new TextDecoder().decode(await decryptBytes(envelope, vaultKey));
|
||||
}
|
||||
|
||||
async function computeHash(bytes: Uint8Array): Promise<string> {
|
||||
const cryptoApi = getCryptoApi();
|
||||
const digest = await cryptoApi.subtle.digest("SHA-256", toOwnedBytes(bytes));
|
||||
return bytesToBase64(new Uint8Array(digest));
|
||||
}
|
||||
|
||||
export async function computeTextHash(value: string): Promise<string> {
|
||||
return computeHash(new TextEncoder().encode(value));
|
||||
}
|
||||
|
||||
export async function computeBinaryHash(value: ArrayBuffer | Uint8Array): Promise<string> {
|
||||
return computeHash(normalizeBytes(value));
|
||||
}
|
||||
2
packages/sync-engine/src/index.ts
Normal file
2
packages/sync-engine/src/index.ts
Normal file
@@ -0,0 +1,2 @@
|
||||
export * from "./crypto";
|
||||
export * from "./merge";
|
||||
59
packages/sync-engine/src/merge.ts
Normal file
59
packages/sync-engine/src/merge.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
export interface MergeTextInput {
|
||||
base?: string;
|
||||
local: string;
|
||||
remote: string;
|
||||
}
|
||||
|
||||
export interface MergeTextResult {
|
||||
status: "merged" | "conflict";
|
||||
content: string;
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export function mergeTextRevisions(input: MergeTextInput): MergeTextResult {
|
||||
const base = input.base ?? "";
|
||||
|
||||
if (input.local === input.remote) {
|
||||
return {
|
||||
status: "merged",
|
||||
content: input.local
|
||||
};
|
||||
}
|
||||
|
||||
if (base === input.local) {
|
||||
return {
|
||||
status: "merged",
|
||||
content: input.remote
|
||||
};
|
||||
}
|
||||
|
||||
if (base === input.remote) {
|
||||
return {
|
||||
status: "merged",
|
||||
content: input.local
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
status: "conflict",
|
||||
reason: "Both local and remote content changed since the last shared base.",
|
||||
content: [
|
||||
"<<<<<<< LOCAL",
|
||||
input.local,
|
||||
"=======",
|
||||
input.remote,
|
||||
">>>>>>> REMOTE"
|
||||
].join("\n")
|
||||
};
|
||||
}
|
||||
|
||||
export function createConflictCopyPath(path: string): string {
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
|
||||
const extensionIndex = path.lastIndexOf(".");
|
||||
|
||||
if (extensionIndex <= 0) {
|
||||
return `${path}.remote-conflict.${timestamp}`;
|
||||
}
|
||||
|
||||
return `${path.slice(0, extensionIndex)}.remote-conflict.${timestamp}${path.slice(extensionIndex)}`;
|
||||
}
|
||||
18
packages/sync-engine/tsconfig.json
Normal file
18
packages/sync-engine/tsconfig.json
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"composite": true,
|
||||
"declaration": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"module": "CommonJS"
|
||||
},
|
||||
"references": [
|
||||
{
|
||||
"path": "../sync-protocol"
|
||||
}
|
||||
],
|
||||
"include": [
|
||||
"src/**/*.ts"
|
||||
]
|
||||
}
|
||||
14
packages/sync-protocol/package.json
Normal file
14
packages/sync-protocol/package.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"name": "@obsidian-sync/sync-protocol",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsc -b",
|
||||
"typecheck": "tsc -b --pretty false"
|
||||
},
|
||||
"dependencies": {
|
||||
"zod": "^3.24.2"
|
||||
}
|
||||
}
|
||||
194
packages/sync-protocol/src/index.ts
Normal file
194
packages/sync-protocol/src/index.ts
Normal file
@@ -0,0 +1,194 @@
|
||||
import { z } from "zod";
|
||||
|
||||
export const SyncFileKindSchema = z.enum(["text", "binary"]);
|
||||
export type SyncFileKind = z.infer<typeof SyncFileKindSchema>;
|
||||
|
||||
export const EncryptionEnvelopeSchema = z.object({
|
||||
algorithm: z.literal("AES-GCM-256"),
|
||||
keyId: z.string().min(1),
|
||||
iv: z.string().min(1),
|
||||
ciphertext: z.string().min(1)
|
||||
});
|
||||
export type EncryptionEnvelope = z.infer<typeof EncryptionEnvelopeSchema>;
|
||||
|
||||
export const FileManifestSchema = z.object({
|
||||
path: z.string().min(1),
|
||||
kind: SyncFileKindSchema,
|
||||
contentHash: z.string().min(1),
|
||||
revisionId: z.string().min(1),
|
||||
baseRevisionId: z.string().min(1).optional(),
|
||||
updatedAt: z.string().datetime(),
|
||||
sizeBytes: z.number().int().nonnegative(),
|
||||
deviceId: z.string().min(1)
|
||||
});
|
||||
export type FileManifest = z.infer<typeof FileManifestSchema>;
|
||||
|
||||
export const SyncFileRecordSchema = z.object({
|
||||
manifest: FileManifestSchema,
|
||||
envelope: EncryptionEnvelopeSchema
|
||||
});
|
||||
export type SyncFileRecord = z.infer<typeof SyncFileRecordSchema>;
|
||||
|
||||
export const TombstoneSchema = z.object({
|
||||
path: z.string().min(1),
|
||||
revisionId: z.string().min(1),
|
||||
baseRevisionId: z.string().min(1).optional(),
|
||||
deletedAt: z.string().datetime(),
|
||||
deviceId: z.string().min(1)
|
||||
});
|
||||
export type Tombstone = z.infer<typeof TombstoneSchema>;
|
||||
|
||||
export const SyncConflictSchema = z.object({
|
||||
path: z.string().min(1),
|
||||
serverRevisionId: z.string().min(1),
|
||||
clientRevisionId: z.string().min(1),
|
||||
reason: z.enum(["revision-mismatch"])
|
||||
});
|
||||
export type SyncConflict = z.infer<typeof SyncConflictSchema>;
|
||||
|
||||
export const SyncChangeSchema = z
|
||||
.object({
|
||||
serverRevision: z.number().int().nonnegative(),
|
||||
file: SyncFileRecordSchema.optional(),
|
||||
tombstone: TombstoneSchema.optional()
|
||||
})
|
||||
.refine((value) => Number(Boolean(value.file)) + Number(Boolean(value.tombstone)) === 1, {
|
||||
message: "A change must include either a file or a tombstone."
|
||||
});
|
||||
export type SyncChange = z.infer<typeof SyncChangeSchema>;
|
||||
|
||||
export const RegisterDeviceRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceName: z.string().min(1)
|
||||
});
|
||||
export type RegisterDeviceRequest = z.infer<typeof RegisterDeviceRequestSchema>;
|
||||
|
||||
export const RegisterDeviceResponseSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1),
|
||||
token: z.string().min(1),
|
||||
issuedAt: z.string().datetime()
|
||||
});
|
||||
export type RegisterDeviceResponse = z.infer<typeof RegisterDeviceResponseSchema>;
|
||||
|
||||
export const RecoveryBundleSchema = z.object({
|
||||
version: z.literal(1),
|
||||
serverUrl: z.string().min(1),
|
||||
vaultId: z.string().min(1),
|
||||
keyId: z.string().min(1),
|
||||
exportedVaultKey: z.string().min(1),
|
||||
generatedAt: z.string().datetime()
|
||||
});
|
||||
export type RecoveryBundle = z.infer<typeof RecoveryBundleSchema>;
|
||||
|
||||
export const DeviceRecordSchema = z.object({
|
||||
deviceId: z.string().min(1),
|
||||
deviceName: z.string().min(1),
|
||||
issuedAt: z.string().datetime(),
|
||||
revokedAt: z.string().datetime().optional()
|
||||
});
|
||||
export type DeviceRecord = z.infer<typeof DeviceRecordSchema>;
|
||||
|
||||
export const ListDevicesRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1)
|
||||
});
|
||||
export type ListDevicesRequest = z.infer<typeof ListDevicesRequestSchema>;
|
||||
|
||||
export const ListDevicesResponseSchema = z.object({
|
||||
devices: z.array(DeviceRecordSchema),
|
||||
activeKeyId: z.string().min(1).optional(),
|
||||
keyRotatedAt: z.string().datetime().optional()
|
||||
});
|
||||
export type ListDevicesResponse = z.infer<typeof ListDevicesResponseSchema>;
|
||||
|
||||
export const RevokeDeviceRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1),
|
||||
targetDeviceId: z.string().min(1)
|
||||
});
|
||||
export type RevokeDeviceRequest = z.infer<typeof RevokeDeviceRequestSchema>;
|
||||
|
||||
export const RevokeDeviceResponseSchema = z.object({
|
||||
targetDeviceId: z.string().min(1),
|
||||
revokedAt: z.string().datetime()
|
||||
});
|
||||
export type RevokeDeviceResponse = z.infer<typeof RevokeDeviceResponseSchema>;
|
||||
|
||||
export const RotateVaultKeyRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1),
|
||||
nextKeyId: z.string().min(1),
|
||||
previousKeyId: z.string().min(1).optional()
|
||||
});
|
||||
export type RotateVaultKeyRequest = z.infer<typeof RotateVaultKeyRequestSchema>;
|
||||
|
||||
export const RotateVaultKeyResponseSchema = z.object({
|
||||
activeKeyId: z.string().min(1),
|
||||
rotatedAt: z.string().datetime()
|
||||
});
|
||||
export type RotateVaultKeyResponse = z.infer<typeof RotateVaultKeyResponseSchema>;
|
||||
|
||||
export const SyncPullRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1),
|
||||
sinceServerRevision: z.number().int().nonnegative(),
|
||||
limit: z.number().int().positive().max(500).optional()
|
||||
});
|
||||
export type SyncPullRequest = z.infer<typeof SyncPullRequestSchema>;
|
||||
|
||||
export const SyncPullResponseSchema = z.object({
|
||||
serverRevision: z.number().int().nonnegative(),
|
||||
changes: z.array(SyncChangeSchema),
|
||||
hasMore: z.boolean(),
|
||||
nextSinceServerRevision: z.number().int().nonnegative(),
|
||||
activeKeyId: z.string().min(1).optional(),
|
||||
keyRotatedAt: z.string().datetime().optional()
|
||||
});
|
||||
export type SyncPullResponse = z.infer<typeof SyncPullResponseSchema>;
|
||||
|
||||
export const SyncPushRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1),
|
||||
knownServerRevision: z.number().int().nonnegative(),
|
||||
files: z.array(SyncFileRecordSchema),
|
||||
tombstones: z.array(TombstoneSchema)
|
||||
});
|
||||
export type SyncPushRequest = z.infer<typeof SyncPushRequestSchema>;
|
||||
|
||||
export const SyncPushResponseSchema = z.object({
|
||||
acceptedServerRevision: z.number().int().nonnegative(),
|
||||
acceptedFilePaths: z.array(z.string()),
|
||||
acceptedTombstones: z.array(z.string()),
|
||||
conflicts: z.array(SyncConflictSchema),
|
||||
activeKeyId: z.string().min(1).optional(),
|
||||
keyRotatedAt: z.string().datetime().optional()
|
||||
});
|
||||
export type SyncPushResponse = z.infer<typeof SyncPushResponseSchema>;
|
||||
|
||||
export const ClientLogEntrySchema = z.object({
|
||||
level: z.enum(["debug", "info", "warn", "error"]),
|
||||
message: z.string().min(1),
|
||||
timestamp: z.string().datetime(),
|
||||
deviceId: z.string().min(1),
|
||||
context: z.record(z.string(), z.unknown()).optional()
|
||||
});
|
||||
export type ClientLogEntry = z.infer<typeof ClientLogEntrySchema>;
|
||||
|
||||
export const ClientLogUploadRequestSchema = z.object({
|
||||
vaultId: z.string().min(1),
|
||||
deviceId: z.string().min(1),
|
||||
runId: z.string().min(1),
|
||||
entries: z.array(ClientLogEntrySchema)
|
||||
});
|
||||
export type ClientLogUploadRequest = z.infer<typeof ClientLogUploadRequestSchema>;
|
||||
|
||||
export const ClientLogUploadResponseSchema = z.object({
|
||||
accepted: z.number().int().nonnegative(),
|
||||
requestId: z.string().min(1)
|
||||
});
|
||||
export type ClientLogUploadResponse = z.infer<typeof ClientLogUploadResponseSchema>;
|
||||
|
||||
export function parseWithSchema<T>(schema: z.ZodSchema<T>, value: unknown): T {
|
||||
return schema.parse(value);
|
||||
}
|
||||
13
packages/sync-protocol/tsconfig.json
Normal file
13
packages/sync-protocol/tsconfig.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"composite": true,
|
||||
"declaration": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"module": "CommonJS"
|
||||
},
|
||||
"include": [
|
||||
"src/**/*.ts"
|
||||
]
|
||||
}
|
||||
39
tests/contract/sync-engine.test.ts
Normal file
39
tests/contract/sync-engine.test.ts
Normal file
@@ -0,0 +1,39 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
import {
|
||||
computeBinaryHash,
|
||||
decryptBytes,
|
||||
encryptBytes,
|
||||
generateVaultKey,
|
||||
mergeTextRevisions
|
||||
} from "../../packages/sync-engine/src";
|
||||
|
||||
test("encryptBytes round-trips binary payloads", async () => {
|
||||
const vaultKey = await generateVaultKey("test-key");
|
||||
const original = new Uint8Array([1, 2, 3, 4, 5]).buffer;
|
||||
|
||||
const encrypted = await encryptBytes(original, vaultKey);
|
||||
const decrypted = await decryptBytes(encrypted, vaultKey);
|
||||
|
||||
assert.deepEqual([...new Uint8Array(decrypted)], [1, 2, 3, 4, 5]);
|
||||
});
|
||||
|
||||
test("computeBinaryHash is stable for identical binary payloads", async () => {
|
||||
const first = await computeBinaryHash(new Uint8Array([9, 8, 7, 6]).buffer);
|
||||
const second = await computeBinaryHash(new Uint8Array([9, 8, 7, 6]).buffer);
|
||||
|
||||
assert.equal(first, second);
|
||||
});
|
||||
|
||||
test("mergeTextRevisions reports conflicts when both sides diverge", () => {
|
||||
const result = mergeTextRevisions({
|
||||
base: "alpha",
|
||||
local: "alpha local",
|
||||
remote: "alpha remote"
|
||||
});
|
||||
|
||||
assert.equal(result.status, "conflict");
|
||||
assert.match(result.content, /<<<<<<< LOCAL/);
|
||||
assert.match(result.content, />>>>>>> REMOTE/);
|
||||
});
|
||||
174
tests/e2e/multi-device-store.test.ts
Normal file
174
tests/e2e/multi-device-store.test.ts
Normal file
@@ -0,0 +1,174 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import test from "node:test";
|
||||
|
||||
import { createFileSyncStore } from "../../apps/sync-server/src/store/fileSyncStore";
|
||||
|
||||
test("file-backed store survives restart and serves another device", () => {
|
||||
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-"));
|
||||
const storePath = path.join(tempDirectory, "sync-state.json");
|
||||
|
||||
try {
|
||||
const serverA = createFileSyncStore(storePath);
|
||||
const deviceA = serverA.registerDevice({
|
||||
vaultId: "vault-1",
|
||||
deviceName: "Desktop"
|
||||
});
|
||||
const deviceB = serverA.registerDevice({
|
||||
vaultId: "vault-1",
|
||||
deviceName: "Phone"
|
||||
});
|
||||
|
||||
const pushResponse = serverA.push({
|
||||
vaultId: "vault-1",
|
||||
deviceId: deviceA.deviceId,
|
||||
knownServerRevision: 0,
|
||||
files: [
|
||||
{
|
||||
manifest: {
|
||||
path: "notes/hello.md",
|
||||
kind: "text",
|
||||
contentHash: "hash-1",
|
||||
revisionId: "rev-1",
|
||||
updatedAt: "2026-04-08T00:00:00.000Z",
|
||||
sizeBytes: 12,
|
||||
deviceId: deviceA.deviceId
|
||||
},
|
||||
envelope: {
|
||||
algorithm: "AES-GCM-256",
|
||||
keyId: "key-1",
|
||||
iv: "iv-1",
|
||||
ciphertext: "cipher-1"
|
||||
}
|
||||
}
|
||||
],
|
||||
tombstones: []
|
||||
});
|
||||
|
||||
assert.deepEqual(pushResponse.acceptedFilePaths, ["notes/hello.md"]);
|
||||
|
||||
const serverB = createFileSyncStore(storePath);
|
||||
assert.equal(serverB.authenticate("vault-1", deviceB.deviceId, deviceB.token), true);
|
||||
|
||||
const pullResponse = serverB.pull("vault-1", 0);
|
||||
assert.equal(pullResponse.changes.length, 1);
|
||||
assert.equal(pullResponse.changes[0].file?.manifest.path, "notes/hello.md");
|
||||
assert.equal(pullResponse.serverRevision, 1);
|
||||
assert.equal(pullResponse.hasMore, false);
|
||||
assert.equal(pullResponse.nextSinceServerRevision, 1);
|
||||
} finally {
|
||||
rmSync(tempDirectory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("file-backed store paginates pull results", () => {
|
||||
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-page-"));
|
||||
const storePath = path.join(tempDirectory, "sync-state.json");
|
||||
|
||||
try {
|
||||
const store = createFileSyncStore(storePath);
|
||||
const device = store.registerDevice({
|
||||
vaultId: "vault-2",
|
||||
deviceName: "Desktop"
|
||||
});
|
||||
|
||||
store.push({
|
||||
vaultId: "vault-2",
|
||||
deviceId: device.deviceId,
|
||||
knownServerRevision: 0,
|
||||
files: [
|
||||
{
|
||||
manifest: {
|
||||
path: "a.md",
|
||||
kind: "text",
|
||||
contentHash: "hash-a",
|
||||
revisionId: "rev-a",
|
||||
updatedAt: "2026-04-08T00:00:00.000Z",
|
||||
sizeBytes: 1,
|
||||
deviceId: device.deviceId
|
||||
},
|
||||
envelope: {
|
||||
algorithm: "AES-GCM-256",
|
||||
keyId: "key-a",
|
||||
iv: "iv-a",
|
||||
ciphertext: "cipher-a"
|
||||
}
|
||||
}
|
||||
],
|
||||
tombstones: []
|
||||
});
|
||||
|
||||
store.push({
|
||||
vaultId: "vault-2",
|
||||
deviceId: device.deviceId,
|
||||
knownServerRevision: 1,
|
||||
files: [
|
||||
{
|
||||
manifest: {
|
||||
path: "b.md",
|
||||
kind: "text",
|
||||
contentHash: "hash-b",
|
||||
revisionId: "rev-b",
|
||||
updatedAt: "2026-04-08T00:01:00.000Z",
|
||||
sizeBytes: 1,
|
||||
deviceId: device.deviceId
|
||||
},
|
||||
envelope: {
|
||||
algorithm: "AES-GCM-256",
|
||||
keyId: "key-a",
|
||||
iv: "iv-b",
|
||||
ciphertext: "cipher-b"
|
||||
}
|
||||
}
|
||||
],
|
||||
tombstones: []
|
||||
});
|
||||
|
||||
const firstPage = store.pull("vault-2", 0, 1);
|
||||
assert.equal(firstPage.changes.length, 1);
|
||||
assert.equal(firstPage.hasMore, true);
|
||||
assert.equal(firstPage.nextSinceServerRevision, 1);
|
||||
|
||||
const secondPage = store.pull("vault-2", firstPage.nextSinceServerRevision, 1);
|
||||
assert.equal(secondPage.changes.length, 1);
|
||||
assert.equal(secondPage.hasMore, false);
|
||||
assert.equal(secondPage.nextSinceServerRevision, 2);
|
||||
assert.equal(secondPage.changes[0].file?.manifest.path, "b.md");
|
||||
} finally {
|
||||
rmSync(tempDirectory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("device revocation and key rotation state are durable", () => {
|
||||
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-device-"));
|
||||
const storePath = path.join(tempDirectory, "sync-state.json");
|
||||
|
||||
try {
|
||||
const store = createFileSyncStore(storePath);
|
||||
const desktop = store.registerDevice({
|
||||
vaultId: "vault-3",
|
||||
deviceName: "Desktop"
|
||||
});
|
||||
const phone = store.registerDevice({
|
||||
vaultId: "vault-3",
|
||||
deviceName: "Phone"
|
||||
});
|
||||
|
||||
const rotation = store.rotateVaultKey("vault-3", "key-rotated");
|
||||
assert.equal(rotation.activeKeyId, "key-rotated");
|
||||
|
||||
const revoked = store.revokeDevice("vault-3", phone.deviceId);
|
||||
assert.equal(revoked.targetDeviceId, phone.deviceId);
|
||||
assert.equal(store.authenticate("vault-3", phone.deviceId, phone.token), false);
|
||||
|
||||
const restartedStore = createFileSyncStore(storePath);
|
||||
const devices = restartedStore.listDevices("vault-3");
|
||||
assert.equal(devices.activeKeyId, "key-rotated");
|
||||
assert.equal(devices.devices.find((device) => device.deviceId === phone.deviceId)?.revokedAt !== undefined, true);
|
||||
assert.equal(restartedStore.authenticate("vault-3", desktop.deviceId, desktop.token), true);
|
||||
} finally {
|
||||
rmSync(tempDirectory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
25
tsconfig.base.json
Normal file
25
tsconfig.base.json
Normal file
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"lib": [
|
||||
"ES2022",
|
||||
"DOM"
|
||||
],
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
"baseUrl": ".",
|
||||
"paths": {
|
||||
"@obsidian-sync/sync-protocol": [
|
||||
"packages/sync-protocol/src"
|
||||
],
|
||||
"@obsidian-sync/sync-engine": [
|
||||
"packages/sync-engine/src"
|
||||
]
|
||||
},
|
||||
"moduleResolution": "Node",
|
||||
"esModuleInterop": true,
|
||||
"resolveJsonModule": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"noEmitOnError": true
|
||||
}
|
||||
}
|
||||
17
tsconfig.json
Normal file
17
tsconfig.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"files": [],
|
||||
"references": [
|
||||
{
|
||||
"path": "./packages/sync-protocol"
|
||||
},
|
||||
{
|
||||
"path": "./packages/sync-engine"
|
||||
},
|
||||
{
|
||||
"path": "./apps/sync-server"
|
||||
},
|
||||
{
|
||||
"path": "./apps/obsidian-plugin"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user