Initial commit

This commit is contained in:
2026-04-08 11:55:27 +01:00
commit 470a1c15b8
36 changed files with 4932 additions and 0 deletions

3
.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
node_modules/
dist/
*.tsbuildinfo

44
Plan.md Normal file
View File

@@ -0,0 +1,44 @@
# Obsidian Sync Plan
## Goal
Build a self-hosted sync system for an Obsidian vault that works across Windows, Linux, and Android. The system will use an Obsidian plugin for clients, a central sync server, and end-to-end encrypted note transfer. Conflicts should try automatic merge first, then fall back to user-visible conflict handling.
## Current Implementation Slice
1. Create a TypeScript monorepo for the plugin, server, and shared libraries.
2. Define shared sync contracts for device registration, pull, push, tombstones, and conflict responses.
3. Add a shared sync engine with AES-GCM encryption, hashing, and safe text merge helpers.
4. Build a durable server that registers devices, stores encrypted note revisions on disk, and writes structured request and client-sync logs.
5. Build an Obsidian plugin with settings, manual sync, scheduled sync, remote pull, local push, attachment sync, conflict marker handling, and sync-run diagnostics upload.
6. Validate the scaffold with a workspace build and focused tests.
## Delivery Phases
### Phase 1
- Monorepo setup
- Protocol definitions
- Shared crypto and merge helpers
- Minimal server API
- Plugin settings and sync loop
### Phase 2
- Durable storage backend for the server
- Attachment sync
- Better device onboarding and recovery flow
- Structured logs and user-visible sync diagnostics
### Phase 3
- Hardening for large vaults and flaky networks
- Packaging and deployment docs
- Cross-device integration testing
## Notes
- The server now persists encrypted sync state to a local data directory and keeps JSON-line diagnostics logs.
- Text conflicts are handled on the client and binary conflicts produce a remote conflict copy.
- Recovery bundle onboarding, device revocation, key rotation, and batched sync are implemented in the current repo.
- The next highest-priority gap is broader multi-instance Obsidian integration testing and a production-grade database-backed server store.

97
README.md Normal file
View File

@@ -0,0 +1,97 @@
# Obsidian Sync
Obsidian Sync is a self-hosted, end-to-end encrypted vault sync system for Obsidian across Windows, Linux, and Android. It consists of an Obsidian plugin, a sync server, and shared protocol and crypto packages inside a TypeScript monorepo.
## Features
- End-to-end encrypted text note and binary attachment sync
- Automatic text merge with conflict markers when both sides change
- Binary conflict copies when attachments diverge
- Durable server state stored on disk
- Structured JSON-line diagnostics and client sync log upload
- Recovery bundle export and import for onboarding new devices
- Device listing and revocation
- Vault key rotation with recovery-bundle refresh
- Paged pull and batched push behavior so interrupted sync runs can resume safely
## Repository Layout
- `apps/obsidian-plugin`: Obsidian plugin entry point, settings UI, and sync orchestration
- `apps/sync-server`: Express server for encrypted revision exchange and device management
- `packages/sync-protocol`: Shared request and response schemas
- `packages/sync-engine`: Shared crypto, hashing, and merge helpers
- `docs`: Architecture and deployment notes
- `tests`: Focused contract and store-level integration tests
## Requirements
- Node.js 22 or newer
- npm 10 or newer
## Install
```bash
npm install
```
## Build
```bash
npm run build
```
## Test
```bash
npm test
```
## Run The Server
```bash
npm run dev:server
```
The default server URL is `http://localhost:8787`.
## First-Time Setup
1. Start the sync server.
2. Load the plugin into an Obsidian vault.
3. Set the server URL, vault ID, and device name in the plugin settings.
4. Run a manual sync to register the first device and generate the initial vault key.
5. Export a recovery bundle from the settings tab and store it securely.
## Add Another Device
1. Install the plugin on the second device.
2. Open the plugin settings and paste the recovery bundle into the import field.
3. Run a manual sync to register that device and pull the encrypted vault contents.
## Rotate The Vault Key
1. Open the plugin settings and use `Rotate key`.
2. Wait for the client to re-encrypt and upload the current vault contents.
3. Export the fresh recovery bundle.
4. Import that new bundle on every other device before the next sync.
## Revoke A Device
Use the plugin settings device list to revoke a device that should no longer have access. A revoked device can no longer authenticate to the server.
## Server Data
- `data/sync-state.json`: durable revision, device, tombstone, and key-status state
- `data/logs/server.jsonl`: structured server and client-sync diagnostics
## Current Limitations
- The server storage backend is file-based rather than SQLite or PostgreSQL.
- Sync validation is focused on shared libraries and the server store; there is not yet a full Obsidian plugin integration harness.
- Device onboarding is driven by a copy-and-paste recovery bundle instead of a richer pairing flow.
## Documentation
- `docs/architecture.md`
- `docs/deployment.md`
- `Plan.md`

2
Specsheet.md Normal file
View File

@@ -0,0 +1,2 @@
# Description
I need a system that syncs my obisian vault between different devices and is connected by a server. For this application, I will need a client side plugin for obisdian which works on both windows, linux, and android obsidian. I will also need a server application that will allow me to sync my notes to it and distribute it out to other devices connected. This system will require a version control system that allow the user to edit it on different devices and sort out the different edits. If there is an error create a log and alert the user.

View File

@@ -0,0 +1,9 @@
{
"id": "obsidian-sync",
"name": "Obsidian Sync",
"version": "0.1.0",
"minAppVersion": "1.5.0",
"description": "Sync a vault through a self-hosted encrypted sync server.",
"author": "Local",
"isDesktopOnly": false
}

View File

@@ -0,0 +1,18 @@
{
"name": "@obsidian-sync/obsidian-plugin",
"private": true,
"version": "0.1.0",
"main": "dist/main.js",
"types": "dist/main.d.ts",
"scripts": {
"build": "tsc -b",
"typecheck": "tsc -b --pretty false"
},
"dependencies": {
"@obsidian-sync/sync-engine": "0.1.0",
"@obsidian-sync/sync-protocol": "0.1.0"
},
"devDependencies": {
"obsidian": "^1.8.7"
}
}

View File

@@ -0,0 +1,96 @@
import { Notice, Plugin } from "obsidian";
import { SyncSettingsTab } from "./settings/SyncSettingsTab";
import { DEFAULT_SETTINGS, type ObsidianSyncSettings } from "./settings/settings";
import { SyncService } from "./sync/SyncService";
export default class ObsidianSyncPlugin extends Plugin {
settings: ObsidianSyncSettings = DEFAULT_SETTINGS;
private syncService?: SyncService;
private scheduledSyncHandle?: number;
async onload(): Promise<void> {
await this.loadSettings();
this.syncService = new SyncService(
this.app,
() => this.settings,
async (settings) => {
this.settings = settings;
await this.saveSettings();
}
);
this.addSettingTab(new SyncSettingsTab(this.app, this, () => this.configureSyncSchedule()));
this.addCommand({
id: "obsidian-sync-run-manual",
name: "Run encrypted sync now",
callback: async () => {
await this.syncService?.runManualSync();
}
});
this.addCommand({
id: "obsidian-sync-rotate-vault-key",
name: "Rotate vault key",
callback: async () => {
const result = await this.getSyncServiceOrThrow().rotateVaultKey();
new Notice(
`Vault key rotated. Uploaded ${result.uploadedFiles} files. Export the fresh recovery bundle to update other devices.`
);
}
});
this.configureSyncSchedule();
new Notice("Obsidian Sync loaded.");
}
onunload(): void {
if (this.scheduledSyncHandle !== undefined) {
window.clearInterval(this.scheduledSyncHandle);
}
}
async loadSettings(): Promise<void> {
const loaded = (await this.loadData()) as Partial<ObsidianSyncSettings> | null;
this.settings = {
...DEFAULT_SETTINGS,
...loaded,
deviceName: loaded?.deviceName || this.app.vault.getName(),
syncState: {
...DEFAULT_SETTINGS.syncState,
...(loaded?.syncState ?? {}),
files: {
...DEFAULT_SETTINGS.syncState.files,
...(loaded?.syncState?.files ?? {})
}
}
};
await this.saveSettings();
}
async saveSettings(): Promise<void> {
await this.saveData(this.settings);
}
getSyncServiceOrThrow(): SyncService {
if (!this.syncService) {
throw new Error("Sync service is not ready yet.");
}
return this.syncService;
}
private configureSyncSchedule(): void {
if (this.scheduledSyncHandle !== undefined) {
window.clearInterval(this.scheduledSyncHandle);
}
const intervalMinutes = Math.max(1, this.settings.syncIntervalMinutes);
this.scheduledSyncHandle = window.setInterval(() => {
void this.syncService?.runScheduledSync();
}, intervalMinutes * 60_000);
}
}

View File

@@ -0,0 +1,228 @@
import { Notice, PluginSettingTab, Setting, TextAreaComponent } from "obsidian";
import type ObsidianSyncPlugin from "../main";
export class SyncSettingsTab extends PluginSettingTab {
private exportedRecoveryBundle = "";
private importedRecoveryBundle = "";
constructor(
app: ObsidianSyncPlugin["app"],
private readonly plugin: ObsidianSyncPlugin,
private readonly onSettingsUpdated: () => void
) {
super(app, plugin);
}
display(): void {
const { containerEl } = this;
containerEl.empty();
containerEl.createEl("h2", { text: "Obsidian Sync" });
const syncService = this.plugin.getSyncServiceOrThrow();
new Setting(containerEl)
.setName("Server URL")
.setDesc("Base URL of the sync server.")
.addText((text) =>
text
.setPlaceholder("http://localhost:8787")
.setValue(this.plugin.settings.serverUrl)
.onChange(async (value) => {
this.plugin.settings.serverUrl = value.trim();
await this.plugin.saveSettings();
})
);
new Setting(containerEl)
.setName("Vault ID")
.setDesc("Stable identifier used to group devices for the same vault.")
.addText((text) =>
text.setValue(this.plugin.settings.vaultId).onChange(async (value) => {
this.plugin.settings.vaultId = value.trim();
await this.plugin.saveSettings();
})
);
new Setting(containerEl)
.setName("Device name")
.setDesc("Human-readable name shown on the server.")
.addText((text) =>
text.setValue(this.plugin.settings.deviceName).onChange(async (value) => {
this.plugin.settings.deviceName = value.trim();
await this.plugin.saveSettings();
})
);
new Setting(containerEl)
.setName("Sync interval")
.setDesc("Minutes between scheduled sync runs.")
.addText((text) =>
text.setValue(String(this.plugin.settings.syncIntervalMinutes)).onChange(async (value) => {
const minutes = Number(value);
this.plugin.settings.syncIntervalMinutes = Number.isFinite(minutes) && minutes > 0 ? minutes : 5;
await this.plugin.saveSettings();
this.onSettingsUpdated();
})
);
new Setting(containerEl)
.setName("Pull batch size")
.setDesc("Maximum remote changes to request in a single pull page.")
.addText((text) =>
text.setValue(String(this.plugin.settings.pullBatchSize)).onChange(async (value) => {
const batchSize = Number(value);
this.plugin.settings.pullBatchSize = Number.isFinite(batchSize) && batchSize > 0 ? Math.floor(batchSize) : 50;
await this.plugin.saveSettings();
})
);
new Setting(containerEl)
.setName("Push batch size")
.setDesc("Maximum local changes to upload in a single push request.")
.addText((text) =>
text.setValue(String(this.plugin.settings.pushBatchSize)).onChange(async (value) => {
const batchSize = Number(value);
this.plugin.settings.pushBatchSize = Number.isFinite(batchSize) && batchSize > 0 ? Math.floor(batchSize) : 50;
await this.plugin.saveSettings();
})
);
containerEl.createEl("h3", { text: "Recovery" });
let exportBundleTextArea: TextAreaComponent | undefined;
new Setting(containerEl)
.setName("Export recovery bundle")
.setDesc("Generate a bundle you can paste into another device to recover the same encrypted vault.")
.addTextArea((textArea) => {
exportBundleTextArea = textArea;
textArea.setValue(this.exportedRecoveryBundle);
textArea.inputEl.rows = 8;
textArea.inputEl.cols = 40;
})
.addButton((button) =>
button.setButtonText("Generate").onClick(async () => {
this.exportedRecoveryBundle = await syncService.exportRecoveryBundle();
exportBundleTextArea?.setValue(this.exportedRecoveryBundle);
new Notice("Recovery bundle generated.");
})
)
.addButton((button) =>
button.setButtonText("Clear").onClick(() => {
this.exportedRecoveryBundle = "";
exportBundleTextArea?.setValue("");
})
);
let importBundleTextArea: TextAreaComponent | undefined;
new Setting(containerEl)
.setName("Import recovery bundle")
.setDesc("Paste a bundle from another device to register this device against the same encrypted vault.")
.addTextArea((textArea) => {
importBundleTextArea = textArea;
textArea.setValue(this.importedRecoveryBundle);
textArea.inputEl.rows = 8;
textArea.inputEl.cols = 40;
textArea.onChange((value) => {
this.importedRecoveryBundle = value;
});
})
.addButton((button) =>
button.setButtonText("Import").onClick(async () => {
await syncService.importRecoveryBundle(this.importedRecoveryBundle);
this.importedRecoveryBundle = "";
this.exportedRecoveryBundle = "";
importBundleTextArea?.setValue("");
new Notice("Recovery bundle imported. Run sync to register this device.");
this.display();
})
);
containerEl.createEl("h3", { text: "Devices" });
const deviceListContainer = containerEl.createDiv();
new Setting(containerEl)
.setName("Connected devices")
.setDesc("Refresh the active device list and revoke devices that should no longer have access.")
.addButton((button) =>
button.setButtonText("Refresh").onClick(async () => {
await this.renderDeviceList(deviceListContainer);
})
);
void this.renderDeviceList(deviceListContainer);
containerEl.createEl("h3", { text: "Key rotation" });
new Setting(containerEl)
.setName("Rotate vault key")
.setDesc("Generate a new vault key, re-encrypt local content, and require your other devices to import a fresh recovery bundle.")
.addButton((button) =>
button.setButtonText("Rotate key").onClick(async () => {
const result = await syncService.rotateVaultKey();
this.exportedRecoveryBundle = result.recoveryBundle;
exportBundleTextArea?.setValue(result.recoveryBundle);
new Notice(
`Vault key rotated. Uploaded ${result.uploadedFiles} files. Share the fresh recovery bundle with your other devices.`
);
await this.renderDeviceList(deviceListContainer);
})
);
}
private async renderDeviceList(containerEl: HTMLElement): Promise<void> {
containerEl.empty();
if (!this.plugin.settings.deviceId || !this.plugin.settings.authToken) {
containerEl.createEl("p", {
text: "Run a sync or import a recovery bundle to register this device before managing devices."
});
return;
}
try {
const response = await this.plugin.getSyncServiceOrThrow().listDevices();
const summaryParts = [`Active server key: ${response.activeKeyId ?? "not set"}`];
if (response.keyRotatedAt) {
summaryParts.push(`Last key rotation: ${response.keyRotatedAt}`);
}
containerEl.createEl("p", {
text: summaryParts.join(" | ")
});
for (const device of response.devices) {
const isCurrentDevice = device.deviceId === this.plugin.settings.deviceId;
const description = [
`ID: ${device.deviceId}`,
`Registered: ${device.issuedAt}`,
device.revokedAt ? `Revoked: ${device.revokedAt}` : "Active"
].join(" | ");
new Setting(containerEl)
.setName(`${device.deviceName}${isCurrentDevice ? " (this device)" : ""}`)
.setDesc(description)
.addButton((button) => {
if (device.revokedAt) {
button.setButtonText("Revoked").setDisabled(true);
return;
}
button.setButtonText(isCurrentDevice ? "Revoke this device" : "Revoke").onClick(async () => {
const selfRevoked = await this.plugin.getSyncServiceOrThrow().revokeDevice(device.deviceId);
new Notice(
selfRevoked
? "This device was revoked. Import a recovery bundle or run sync again to register a new device identity."
: "Device revoked."
);
await this.renderDeviceList(containerEl);
});
});
}
} catch (error) {
containerEl.createEl("p", {
text: error instanceof Error ? `Failed to load devices: ${error.message}` : "Failed to load devices."
});
}
}
}

View File

@@ -0,0 +1,47 @@
import type { SyncFileKind } from "@obsidian-sync/sync-protocol";
export interface SyncedFileState {
kind: SyncFileKind;
revisionId: string;
contentHash: string;
lastSyncedContent?: string;
updatedAt: string;
pendingConflict?: boolean;
pendingConflictHash?: string;
}
export interface SyncState {
serverRevision: number;
files: Record<string, SyncedFileState>;
}
export interface ObsidianSyncSettings {
serverUrl: string;
vaultId: string;
deviceId: string;
deviceName: string;
authToken: string;
keyId: string;
exportedVaultKey: string;
syncIntervalMinutes: number;
pullBatchSize: number;
pushBatchSize: number;
syncState: SyncState;
}
export const DEFAULT_SETTINGS: ObsidianSyncSettings = {
serverUrl: "http://localhost:8787",
vaultId: "primary-vault",
deviceId: "",
deviceName: "",
authToken: "",
keyId: "",
exportedVaultKey: "",
syncIntervalMinutes: 5,
pullBatchSize: 50,
pushBatchSize: 50,
syncState: {
serverRevision: 0,
files: {}
}
};

View File

@@ -0,0 +1,91 @@
import type { ClientLogEntry, ClientLogUploadRequest } from "@obsidian-sync/sync-protocol";
type LogLevel = ClientLogEntry["level"];
function normalizeError(error: unknown): Record<string, unknown> {
if (error instanceof Error) {
return {
name: error.name,
message: error.message,
stack: error.stack
};
}
return {
value: String(error)
};
}
export class SyncRunLogger {
private readonly runId = globalThis.crypto.randomUUID();
private readonly entries: ClientLogEntry[] = [];
private deviceId: string;
constructor(deviceId: string) {
this.deviceId = deviceId || "unregistered-device";
}
setDeviceId(deviceId: string): void {
if (deviceId) {
this.deviceId = deviceId;
}
}
getRunId(): string {
return this.runId;
}
hasEntries(): boolean {
return this.entries.length > 0;
}
toUploadRequest(vaultId: string): ClientLogUploadRequest {
return {
vaultId,
deviceId: this.deviceId,
runId: this.runId,
entries: [...this.entries]
};
}
debug(message: string, context?: Record<string, unknown>): void {
this.log("debug", message, context);
}
info(message: string, context?: Record<string, unknown>): void {
this.log("info", message, context);
}
warn(message: string, context?: Record<string, unknown>): void {
this.log("warn", message, context);
}
error(message: string, error?: unknown, context?: Record<string, unknown>): void {
this.log("error", message, {
...context,
...(error === undefined ? {} : { error: normalizeError(error) })
});
}
private log(level: LogLevel, message: string, context?: Record<string, unknown>): void {
const entry: ClientLogEntry = {
level,
message,
timestamp: new Date().toISOString(),
deviceId: this.deviceId,
context: {
runId: this.runId,
...(context ?? {})
}
};
this.entries.push(entry);
const consoleMethod =
level === "debug" ? console.debug : level === "info" ? console.info : level === "warn" ? console.warn : console.error;
consoleMethod(`[Obsidian Sync][${this.runId}] ${message}`, entry.context ?? {});
}
}

View File

@@ -0,0 +1,984 @@
import { App, Notice, TFile, normalizePath } from "obsidian";
import {
computeBinaryHash,
computeTextHash,
createConflictCopyPath,
decryptBytes,
decryptText,
encryptBytes,
encryptText,
generateVaultKey,
importVaultKey,
mergeTextRevisions,
type VaultKeyHandle
} from "@obsidian-sync/sync-engine";
import {
ClientLogUploadResponseSchema,
ListDevicesResponseSchema,
RecoveryBundleSchema,
RegisterDeviceResponseSchema,
RevokeDeviceResponseSchema,
RotateVaultKeyResponseSchema,
SyncPullResponseSchema,
SyncPushResponseSchema,
type ListDevicesResponse,
type RecoveryBundle,
type SyncConflict,
type SyncFileKind,
type SyncFileRecord,
type SyncPullResponse,
type Tombstone
} from "@obsidian-sync/sync-protocol";
import type { ObsidianSyncSettings, SyncedFileState } from "../settings/settings";
import { SyncRunLogger } from "./SyncRunLogger";
type SettingsReader = () => ObsidianSyncSettings;
type SettingsWriter = (settings: ObsidianSyncSettings) => Promise<void>;
type SyncMode = "manual" | "scheduled";
const DEFAULT_BATCH_SIZE = 50;
const IGNORED_PATH_PREFIXES = [".obsidian/"];
const TEXT_FILE_EXTENSIONS = new Set(["md", "txt", "markdown", "canvas", "json", "yaml", "yml", "csv"]);
interface CollectLocalChangeOptions {
forceAllFiles?: boolean;
}
interface LocalChangeSet {
files: SyncFileRecord[];
tombstones: Tombstone[];
statesByPath: Record<string, SyncedFileState>;
}
interface LocalFileSnapshot {
kind: SyncFileKind;
contentHash: string;
updatedAt: string;
sizeBytes: number;
textContent?: string;
binaryContent?: ArrayBuffer;
}
interface PullSummary {
pulledChanges: number;
finalServerRevision: number;
}
interface PushSummary {
acceptedServerRevision: number;
acceptedFilePaths: string[];
acceptedTombstones: string[];
conflicts: SyncConflict[];
}
export interface RotateVaultKeyResult {
recoveryBundle: string;
activeKeyId: string;
rotatedAt: string;
uploadedFiles: number;
}
export class SyncService {
constructor(
private readonly app: App,
private readonly readSettings: SettingsReader,
private readonly writeSettings: SettingsWriter
) {}
async runManualSync(): Promise<void> {
await this.runSync("manual");
}
async runScheduledSync(): Promise<void> {
await this.runSync("scheduled");
}
async exportRecoveryBundle(): Promise<string> {
await this.ensureVaultKey();
const settings = this.readSettings();
return this.serializeRecoveryBundle(settings);
}
async importRecoveryBundle(serializedBundle: string): Promise<void> {
const parsed = RecoveryBundleSchema.parse(JSON.parse(serializedBundle) as RecoveryBundle);
const nextSettings = this.cloneSettings();
nextSettings.serverUrl = parsed.serverUrl;
nextSettings.vaultId = parsed.vaultId;
nextSettings.keyId = parsed.keyId;
nextSettings.exportedVaultKey = parsed.exportedVaultKey;
nextSettings.deviceId = "";
nextSettings.authToken = "";
nextSettings.syncState = {
serverRevision: 0,
files: {}
};
await this.writeSettings(nextSettings);
}
async listDevices(): Promise<ListDevicesResponse> {
await this.ensureRegisteredDevice();
const settings = this.readSettings();
return this.postJson(
"/api/devices/list",
{
vaultId: settings.vaultId,
deviceId: settings.deviceId
},
ListDevicesResponseSchema,
settings.authToken
);
}
async revokeDevice(targetDeviceId: string): Promise<boolean> {
await this.ensureRegisteredDevice();
const settings = this.readSettings();
await this.postJson(
"/api/devices/revoke",
{
vaultId: settings.vaultId,
deviceId: settings.deviceId,
targetDeviceId
},
RevokeDeviceResponseSchema,
settings.authToken
);
if (targetDeviceId === settings.deviceId) {
const nextSettings = this.cloneSettings();
nextSettings.deviceId = "";
nextSettings.authToken = "";
await this.writeSettings(nextSettings);
return true;
}
return false;
}
async rotateVaultKey(): Promise<RotateVaultKeyResult> {
const diagnostics = new SyncRunLogger(this.readSettings().deviceId);
try {
const currentVaultKey = await this.ensureVaultKey();
diagnostics.info("vault-key-ready", {
keyId: currentVaultKey.keyId
});
await this.ensureDeviceRegistration(diagnostics);
diagnostics.setDeviceId(this.readSettings().deviceId);
const generatedKey = await generateVaultKey();
const localChanges = await this.collectLocalChanges(
{
keyId: generatedKey.keyId,
key: generatedKey.key
},
diagnostics,
{
forceAllFiles: true
}
);
const pushSummary = await this.pushLocalChanges(localChanges, diagnostics);
const settings = this.readSettings();
const rotated = await this.postJson(
"/api/keys/rotate",
{
vaultId: settings.vaultId,
deviceId: settings.deviceId,
nextKeyId: generatedKey.keyId,
previousKeyId: settings.keyId || undefined
},
RotateVaultKeyResponseSchema,
settings.authToken
);
const nextSettings = this.cloneSettings();
nextSettings.keyId = generatedKey.keyId;
nextSettings.exportedVaultKey = generatedKey.exportedKey;
nextSettings.syncState.serverRevision = Math.max(nextSettings.syncState.serverRevision, pushSummary.acceptedServerRevision);
await this.writeSettings(nextSettings);
diagnostics.info("vault-key-rotated", {
activeKeyId: rotated.activeKeyId,
rotatedAt: rotated.rotatedAt,
uploadedFiles: pushSummary.acceptedFilePaths.length
});
return {
recoveryBundle: this.serializeRecoveryBundle(nextSettings),
activeKeyId: rotated.activeKeyId,
rotatedAt: rotated.rotatedAt,
uploadedFiles: pushSummary.acceptedFilePaths.length
};
} catch (error) {
diagnostics.error("vault-key-rotation-failed", error);
throw error;
} finally {
await this.uploadDiagnostics(diagnostics);
}
}
private async runSync(mode: SyncMode): Promise<void> {
const settings = this.readSettings();
const diagnostics = new SyncRunLogger(settings.deviceId);
if (!settings.serverUrl || !settings.vaultId) {
if (mode === "manual") {
new Notice("Set a server URL and vault ID before syncing.");
}
return;
}
diagnostics.info("sync-started", {
mode,
serverRevision: settings.syncState.serverRevision
});
try {
const vaultKey = await this.ensureVaultKey();
diagnostics.info("vault-key-ready", {
keyId: vaultKey.keyId
});
await this.ensureDeviceRegistration(diagnostics);
diagnostics.setDeviceId(this.readSettings().deviceId);
const pullSummary = await this.pullAndApplyRemoteChanges(vaultKey, diagnostics);
const localChanges = await this.collectLocalChanges(vaultKey, diagnostics);
const pushSummary = await this.pushLocalChanges(localChanges, diagnostics);
const nextSettings = this.cloneSettings();
nextSettings.syncState.serverRevision = Math.max(pullSummary.finalServerRevision, pushSummary.acceptedServerRevision);
await this.writeSettings(nextSettings);
if (mode === "manual") {
new Notice(
`Sync complete. Pulled ${pullSummary.pulledChanges} changes, uploaded ${pushSummary.acceptedFilePaths.length} files.`
);
}
if (pushSummary.conflicts.length > 0) {
diagnostics.warn("server-conflicts-reported", {
count: pushSummary.conflicts.length,
paths: pushSummary.conflicts.map((conflict) => conflict.path)
});
new Notice(`${pushSummary.conflicts.length} server conflicts need review.`);
}
} catch (error) {
diagnostics.error("sync-failed", error, {
mode
});
console.error("Obsidian Sync failed", error);
new Notice(
error instanceof Error
? `Sync failed: ${error.message}. Run ${diagnostics.getRunId()}.`
: `Sync failed. Run ${diagnostics.getRunId()}.`
);
} finally {
await this.uploadDiagnostics(diagnostics);
}
}
private cloneSettings(): ObsidianSyncSettings {
const current = this.readSettings();
return {
...current,
syncState: {
...current.syncState,
files: { ...current.syncState.files }
}
};
}
private getBaseUrl(): string {
return this.readSettings().serverUrl.replace(/\/$/, "");
}
private getPullBatchSize(): number {
return Math.max(1, this.readSettings().pullBatchSize || DEFAULT_BATCH_SIZE);
}
private getPushBatchSize(): number {
return Math.max(1, this.readSettings().pushBatchSize || DEFAULT_BATCH_SIZE);
}
private getSyncableFiles(): TFile[] {
return this.app.vault.getFiles().filter((file) => !IGNORED_PATH_PREFIXES.some((prefix) => file.path.startsWith(prefix)));
}
private resolveFileKind(file: TFile, previous?: SyncedFileState): SyncFileKind {
if (previous?.kind) {
return previous.kind;
}
return TEXT_FILE_EXTENSIONS.has(file.extension.toLowerCase()) ? "text" : "binary";
}
private createSyncedState(
kind: SyncFileKind,
revisionId: string,
contentHash: string,
updatedAt: string,
lastSyncedContent?: string
): SyncedFileState {
return {
kind,
revisionId,
contentHash,
updatedAt,
...(kind === "text" && lastSyncedContent !== undefined ? { lastSyncedContent } : {})
};
}
private buildRecoveryBundle(settings: ObsidianSyncSettings): RecoveryBundle {
return RecoveryBundleSchema.parse({
version: 1,
serverUrl: settings.serverUrl,
vaultId: settings.vaultId,
keyId: settings.keyId,
exportedVaultKey: settings.exportedVaultKey,
generatedAt: new Date().toISOString()
});
}
private serializeRecoveryBundle(settings: ObsidianSyncSettings): string {
return JSON.stringify(this.buildRecoveryBundle(settings), null, 2);
}
private async readLocalFileSnapshot(file: TFile, previous?: SyncedFileState): Promise<LocalFileSnapshot> {
const kind = this.resolveFileKind(file, previous);
if (kind === "text") {
const textContent = await this.app.vault.cachedRead(file);
return {
kind,
textContent,
contentHash: await computeTextHash(textContent),
updatedAt: new Date(file.stat.mtime).toISOString(),
sizeBytes: new TextEncoder().encode(textContent).byteLength
};
}
const binaryContent = await this.app.vault.readBinary(file);
return {
kind,
binaryContent,
contentHash: await computeBinaryHash(binaryContent),
updatedAt: new Date(file.stat.mtime).toISOString(),
sizeBytes: file.stat.size
};
}
private async ensureVaultKey(): Promise<VaultKeyHandle> {
const settings = this.readSettings();
if (settings.exportedVaultKey && settings.keyId) {
return importVaultKey(settings.exportedVaultKey, settings.keyId);
}
const nextSettings = this.cloneSettings();
const generated = await generateVaultKey();
nextSettings.keyId = generated.keyId;
nextSettings.exportedVaultKey = generated.exportedKey;
await this.writeSettings(nextSettings);
return {
keyId: generated.keyId,
key: generated.key
};
}
private async ensureRegisteredDevice(): Promise<void> {
await this.ensureDeviceRegistration(new SyncRunLogger(this.readSettings().deviceId));
}
private async ensureDeviceRegistration(diagnostics: SyncRunLogger): Promise<void> {
const settings = this.readSettings();
if (settings.deviceId && settings.authToken) {
return;
}
const deviceName = settings.deviceName || this.app.vault.getName();
const response = await this.postJson(
"/api/devices/register",
{
vaultId: settings.vaultId,
deviceName
},
RegisterDeviceResponseSchema
);
diagnostics.info("device-registered", {
deviceId: response.deviceId,
deviceName
});
const nextSettings = this.cloneSettings();
nextSettings.deviceId = response.deviceId;
nextSettings.authToken = response.token;
nextSettings.deviceName = deviceName;
await this.writeSettings(nextSettings);
}
private ensureCompatibleVaultKey(pullResponse: SyncPullResponse): void {
const settings = this.readSettings();
if (pullResponse.activeKeyId && settings.keyId && pullResponse.activeKeyId !== settings.keyId) {
throw new Error("The server is using a different vault key. Import the latest recovery bundle before syncing.");
}
}
private async pullRemoteChangesPage(sinceServerRevision: number): Promise<SyncPullResponse> {
const settings = this.readSettings();
return this.postJson(
"/api/sync/pull",
{
vaultId: settings.vaultId,
deviceId: settings.deviceId,
sinceServerRevision,
limit: this.getPullBatchSize()
},
SyncPullResponseSchema,
settings.authToken
);
}
private async pullAndApplyRemoteChanges(vaultKey: VaultKeyHandle, diagnostics: SyncRunLogger): Promise<PullSummary> {
let pulledChanges = 0;
let nextSinceServerRevision = this.readSettings().syncState.serverRevision;
while (true) {
const pullResponse = await this.pullRemoteChangesPage(nextSinceServerRevision);
this.ensureCompatibleVaultKey(pullResponse);
if (pullResponse.changes.length === 0) {
diagnostics.info("remote-pull-finished", {
changes: pulledChanges,
serverRevision: pullResponse.serverRevision,
hasMore: pullResponse.hasMore
});
break;
}
await this.applyRemoteChanges(pullResponse, vaultKey, diagnostics);
pulledChanges += pullResponse.changes.length;
nextSinceServerRevision = pullResponse.nextSinceServerRevision;
diagnostics.info("remote-pull-page-applied", {
pageChanges: pullResponse.changes.length,
nextSinceServerRevision,
hasMore: pullResponse.hasMore
});
if (!pullResponse.hasMore) {
break;
}
}
return {
pulledChanges,
finalServerRevision: this.readSettings().syncState.serverRevision
};
}
private async pushLocalChanges(localChanges: LocalChangeSet, diagnostics: SyncRunLogger): Promise<PushSummary> {
const settings = this.readSettings();
if (localChanges.files.length === 0 && localChanges.tombstones.length === 0) {
return {
acceptedServerRevision: settings.syncState.serverRevision,
acceptedFilePaths: [],
acceptedTombstones: [],
conflicts: []
};
}
const acceptedFilePaths: string[] = [];
const acceptedTombstones: string[] = [];
const conflicts: SyncConflict[] = [];
let acceptedServerRevision = settings.syncState.serverRevision;
let batchNumber = 0;
const fileBatches = this.chunkEntries(localChanges.files, this.getPushBatchSize());
const tombstoneBatches = this.chunkEntries(localChanges.tombstones, this.getPushBatchSize());
for (const fileBatch of fileBatches) {
batchNumber += 1;
const response = await this.pushBatch(fileBatch, [], acceptedServerRevision);
await this.reconcileAcceptedLocalChanges(localChanges.statesByPath, response.acceptedFilePaths, response.acceptedTombstones);
acceptedServerRevision = Math.max(acceptedServerRevision, response.acceptedServerRevision);
acceptedFilePaths.push(...response.acceptedFilePaths);
acceptedTombstones.push(...response.acceptedTombstones);
conflicts.push(...response.conflicts);
await this.persistAcceptedServerRevision(acceptedServerRevision);
diagnostics.info("push-batch-finished", {
batchNumber,
batchKind: "files",
attempted: fileBatch.length,
accepted: response.acceptedFilePaths.length,
conflicts: response.conflicts.length
});
}
for (const tombstoneBatch of tombstoneBatches) {
batchNumber += 1;
const response = await this.pushBatch([], tombstoneBatch, acceptedServerRevision);
await this.reconcileAcceptedLocalChanges(localChanges.statesByPath, response.acceptedFilePaths, response.acceptedTombstones);
acceptedServerRevision = Math.max(acceptedServerRevision, response.acceptedServerRevision);
acceptedFilePaths.push(...response.acceptedFilePaths);
acceptedTombstones.push(...response.acceptedTombstones);
conflicts.push(...response.conflicts);
await this.persistAcceptedServerRevision(acceptedServerRevision);
diagnostics.info("push-batch-finished", {
batchNumber,
batchKind: "tombstones",
attempted: tombstoneBatch.length,
accepted: response.acceptedTombstones.length,
conflicts: response.conflicts.length
});
}
diagnostics.info("remote-push-finished", {
acceptedFiles: acceptedFilePaths.length,
acceptedDeletes: acceptedTombstones.length,
conflicts: conflicts.length
});
return {
acceptedServerRevision,
acceptedFilePaths,
acceptedTombstones,
conflicts
};
}
private async pushBatch(
files: SyncFileRecord[],
tombstones: Tombstone[],
knownServerRevision: number
): Promise<ReturnType<typeof SyncPushResponseSchema.parse>> {
const settings = this.readSettings();
return this.postJson(
"/api/sync/push",
{
vaultId: settings.vaultId,
deviceId: settings.deviceId,
knownServerRevision,
files,
tombstones
},
SyncPushResponseSchema,
settings.authToken
);
}
private chunkEntries<TValue>(entries: TValue[], batchSize: number): TValue[][] {
if (entries.length === 0) {
return [];
}
const batches: TValue[][] = [];
for (let index = 0; index < entries.length; index += batchSize) {
batches.push(entries.slice(index, index + batchSize));
}
return batches;
}
private async collectLocalChanges(
vaultKey: VaultKeyHandle,
diagnostics: SyncRunLogger,
options: CollectLocalChangeOptions = {}
): Promise<LocalChangeSet> {
const settings = this.readSettings();
const files: SyncFileRecord[] = [];
const tombstones: Tombstone[] = [];
const statesByPath: Record<string, SyncedFileState> = {};
const seenPaths = new Set<string>();
for (const file of this.getSyncableFiles()) {
const previous = settings.syncState.files[file.path];
const snapshot = await this.readLocalFileSnapshot(file, previous);
seenPaths.add(file.path);
if (previous?.pendingConflict && previous.pendingConflictHash === snapshot.contentHash) {
diagnostics.debug("skipped-pending-conflict-file", {
path: file.path,
kind: snapshot.kind
});
continue;
}
if (!options.forceAllFiles && previous && previous.contentHash === snapshot.contentHash) {
continue;
}
const envelope =
snapshot.kind === "text"
? await encryptText(snapshot.textContent ?? "", vaultKey)
: await encryptBytes(snapshot.binaryContent ?? new ArrayBuffer(0), vaultKey);
const revisionId = globalThis.crypto.randomUUID();
files.push({
manifest: {
path: file.path,
kind: snapshot.kind,
contentHash: snapshot.contentHash,
revisionId,
baseRevisionId: previous?.revisionId,
updatedAt: snapshot.updatedAt,
sizeBytes: snapshot.sizeBytes,
deviceId: settings.deviceId
},
envelope
});
statesByPath[file.path] = {
kind: snapshot.kind,
revisionId,
contentHash: snapshot.contentHash,
updatedAt: snapshot.updatedAt,
...(snapshot.kind === "text" ? { lastSyncedContent: snapshot.textContent } : {})
};
}
for (const [path, state] of Object.entries(settings.syncState.files)) {
if (seenPaths.has(path)) {
continue;
}
tombstones.push({
path,
revisionId: globalThis.crypto.randomUUID(),
baseRevisionId: state.revisionId,
deletedAt: new Date().toISOString(),
deviceId: settings.deviceId
});
}
diagnostics.info("local-scan-finished", {
stagedFiles: files.length,
stagedDeletes: tombstones.length,
forceAllFiles: Boolean(options.forceAllFiles)
});
return {
files,
tombstones,
statesByPath
};
}
private async persistAcceptedServerRevision(serverRevision: number): Promise<void> {
const nextSettings = this.cloneSettings();
nextSettings.syncState.serverRevision = Math.max(nextSettings.syncState.serverRevision, serverRevision);
await this.writeSettings(nextSettings);
}
private async reconcileAcceptedLocalChanges(
statesByPath: Record<string, SyncedFileState>,
acceptedFilePaths: string[],
acceptedTombstones: string[]
): Promise<void> {
if (acceptedFilePaths.length === 0 && acceptedTombstones.length === 0) {
return;
}
const nextSettings = this.cloneSettings();
for (const path of acceptedFilePaths) {
const state = statesByPath[path];
if (state) {
nextSettings.syncState.files[path] = state;
}
}
for (const path of acceptedTombstones) {
delete nextSettings.syncState.files[path];
}
await this.writeSettings(nextSettings);
}
private async applyRemoteChanges(
pullResponse: SyncPullResponse,
vaultKey: VaultKeyHandle,
diagnostics: SyncRunLogger
): Promise<void> {
if (pullResponse.changes.length === 0) {
return;
}
const nextSettings = this.cloneSettings();
for (const change of pullResponse.changes) {
if (change.file) {
const remotePath = normalizePath(change.file.manifest.path);
const localAbstract = this.app.vault.getAbstractFileByPath(remotePath);
const previous = nextSettings.syncState.files[remotePath];
if (change.file.manifest.kind === "text") {
const remoteContent = await decryptText(change.file.envelope, vaultKey);
if (!(localAbstract instanceof TFile)) {
await this.ensureParentFolders(remotePath);
await this.app.vault.create(remotePath, remoteContent);
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"text",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt,
remoteContent
);
diagnostics.info("remote-text-created", {
path: remotePath
});
continue;
}
const localContent = await this.app.vault.cachedRead(localAbstract);
const localHash = await computeTextHash(localContent);
if (localHash === change.file.manifest.contentHash) {
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"text",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt,
remoteContent
);
continue;
}
if (!previous || localHash === previous.contentHash || localContent === previous.lastSyncedContent) {
await this.app.vault.modify(localAbstract, remoteContent);
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"text",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt,
remoteContent
);
diagnostics.info("remote-text-applied", {
path: remotePath
});
continue;
}
const merged = mergeTextRevisions({
base: previous.lastSyncedContent,
local: localContent,
remote: remoteContent
});
await this.app.vault.modify(localAbstract, merged.content);
if (merged.status === "conflict") {
const conflictCopyPath = createConflictCopyPath(remotePath);
await this.ensureParentFolders(conflictCopyPath);
await this.writeTextFile(conflictCopyPath, remoteContent);
nextSettings.syncState.files[remotePath] = {
...this.createSyncedState(
"text",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt,
remoteContent
),
pendingConflict: true,
pendingConflictHash: await computeTextHash(merged.content)
};
diagnostics.warn("remote-text-conflict", {
path: remotePath,
conflictCopyPath
});
new Notice(`Conflict markers added to ${remotePath}. Review the file before the next sync.`);
continue;
}
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"text",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt,
remoteContent
);
continue;
}
const remoteBinary = await decryptBytes(change.file.envelope, vaultKey);
if (!(localAbstract instanceof TFile)) {
await this.ensureParentFolders(remotePath);
await this.app.vault.createBinary(remotePath, remoteBinary);
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"binary",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt
);
diagnostics.info("remote-binary-created", {
path: remotePath
});
continue;
}
const localBinary = await this.app.vault.readBinary(localAbstract);
const localHash = await computeBinaryHash(localBinary);
if (localHash === change.file.manifest.contentHash) {
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"binary",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt
);
continue;
}
if (!previous || localHash === previous.contentHash) {
await this.app.vault.modifyBinary(localAbstract, remoteBinary);
nextSettings.syncState.files[remotePath] = this.createSyncedState(
"binary",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt
);
diagnostics.info("remote-binary-applied", {
path: remotePath
});
continue;
}
const conflictCopyPath = createConflictCopyPath(remotePath);
await this.ensureParentFolders(conflictCopyPath);
await this.writeBinaryFile(conflictCopyPath, remoteBinary);
nextSettings.syncState.files[remotePath] = {
...this.createSyncedState(
"binary",
change.file.manifest.revisionId,
change.file.manifest.contentHash,
change.file.manifest.updatedAt
),
pendingConflict: true,
pendingConflictHash: localHash
};
diagnostics.warn("remote-binary-conflict", {
path: remotePath,
conflictCopyPath
});
new Notice(`Attachment conflict detected for ${remotePath}. Review the remote conflict copy before the next sync.`);
}
if (change.tombstone) {
const localAbstract = this.app.vault.getAbstractFileByPath(change.tombstone.path);
const previous = nextSettings.syncState.files[change.tombstone.path];
if (!(localAbstract instanceof TFile)) {
delete nextSettings.syncState.files[change.tombstone.path];
continue;
}
const snapshot = await this.readLocalFileSnapshot(localAbstract, previous);
if (!previous || snapshot.contentHash === previous.contentHash) {
await this.app.vault.delete(localAbstract, true);
delete nextSettings.syncState.files[change.tombstone.path];
diagnostics.info("remote-delete-applied", {
path: change.tombstone.path
});
continue;
}
diagnostics.warn("remote-delete-skipped", {
path: change.tombstone.path
});
new Notice(`Skipped remote delete for ${change.tombstone.path} because local changes still exist.`);
}
}
nextSettings.syncState.serverRevision = pullResponse.nextSinceServerRevision;
await this.writeSettings(nextSettings);
}
private async ensureParentFolders(path: string): Promise<void> {
const segments = normalizePath(path).split("/");
segments.pop();
let currentPath = "";
for (const segment of segments) {
currentPath = currentPath ? `${currentPath}/${segment}` : segment;
if (!this.app.vault.getAbstractFileByPath(currentPath)) {
await this.app.vault.createFolder(currentPath);
}
}
}
private async writeTextFile(path: string, content: string): Promise<void> {
const existing = this.app.vault.getAbstractFileByPath(path);
if (existing instanceof TFile) {
await this.app.vault.modify(existing, content);
return;
}
await this.app.vault.create(path, content);
}
private async writeBinaryFile(path: string, content: ArrayBuffer): Promise<void> {
const existing = this.app.vault.getAbstractFileByPath(path);
if (existing instanceof TFile) {
await this.app.vault.modifyBinary(existing, content);
return;
}
await this.app.vault.createBinary(path, content);
}
private async uploadDiagnostics(diagnostics: SyncRunLogger): Promise<void> {
const settings = this.readSettings();
diagnostics.setDeviceId(settings.deviceId);
if (!diagnostics.hasEntries() || !settings.deviceId || !settings.authToken) {
return;
}
try {
await this.postJson(
"/api/logs",
diagnostics.toUploadRequest(settings.vaultId),
ClientLogUploadResponseSchema,
settings.authToken
);
} catch (error) {
console.warn("Failed to upload sync diagnostics", error);
}
}
private async postJson<TSchema>(
path: string,
body: unknown,
schema: { parse: (value: unknown) => TSchema },
authToken?: string
): Promise<TSchema> {
const response = await fetch(`${this.getBaseUrl()}${path}`, {
method: "POST",
headers: {
"content-type": "application/json",
...(authToken ? { authorization: `Bearer ${authToken}` } : {})
},
body: JSON.stringify(body)
});
if (!response.ok) {
throw new Error(`${response.status} ${response.statusText}`);
}
const json = (await response.json()) as unknown;
return schema.parse(json);
}
}

View File

@@ -0,0 +1,21 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"composite": true,
"declaration": true,
"outDir": "dist",
"rootDir": "src",
"module": "CommonJS"
},
"references": [
{
"path": "../../packages/sync-protocol"
},
{
"path": "../../packages/sync-engine"
}
],
"include": [
"src/**/*.ts"
]
}

View File

@@ -0,0 +1,23 @@
{
"name": "@obsidian-sync/sync-server",
"private": true,
"version": "0.1.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc -b",
"dev": "tsx watch src/index.ts",
"start": "node dist/index.js",
"typecheck": "tsc -b --pretty false"
},
"dependencies": {
"@obsidian-sync/sync-protocol": "0.1.0",
"cors": "^2.8.5",
"express": "^4.21.2"
},
"devDependencies": {
"@types/cors": "^2.8.17",
"@types/express": "^4.17.21",
"tsx": "^4.19.3"
}
}

View File

@@ -0,0 +1,38 @@
import path from "node:path";
import cors from "cors";
import express from "express";
import { createJsonFileLogger } from "./logging/jsonFileLogger";
import { createRequestLoggingMiddleware } from "./middleware/requestLoggingMiddleware";
import { createSyncRouter } from "./routes/createSyncRouter";
import { createFileSyncStore } from "./store/fileSyncStore";
const port = Number(process.env.PORT ?? 8787);
const dataDirectory = process.env.SYNC_DATA_DIR
? path.resolve(process.env.SYNC_DATA_DIR)
: path.resolve(process.cwd(), "data");
const app = express();
const logger = createJsonFileLogger(path.join(dataDirectory, "logs", "server.jsonl"));
const store = createFileSyncStore(path.join(dataDirectory, "sync-state.json"));
app.use(cors());
app.use(express.json({ limit: "10mb" }));
app.use(createRequestLoggingMiddleware(logger));
app.get("/health", (_request, response) => {
response.json({
status: "ok",
dataDirectory
});
});
app.use("/api", createSyncRouter(store, logger));
app.listen(port, () => {
logger.info("server-started", {
port,
dataDirectory
});
console.log(`Obsidian Sync server listening on http://localhost:${port}`);
});

View File

@@ -0,0 +1,45 @@
import { appendFileSync, mkdirSync } from "node:fs";
import { dirname } from "node:path";
export type LogLevel = "debug" | "info" | "warn" | "error";
export interface StructuredLogger {
log(level: LogLevel, message: string, context?: Record<string, unknown>): void;
debug(message: string, context?: Record<string, unknown>): void;
info(message: string, context?: Record<string, unknown>): void;
warn(message: string, context?: Record<string, unknown>): void;
error(message: string, context?: Record<string, unknown>): void;
}
export function createJsonFileLogger(logFilePath: string): StructuredLogger {
mkdirSync(dirname(logFilePath), { recursive: true });
const write = (level: LogLevel, message: string, context?: Record<string, unknown>) => {
const payload = {
timestamp: new Date().toISOString(),
level,
message,
context: context ?? {}
};
appendFileSync(logFilePath, `${JSON.stringify(payload)}\n`, "utf8");
};
return {
log(level, message, context) {
write(level, message, context);
},
debug(message, context) {
write("debug", message, context);
},
info(message, context) {
write("info", message, context);
},
warn(message, context) {
write("warn", message, context);
},
error(message, context) {
write("error", message, context);
}
};
}

View File

@@ -0,0 +1,26 @@
import type { Request, Response } from "express";
import type { NextFunction } from "express";
import type { StructuredLogger } from "../logging/jsonFileLogger";
export function createRequestLoggingMiddleware(logger: StructuredLogger) {
return (request: Request, response: Response, next: NextFunction) => {
const requestId = globalThis.crypto.randomUUID();
const startedAt = Date.now();
response.locals.requestId = requestId;
response.setHeader("x-request-id", requestId);
response.on("finish", () => {
logger.info("http-request", {
requestId,
method: request.method,
path: request.originalUrl,
statusCode: response.statusCode,
durationMs: Date.now() - startedAt
});
});
next();
};
}

View File

@@ -0,0 +1,206 @@
import type { Request, Response } from "express";
import express from "express";
import {
type ClientLogUploadRequest,
type ListDevicesRequest,
type RegisterDeviceRequest,
type RevokeDeviceRequest,
type RotateVaultKeyRequest,
type SyncPullRequest,
type SyncPushRequest,
ClientLogUploadRequestSchema,
ClientLogUploadResponseSchema,
ListDevicesRequestSchema,
ListDevicesResponseSchema,
RegisterDeviceRequestSchema,
RevokeDeviceRequestSchema,
RevokeDeviceResponseSchema,
RotateVaultKeyRequestSchema,
RotateVaultKeyResponseSchema,
SyncPullRequestSchema,
SyncPushRequestSchema,
parseWithSchema
} from "@obsidian-sync/sync-protocol";
import type { StructuredLogger } from "../logging/jsonFileLogger";
import type { SyncStore } from "../store/syncStore";
function readBearerToken(request: Request): string | undefined {
const authorization = request.header("authorization");
return authorization?.replace(/^Bearer\s+/i, "");
}
function getRequestId(response: Response): string {
return String(response.locals.requestId ?? "unknown-request");
}
function unauthorized(request: Request, response: Response, logger: StructuredLogger): void {
logger.warn("request-unauthorized", {
requestId: getRequestId(response),
method: request.method,
path: request.originalUrl
});
response.status(401).json({
error: "Unauthorized"
});
}
function badRequest(request: Request, response: Response, logger: StructuredLogger, error: unknown): void {
logger.warn("request-invalid", {
requestId: getRequestId(response),
method: request.method,
path: request.originalUrl,
error: error instanceof Error ? error.message : "Invalid request"
});
response.status(400).json({
error: error instanceof Error ? error.message : "Invalid request"
});
}
export function createSyncRouter(store: SyncStore, logger: StructuredLogger): express.Router {
const router = express.Router();
router.post("/devices/register", (request: Request, response: Response) => {
try {
const body: RegisterDeviceRequest = parseWithSchema(RegisterDeviceRequestSchema, request.body);
const registered = store.registerDevice(body);
logger.info("device-registered", {
requestId: getRequestId(response),
vaultId: registered.vaultId,
deviceId: registered.deviceId,
deviceName: body.deviceName
});
response.status(201).json(registered);
} catch (error) {
badRequest(request, response, logger, error);
}
});
router.post("/sync/pull", (request: Request, response: Response) => {
try {
const body: SyncPullRequest = parseWithSchema(SyncPullRequestSchema, request.body);
const token = readBearerToken(request);
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
unauthorized(request, response, logger);
return;
}
response.json(store.pull(body.vaultId, body.sinceServerRevision, body.limit));
} catch (error) {
badRequest(request, response, logger, error);
}
});
router.post("/sync/push", (request: Request, response: Response) => {
try {
const body: SyncPushRequest = parseWithSchema(SyncPushRequestSchema, request.body);
const token = readBearerToken(request);
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
unauthorized(request, response, logger);
return;
}
response.json(store.push(body));
} catch (error) {
badRequest(request, response, logger, error);
}
});
router.post("/devices/list", (request: Request, response: Response) => {
try {
const body: ListDevicesRequest = parseWithSchema(ListDevicesRequestSchema, request.body);
const token = readBearerToken(request);
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
unauthorized(request, response, logger);
return;
}
response.json(ListDevicesResponseSchema.parse(store.listDevices(body.vaultId)));
} catch (error) {
badRequest(request, response, logger, error);
}
});
router.post("/devices/revoke", (request: Request, response: Response) => {
try {
const body: RevokeDeviceRequest = parseWithSchema(RevokeDeviceRequestSchema, request.body);
const token = readBearerToken(request);
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
unauthorized(request, response, logger);
return;
}
const revoked = store.revokeDevice(body.vaultId, body.targetDeviceId);
logger.warn("device-revoked", {
requestId: getRequestId(response),
vaultId: body.vaultId,
requestedBy: body.deviceId,
targetDeviceId: body.targetDeviceId,
revokedAt: revoked.revokedAt
});
response.json(RevokeDeviceResponseSchema.parse(revoked));
} catch (error) {
badRequest(request, response, logger, error);
}
});
router.post("/keys/rotate", (request: Request, response: Response) => {
try {
const body: RotateVaultKeyRequest = parseWithSchema(RotateVaultKeyRequestSchema, request.body);
const token = readBearerToken(request);
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
unauthorized(request, response, logger);
return;
}
const rotated = store.rotateVaultKey(body.vaultId, body.nextKeyId, body.previousKeyId);
logger.info("vault-key-rotated", {
requestId: getRequestId(response),
vaultId: body.vaultId,
requestedBy: body.deviceId,
activeKeyId: rotated.activeKeyId,
rotatedAt: rotated.rotatedAt
});
response.json(RotateVaultKeyResponseSchema.parse(rotated));
} catch (error) {
badRequest(request, response, logger, error);
}
});
router.post("/logs", (request: Request, response: Response) => {
try {
const body: ClientLogUploadRequest = parseWithSchema(ClientLogUploadRequestSchema, request.body);
const token = readBearerToken(request);
if (!token || !store.authenticate(body.vaultId, body.deviceId, token)) {
unauthorized(request, response, logger);
return;
}
for (const entry of body.entries) {
logger.log(entry.level, "client-sync-log", {
requestId: getRequestId(response),
vaultId: body.vaultId,
deviceId: body.deviceId,
runId: body.runId,
clientTimestamp: entry.timestamp,
clientMessage: entry.message,
context: entry.context ?? {}
});
}
response.status(202).json(
ClientLogUploadResponseSchema.parse({
accepted: body.entries.length,
requestId: getRequestId(response)
})
);
} catch (error) {
badRequest(request, response, logger, error);
}
});
return router;
}

View File

@@ -0,0 +1,341 @@
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import type {
ListDevicesResponse,
RegisterDeviceRequest,
RegisterDeviceResponse,
RevokeDeviceResponse,
RotateVaultKeyResponse,
SyncChange,
SyncConflict,
SyncFileRecord,
SyncPullResponse,
SyncPushRequest,
SyncPushResponse,
Tombstone
} from "@obsidian-sync/sync-protocol";
import type { SyncStore } from "./syncStore";
interface VaultDevice {
deviceId: string;
deviceName: string;
token: string;
issuedAt: string;
revokedAt?: string;
}
interface VaultState {
serverRevision: number;
devices: Map<string, VaultDevice>;
files: Map<string, SyncFileRecord>;
tombstones: Map<string, Tombstone>;
changes: SyncChange[];
activeKeyId?: string;
keyRotatedAt?: string;
}
interface PersistedVaultState {
serverRevision: number;
devices: Record<string, VaultDevice>;
files: Record<string, SyncFileRecord>;
tombstones: Record<string, Tombstone>;
changes: SyncChange[];
activeKeyId?: string;
keyRotatedAt?: string;
}
interface PersistedStore {
vaults: Record<string, PersistedVaultState>;
}
function createVaultState(): VaultState {
return {
serverRevision: 0,
devices: new Map(),
files: new Map(),
tombstones: new Map(),
changes: []
};
}
function mapFromRecord<TValue>(record: Record<string, TValue> | undefined): Map<string, TValue> {
return new Map(Object.entries(record ?? {}));
}
function recordFromMap<TValue>(map: Map<string, TValue>): Record<string, TValue> {
return Object.fromEntries(map.entries());
}
function loadVaults(filePath: string): Map<string, VaultState> {
if (!existsSync(filePath)) {
return new Map();
}
const raw = readFileSync(filePath, "utf8");
if (!raw.trim()) {
return new Map();
}
const parsed = JSON.parse(raw) as Partial<PersistedStore>;
const vaults = new Map<string, VaultState>();
for (const [vaultId, vault] of Object.entries(parsed.vaults ?? {})) {
vaults.set(vaultId, {
serverRevision: vault.serverRevision ?? 0,
devices: mapFromRecord(vault.devices),
files: mapFromRecord(vault.files),
tombstones: mapFromRecord(vault.tombstones),
changes: [...(vault.changes ?? [])],
activeKeyId: vault.activeKeyId,
keyRotatedAt: vault.keyRotatedAt
});
}
return vaults;
}
function persistVaults(filePath: string, vaults: Map<string, VaultState>): void {
mkdirSync(dirname(filePath), { recursive: true });
const payload: PersistedStore = {
vaults: Object.fromEntries(
[...vaults.entries()].map(([vaultId, vault]) => [
vaultId,
{
serverRevision: vault.serverRevision,
devices: recordFromMap(vault.devices),
files: recordFromMap(vault.files),
tombstones: recordFromMap(vault.tombstones),
changes: vault.changes,
activeKeyId: vault.activeKeyId,
keyRotatedAt: vault.keyRotatedAt
}
])
)
};
const temporaryPath = `${filePath}.tmp`;
writeFileSync(temporaryPath, JSON.stringify(payload, null, 2), "utf8");
renameSync(temporaryPath, filePath);
}
function appendFileChange(vault: VaultState, file: SyncFileRecord): void {
vault.serverRevision += 1;
vault.changes.push({
serverRevision: vault.serverRevision,
file
});
}
function appendTombstoneChange(vault: VaultState, tombstone: Tombstone): void {
vault.serverRevision += 1;
vault.changes.push({
serverRevision: vault.serverRevision,
tombstone
});
}
function detectFileConflict(current: SyncFileRecord | undefined, incoming: SyncFileRecord): SyncConflict | undefined {
if (!current) {
return undefined;
}
if (current.manifest.contentHash === incoming.manifest.contentHash) {
return undefined;
}
if (!incoming.manifest.baseRevisionId || incoming.manifest.baseRevisionId !== current.manifest.revisionId) {
return {
path: incoming.manifest.path,
serverRevisionId: current.manifest.revisionId,
clientRevisionId: incoming.manifest.revisionId,
reason: "revision-mismatch"
};
}
return undefined;
}
function detectTombstoneConflict(current: SyncFileRecord | undefined, incoming: Tombstone): SyncConflict | undefined {
if (!current) {
return undefined;
}
if (!incoming.baseRevisionId || incoming.baseRevisionId !== current.manifest.revisionId) {
return {
path: incoming.path,
serverRevisionId: current.manifest.revisionId,
clientRevisionId: incoming.revisionId,
reason: "revision-mismatch"
};
}
return undefined;
}
export function createFileSyncStore(filePath: string): SyncStore {
const vaults = loadVaults(filePath);
function getVault(vaultId: string): VaultState {
let vault = vaults.get(vaultId);
if (!vault) {
vault = createVaultState();
vaults.set(vaultId, vault);
}
return vault;
}
function persist(): void {
persistVaults(filePath, vaults);
}
return {
registerDevice(request: RegisterDeviceRequest): RegisterDeviceResponse {
const vault = getVault(request.vaultId);
const deviceId = globalThis.crypto.randomUUID();
const token = globalThis.crypto.randomUUID();
const issuedAt = new Date().toISOString();
vault.devices.set(deviceId, {
deviceId,
deviceName: request.deviceName,
token,
issuedAt
});
persist();
return {
vaultId: request.vaultId,
deviceId,
token,
issuedAt
};
},
authenticate(vaultId: string, deviceId: string, token: string): boolean {
const vault = vaults.get(vaultId);
const device = vault?.devices.get(deviceId);
return Boolean(device && !device.revokedAt && device.token === token);
},
pull(vaultId: string, sinceServerRevision: number, limit?: number): SyncPullResponse {
const vault = getVault(vaultId);
const matchingChanges = vault.changes.filter((change) => change.serverRevision > sinceServerRevision);
const page = matchingChanges.slice(0, limit ?? matchingChanges.length);
const nextSinceServerRevision = page.at(-1)?.serverRevision ?? sinceServerRevision;
return {
serverRevision: vault.serverRevision,
changes: page,
hasMore: matchingChanges.length > page.length,
nextSinceServerRevision,
activeKeyId: vault.activeKeyId,
keyRotatedAt: vault.keyRotatedAt
};
},
push(request: SyncPushRequest): SyncPushResponse {
const vault = getVault(request.vaultId);
const acceptedFilePaths: string[] = [];
const acceptedTombstones: string[] = [];
const conflicts: SyncConflict[] = [];
let acceptedKeyId: string | undefined;
for (const file of request.files) {
const current = vault.files.get(file.manifest.path);
const conflict = detectFileConflict(current, file);
if (conflict) {
conflicts.push(conflict);
continue;
}
vault.files.set(file.manifest.path, file);
vault.tombstones.delete(file.manifest.path);
appendFileChange(vault, file);
acceptedFilePaths.push(file.manifest.path);
acceptedKeyId = file.envelope.keyId;
}
for (const tombstone of request.tombstones) {
const current = vault.files.get(tombstone.path);
const conflict = detectTombstoneConflict(current, tombstone);
if (conflict) {
conflicts.push(conflict);
continue;
}
vault.files.delete(tombstone.path);
vault.tombstones.set(tombstone.path, tombstone);
appendTombstoneChange(vault, tombstone);
acceptedTombstones.push(tombstone.path);
}
if (acceptedKeyId && acceptedKeyId !== vault.activeKeyId) {
vault.activeKeyId = acceptedKeyId;
vault.keyRotatedAt = new Date().toISOString();
}
if (acceptedFilePaths.length > 0 || acceptedTombstones.length > 0) {
persist();
}
return {
acceptedServerRevision: vault.serverRevision,
acceptedFilePaths,
acceptedTombstones,
conflicts,
activeKeyId: vault.activeKeyId,
keyRotatedAt: vault.keyRotatedAt
};
},
listDevices(vaultId: string): ListDevicesResponse {
const vault = getVault(vaultId);
const devices = [...vault.devices.values()].sort((left, right) => left.issuedAt.localeCompare(right.issuedAt));
return {
devices,
activeKeyId: vault.activeKeyId,
keyRotatedAt: vault.keyRotatedAt
};
},
revokeDevice(vaultId: string, targetDeviceId: string): RevokeDeviceResponse {
const vault = getVault(vaultId);
const device = vault.devices.get(targetDeviceId);
if (!device) {
throw new Error("Device not found.");
}
if (!device.revokedAt) {
device.revokedAt = new Date().toISOString();
vault.devices.set(targetDeviceId, device);
persist();
}
return {
targetDeviceId,
revokedAt: device.revokedAt
};
},
rotateVaultKey(vaultId: string, nextKeyId: string, previousKeyId?: string): RotateVaultKeyResponse {
const vault = getVault(vaultId);
if (previousKeyId && vault.activeKeyId && previousKeyId !== vault.activeKeyId) {
throw new Error("The provided previous key does not match the current server key.");
}
vault.activeKeyId = nextKeyId;
vault.keyRotatedAt = new Date().toISOString();
persist();
return {
activeKeyId: nextKeyId,
rotatedAt: vault.keyRotatedAt
};
}
};
}

View File

@@ -0,0 +1,20 @@
import type {
ListDevicesResponse,
RegisterDeviceRequest,
RegisterDeviceResponse,
RevokeDeviceResponse,
RotateVaultKeyResponse,
SyncPullResponse,
SyncPushRequest,
SyncPushResponse
} from "@obsidian-sync/sync-protocol";
export interface SyncStore {
registerDevice(request: RegisterDeviceRequest): RegisterDeviceResponse;
authenticate(vaultId: string, deviceId: string, token: string): boolean;
pull(vaultId: string, sinceServerRevision: number, limit?: number): SyncPullResponse;
push(request: SyncPushRequest): SyncPushResponse;
listDevices(vaultId: string): ListDevicesResponse;
revokeDevice(vaultId: string, targetDeviceId: string): RevokeDeviceResponse;
rotateVaultKey(vaultId: string, nextKeyId: string, previousKeyId?: string): RotateVaultKeyResponse;
}

View File

@@ -0,0 +1,18 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"composite": true,
"declaration": true,
"outDir": "dist",
"rootDir": "src",
"module": "CommonJS"
},
"references": [
{
"path": "../../packages/sync-protocol"
}
],
"include": [
"src/**/*.ts"
]
}

31
docs/architecture.md Normal file
View File

@@ -0,0 +1,31 @@
# Architecture
## Components
- `apps/obsidian-plugin`: Obsidian client plugin for local vault scanning, sync orchestration, merge handling, and user notifications.
- `apps/sync-server`: Self-hosted HTTP API for device registration and encrypted revision exchange.
- `packages/sync-protocol`: Shared schemas and request and response types.
- `packages/sync-engine`: Shared merge, hashing, and crypto helpers.
## Current Data Flow
1. The plugin registers a device with the server and stores the returned device identity and token.
2. The plugin creates or imports a local vault key.
3. The plugin pulls encrypted remote changes and applies them locally for both text notes and binary attachments.
4. Pull operations are paged so the client can resume from the last applied server revision after an interruption.
5. The plugin scans syncable vault files, hashes changed content, encrypts it, and pushes encrypted revisions in batches.
6. The server stores ciphertext, revision metadata, tombstones, device state, and active key status in a file-backed state store.
7. Each sync run emits client-side diagnostics that can be uploaded to the server and correlated with request logs through request IDs.
8. Recovery bundles package the vault ID, server URL, and current client-side vault key so a new device can join the same encrypted vault.
## Conflict Policy
- If only one side changed since the last synced base, accept the changed side.
- If both sides changed, write conflict markers into the local file, create a separate remote conflict copy, and require user review before upload continues.
## Known Gaps
- The server storage backend is durable but still file-based rather than relational.
- Key distribution between devices still depends on manually sharing a recovery bundle.
- Full plugin integration tests across multiple Obsidian instances are not implemented yet.

50
docs/deployment.md Normal file
View File

@@ -0,0 +1,50 @@
# Deployment
## Requirements
- Node.js 22 or newer
- npm 10 or newer
## Install
```bash
npm install
```
## Build
```bash
npm run build
```
## Run The Sync Server
```bash
npm run dev:server
```
The server listens on `http://localhost:8787` by default.
## Environment Variables
- `PORT`: overrides the default HTTP port.
- `SYNC_DATA_DIR`: overrides the default data directory. If omitted, the server writes data under `./data` from the workspace root.
## Data Written By The Server
- `data/sync-state.json`: durable encrypted revision metadata, devices, tombstones, and sync history.
- `data/logs/server.jsonl`: structured JSON-line server and client-sync diagnostics.
## Management Capabilities
- `POST /api/devices/list`: lists known devices for the vault plus active key status.
- `POST /api/devices/revoke`: revokes a device so it can no longer authenticate.
- `POST /api/keys/rotate`: stores the currently active vault key identifier.
- `POST /api/sync/pull`: supports paged pulls through `sinceServerRevision` and `limit`.
## Operational Notes
- The server never needs plaintext note bodies. Clients encrypt before upload and decrypt after download.
- Request logs include a response header named `x-request-id` for correlating client and server failures.
- The current storage backend is file-based. It is durable for a single-node deployment but not yet designed for clustered or high-availability hosting.
- The plugin updates local sync state after each accepted push batch and each applied pull page, which lets interrupted sync runs resume from the last durable checkpoint.

1723
package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

20
package.json Normal file
View File

@@ -0,0 +1,20 @@
{
"name": "obsidian-sync",
"private": true,
"version": "0.1.0",
"workspaces": [
"apps/*",
"packages/*"
],
"scripts": {
"build": "tsc -b tsconfig.json",
"test": "tsx --test tests/**/*.test.ts",
"typecheck": "tsc -b tsconfig.json --pretty false",
"dev:server": "npm run dev --workspace @obsidian-sync/sync-server"
},
"devDependencies": {
"@types/node": "^22.14.1",
"tsx": "^4.19.3",
"typescript": "^5.8.3"
}
}

View File

@@ -0,0 +1,14 @@
{
"name": "@obsidian-sync/sync-engine",
"private": true,
"version": "0.1.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc -b",
"typecheck": "tsc -b --pretty false"
},
"dependencies": {
"@obsidian-sync/sync-protocol": "0.1.0"
}
}

View File

@@ -0,0 +1,182 @@
import type { EncryptionEnvelope } from "@obsidian-sync/sync-protocol";
export interface VaultKeyHandle {
keyId: string;
key: CryptoKey;
}
export interface GeneratedVaultKey extends VaultKeyHandle {
exportedKey: string;
}
const BASE64_ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
function getCryptoApi(): Crypto {
if (!globalThis.crypto?.subtle) {
throw new Error("Web Crypto API is unavailable in this environment.");
}
return globalThis.crypto;
}
function bytesToBase64(bytes: Uint8Array): string {
let output = "";
for (let index = 0; index < bytes.length; index += 3) {
const first = bytes[index] ?? 0;
const second = bytes[index + 1] ?? 0;
const third = bytes[index + 2] ?? 0;
const combined = (first << 16) | (second << 8) | third;
output += BASE64_ALPHABET[(combined >> 18) & 63];
output += BASE64_ALPHABET[(combined >> 12) & 63];
output += index + 1 < bytes.length ? BASE64_ALPHABET[(combined >> 6) & 63] : "=";
output += index + 2 < bytes.length ? BASE64_ALPHABET[combined & 63] : "=";
}
return output;
}
function base64ToBytes(value: string): Uint8Array {
const normalized = value.replace(/\s+/g, "");
const padding = normalized.endsWith("==") ? 2 : normalized.endsWith("=") ? 1 : 0;
const outputLength = Math.floor((normalized.length * 3) / 4) - padding;
const bytes = new Uint8Array(outputLength);
let byteIndex = 0;
for (let index = 0; index < normalized.length; index += 4) {
const first = BASE64_ALPHABET.indexOf(normalized[index] ?? "A");
const second = BASE64_ALPHABET.indexOf(normalized[index + 1] ?? "A");
const thirdChar = normalized[index + 2] ?? "=";
const fourthChar = normalized[index + 3] ?? "=";
const third = thirdChar === "=" ? 0 : BASE64_ALPHABET.indexOf(thirdChar);
const fourth = fourthChar === "=" ? 0 : BASE64_ALPHABET.indexOf(fourthChar);
const combined = (first << 18) | (second << 12) | (third << 6) | fourth;
bytes[byteIndex] = (combined >> 16) & 255;
byteIndex += 1;
if (thirdChar !== "=" && byteIndex < outputLength + 1) {
bytes[byteIndex] = (combined >> 8) & 255;
byteIndex += 1;
}
if (fourthChar !== "=" && byteIndex < outputLength + 1) {
bytes[byteIndex] = combined & 255;
byteIndex += 1;
}
}
return bytes;
}
function toOwnedBytes(bytes: Uint8Array): Uint8Array<ArrayBuffer> {
const buffer = new ArrayBuffer(bytes.byteLength);
const view = new Uint8Array(buffer);
view.set(bytes);
return view;
}
function normalizeBytes(value: ArrayBuffer | Uint8Array): Uint8Array<ArrayBuffer> {
return value instanceof Uint8Array ? toOwnedBytes(value) : toOwnedBytes(new Uint8Array(value));
}
function toArrayBuffer(bytes: Uint8Array): ArrayBuffer {
return toOwnedBytes(bytes).buffer;
}
export async function generateVaultKey(keyId = globalThis.crypto.randomUUID()): Promise<GeneratedVaultKey> {
const cryptoApi = getCryptoApi();
const key = await cryptoApi.subtle.generateKey(
{
name: "AES-GCM",
length: 256
},
true,
["encrypt", "decrypt"]
);
const rawKey = await cryptoApi.subtle.exportKey("raw", key);
return {
keyId,
key,
exportedKey: bytesToBase64(new Uint8Array(rawKey))
};
}
export async function importVaultKey(exportedKey: string, keyId: string): Promise<VaultKeyHandle> {
const cryptoApi = getCryptoApi();
const rawKey = base64ToBytes(exportedKey);
const key = await cryptoApi.subtle.importKey(
"raw",
toArrayBuffer(rawKey),
{
name: "AES-GCM",
length: 256
},
true,
["encrypt", "decrypt"]
);
return {
keyId,
key
};
}
export async function encryptBytes(plaintext: ArrayBuffer | Uint8Array, vaultKey: VaultKeyHandle): Promise<EncryptionEnvelope> {
const cryptoApi = getCryptoApi();
const iv = cryptoApi.getRandomValues(new Uint8Array(12));
const ciphertext = await cryptoApi.subtle.encrypt(
{
name: "AES-GCM",
iv
},
vaultKey.key,
normalizeBytes(plaintext)
);
return {
algorithm: "AES-GCM-256",
keyId: vaultKey.keyId,
iv: bytesToBase64(iv),
ciphertext: bytesToBase64(new Uint8Array(ciphertext))
};
}
export async function decryptBytes(envelope: EncryptionEnvelope, vaultKey: VaultKeyHandle): Promise<ArrayBuffer> {
const cryptoApi = getCryptoApi();
const plaintext = await cryptoApi.subtle.decrypt(
{
name: "AES-GCM",
iv: toOwnedBytes(base64ToBytes(envelope.iv))
},
vaultKey.key,
toOwnedBytes(base64ToBytes(envelope.ciphertext))
);
return plaintext.slice(0);
}
export async function encryptText(plaintext: string, vaultKey: VaultKeyHandle): Promise<EncryptionEnvelope> {
return encryptBytes(new TextEncoder().encode(plaintext), vaultKey);
}
export async function decryptText(envelope: EncryptionEnvelope, vaultKey: VaultKeyHandle): Promise<string> {
return new TextDecoder().decode(await decryptBytes(envelope, vaultKey));
}
async function computeHash(bytes: Uint8Array): Promise<string> {
const cryptoApi = getCryptoApi();
const digest = await cryptoApi.subtle.digest("SHA-256", toOwnedBytes(bytes));
return bytesToBase64(new Uint8Array(digest));
}
export async function computeTextHash(value: string): Promise<string> {
return computeHash(new TextEncoder().encode(value));
}
export async function computeBinaryHash(value: ArrayBuffer | Uint8Array): Promise<string> {
return computeHash(normalizeBytes(value));
}

View File

@@ -0,0 +1,2 @@
export * from "./crypto";
export * from "./merge";

View File

@@ -0,0 +1,59 @@
export interface MergeTextInput {
base?: string;
local: string;
remote: string;
}
export interface MergeTextResult {
status: "merged" | "conflict";
content: string;
reason?: string;
}
export function mergeTextRevisions(input: MergeTextInput): MergeTextResult {
const base = input.base ?? "";
if (input.local === input.remote) {
return {
status: "merged",
content: input.local
};
}
if (base === input.local) {
return {
status: "merged",
content: input.remote
};
}
if (base === input.remote) {
return {
status: "merged",
content: input.local
};
}
return {
status: "conflict",
reason: "Both local and remote content changed since the last shared base.",
content: [
"<<<<<<< LOCAL",
input.local,
"=======",
input.remote,
">>>>>>> REMOTE"
].join("\n")
};
}
export function createConflictCopyPath(path: string): string {
const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
const extensionIndex = path.lastIndexOf(".");
if (extensionIndex <= 0) {
return `${path}.remote-conflict.${timestamp}`;
}
return `${path.slice(0, extensionIndex)}.remote-conflict.${timestamp}${path.slice(extensionIndex)}`;
}

View File

@@ -0,0 +1,18 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"composite": true,
"declaration": true,
"outDir": "dist",
"rootDir": "src",
"module": "CommonJS"
},
"references": [
{
"path": "../sync-protocol"
}
],
"include": [
"src/**/*.ts"
]
}

View File

@@ -0,0 +1,14 @@
{
"name": "@obsidian-sync/sync-protocol",
"private": true,
"version": "0.1.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc -b",
"typecheck": "tsc -b --pretty false"
},
"dependencies": {
"zod": "^3.24.2"
}
}

View File

@@ -0,0 +1,194 @@
import { z } from "zod";
export const SyncFileKindSchema = z.enum(["text", "binary"]);
export type SyncFileKind = z.infer<typeof SyncFileKindSchema>;
export const EncryptionEnvelopeSchema = z.object({
algorithm: z.literal("AES-GCM-256"),
keyId: z.string().min(1),
iv: z.string().min(1),
ciphertext: z.string().min(1)
});
export type EncryptionEnvelope = z.infer<typeof EncryptionEnvelopeSchema>;
export const FileManifestSchema = z.object({
path: z.string().min(1),
kind: SyncFileKindSchema,
contentHash: z.string().min(1),
revisionId: z.string().min(1),
baseRevisionId: z.string().min(1).optional(),
updatedAt: z.string().datetime(),
sizeBytes: z.number().int().nonnegative(),
deviceId: z.string().min(1)
});
export type FileManifest = z.infer<typeof FileManifestSchema>;
export const SyncFileRecordSchema = z.object({
manifest: FileManifestSchema,
envelope: EncryptionEnvelopeSchema
});
export type SyncFileRecord = z.infer<typeof SyncFileRecordSchema>;
export const TombstoneSchema = z.object({
path: z.string().min(1),
revisionId: z.string().min(1),
baseRevisionId: z.string().min(1).optional(),
deletedAt: z.string().datetime(),
deviceId: z.string().min(1)
});
export type Tombstone = z.infer<typeof TombstoneSchema>;
export const SyncConflictSchema = z.object({
path: z.string().min(1),
serverRevisionId: z.string().min(1),
clientRevisionId: z.string().min(1),
reason: z.enum(["revision-mismatch"])
});
export type SyncConflict = z.infer<typeof SyncConflictSchema>;
export const SyncChangeSchema = z
.object({
serverRevision: z.number().int().nonnegative(),
file: SyncFileRecordSchema.optional(),
tombstone: TombstoneSchema.optional()
})
.refine((value) => Number(Boolean(value.file)) + Number(Boolean(value.tombstone)) === 1, {
message: "A change must include either a file or a tombstone."
});
export type SyncChange = z.infer<typeof SyncChangeSchema>;
export const RegisterDeviceRequestSchema = z.object({
vaultId: z.string().min(1),
deviceName: z.string().min(1)
});
export type RegisterDeviceRequest = z.infer<typeof RegisterDeviceRequestSchema>;
export const RegisterDeviceResponseSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1),
token: z.string().min(1),
issuedAt: z.string().datetime()
});
export type RegisterDeviceResponse = z.infer<typeof RegisterDeviceResponseSchema>;
export const RecoveryBundleSchema = z.object({
version: z.literal(1),
serverUrl: z.string().min(1),
vaultId: z.string().min(1),
keyId: z.string().min(1),
exportedVaultKey: z.string().min(1),
generatedAt: z.string().datetime()
});
export type RecoveryBundle = z.infer<typeof RecoveryBundleSchema>;
export const DeviceRecordSchema = z.object({
deviceId: z.string().min(1),
deviceName: z.string().min(1),
issuedAt: z.string().datetime(),
revokedAt: z.string().datetime().optional()
});
export type DeviceRecord = z.infer<typeof DeviceRecordSchema>;
export const ListDevicesRequestSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1)
});
export type ListDevicesRequest = z.infer<typeof ListDevicesRequestSchema>;
export const ListDevicesResponseSchema = z.object({
devices: z.array(DeviceRecordSchema),
activeKeyId: z.string().min(1).optional(),
keyRotatedAt: z.string().datetime().optional()
});
export type ListDevicesResponse = z.infer<typeof ListDevicesResponseSchema>;
export const RevokeDeviceRequestSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1),
targetDeviceId: z.string().min(1)
});
export type RevokeDeviceRequest = z.infer<typeof RevokeDeviceRequestSchema>;
export const RevokeDeviceResponseSchema = z.object({
targetDeviceId: z.string().min(1),
revokedAt: z.string().datetime()
});
export type RevokeDeviceResponse = z.infer<typeof RevokeDeviceResponseSchema>;
export const RotateVaultKeyRequestSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1),
nextKeyId: z.string().min(1),
previousKeyId: z.string().min(1).optional()
});
export type RotateVaultKeyRequest = z.infer<typeof RotateVaultKeyRequestSchema>;
export const RotateVaultKeyResponseSchema = z.object({
activeKeyId: z.string().min(1),
rotatedAt: z.string().datetime()
});
export type RotateVaultKeyResponse = z.infer<typeof RotateVaultKeyResponseSchema>;
export const SyncPullRequestSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1),
sinceServerRevision: z.number().int().nonnegative(),
limit: z.number().int().positive().max(500).optional()
});
export type SyncPullRequest = z.infer<typeof SyncPullRequestSchema>;
export const SyncPullResponseSchema = z.object({
serverRevision: z.number().int().nonnegative(),
changes: z.array(SyncChangeSchema),
hasMore: z.boolean(),
nextSinceServerRevision: z.number().int().nonnegative(),
activeKeyId: z.string().min(1).optional(),
keyRotatedAt: z.string().datetime().optional()
});
export type SyncPullResponse = z.infer<typeof SyncPullResponseSchema>;
export const SyncPushRequestSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1),
knownServerRevision: z.number().int().nonnegative(),
files: z.array(SyncFileRecordSchema),
tombstones: z.array(TombstoneSchema)
});
export type SyncPushRequest = z.infer<typeof SyncPushRequestSchema>;
export const SyncPushResponseSchema = z.object({
acceptedServerRevision: z.number().int().nonnegative(),
acceptedFilePaths: z.array(z.string()),
acceptedTombstones: z.array(z.string()),
conflicts: z.array(SyncConflictSchema),
activeKeyId: z.string().min(1).optional(),
keyRotatedAt: z.string().datetime().optional()
});
export type SyncPushResponse = z.infer<typeof SyncPushResponseSchema>;
export const ClientLogEntrySchema = z.object({
level: z.enum(["debug", "info", "warn", "error"]),
message: z.string().min(1),
timestamp: z.string().datetime(),
deviceId: z.string().min(1),
context: z.record(z.string(), z.unknown()).optional()
});
export type ClientLogEntry = z.infer<typeof ClientLogEntrySchema>;
export const ClientLogUploadRequestSchema = z.object({
vaultId: z.string().min(1),
deviceId: z.string().min(1),
runId: z.string().min(1),
entries: z.array(ClientLogEntrySchema)
});
export type ClientLogUploadRequest = z.infer<typeof ClientLogUploadRequestSchema>;
export const ClientLogUploadResponseSchema = z.object({
accepted: z.number().int().nonnegative(),
requestId: z.string().min(1)
});
export type ClientLogUploadResponse = z.infer<typeof ClientLogUploadResponseSchema>;
export function parseWithSchema<T>(schema: z.ZodSchema<T>, value: unknown): T {
return schema.parse(value);
}

View File

@@ -0,0 +1,13 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"composite": true,
"declaration": true,
"outDir": "dist",
"rootDir": "src",
"module": "CommonJS"
},
"include": [
"src/**/*.ts"
]
}

View File

@@ -0,0 +1,39 @@
import assert from "node:assert/strict";
import test from "node:test";
import {
computeBinaryHash,
decryptBytes,
encryptBytes,
generateVaultKey,
mergeTextRevisions
} from "../../packages/sync-engine/src";
test("encryptBytes round-trips binary payloads", async () => {
const vaultKey = await generateVaultKey("test-key");
const original = new Uint8Array([1, 2, 3, 4, 5]).buffer;
const encrypted = await encryptBytes(original, vaultKey);
const decrypted = await decryptBytes(encrypted, vaultKey);
assert.deepEqual([...new Uint8Array(decrypted)], [1, 2, 3, 4, 5]);
});
test("computeBinaryHash is stable for identical binary payloads", async () => {
const first = await computeBinaryHash(new Uint8Array([9, 8, 7, 6]).buffer);
const second = await computeBinaryHash(new Uint8Array([9, 8, 7, 6]).buffer);
assert.equal(first, second);
});
test("mergeTextRevisions reports conflicts when both sides diverge", () => {
const result = mergeTextRevisions({
base: "alpha",
local: "alpha local",
remote: "alpha remote"
});
assert.equal(result.status, "conflict");
assert.match(result.content, /<<<<<<< LOCAL/);
assert.match(result.content, />>>>>>> REMOTE/);
});

View File

@@ -0,0 +1,174 @@
import assert from "node:assert/strict";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import { createFileSyncStore } from "../../apps/sync-server/src/store/fileSyncStore";
test("file-backed store survives restart and serves another device", () => {
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-"));
const storePath = path.join(tempDirectory, "sync-state.json");
try {
const serverA = createFileSyncStore(storePath);
const deviceA = serverA.registerDevice({
vaultId: "vault-1",
deviceName: "Desktop"
});
const deviceB = serverA.registerDevice({
vaultId: "vault-1",
deviceName: "Phone"
});
const pushResponse = serverA.push({
vaultId: "vault-1",
deviceId: deviceA.deviceId,
knownServerRevision: 0,
files: [
{
manifest: {
path: "notes/hello.md",
kind: "text",
contentHash: "hash-1",
revisionId: "rev-1",
updatedAt: "2026-04-08T00:00:00.000Z",
sizeBytes: 12,
deviceId: deviceA.deviceId
},
envelope: {
algorithm: "AES-GCM-256",
keyId: "key-1",
iv: "iv-1",
ciphertext: "cipher-1"
}
}
],
tombstones: []
});
assert.deepEqual(pushResponse.acceptedFilePaths, ["notes/hello.md"]);
const serverB = createFileSyncStore(storePath);
assert.equal(serverB.authenticate("vault-1", deviceB.deviceId, deviceB.token), true);
const pullResponse = serverB.pull("vault-1", 0);
assert.equal(pullResponse.changes.length, 1);
assert.equal(pullResponse.changes[0].file?.manifest.path, "notes/hello.md");
assert.equal(pullResponse.serverRevision, 1);
assert.equal(pullResponse.hasMore, false);
assert.equal(pullResponse.nextSinceServerRevision, 1);
} finally {
rmSync(tempDirectory, { recursive: true, force: true });
}
});
test("file-backed store paginates pull results", () => {
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-page-"));
const storePath = path.join(tempDirectory, "sync-state.json");
try {
const store = createFileSyncStore(storePath);
const device = store.registerDevice({
vaultId: "vault-2",
deviceName: "Desktop"
});
store.push({
vaultId: "vault-2",
deviceId: device.deviceId,
knownServerRevision: 0,
files: [
{
manifest: {
path: "a.md",
kind: "text",
contentHash: "hash-a",
revisionId: "rev-a",
updatedAt: "2026-04-08T00:00:00.000Z",
sizeBytes: 1,
deviceId: device.deviceId
},
envelope: {
algorithm: "AES-GCM-256",
keyId: "key-a",
iv: "iv-a",
ciphertext: "cipher-a"
}
}
],
tombstones: []
});
store.push({
vaultId: "vault-2",
deviceId: device.deviceId,
knownServerRevision: 1,
files: [
{
manifest: {
path: "b.md",
kind: "text",
contentHash: "hash-b",
revisionId: "rev-b",
updatedAt: "2026-04-08T00:01:00.000Z",
sizeBytes: 1,
deviceId: device.deviceId
},
envelope: {
algorithm: "AES-GCM-256",
keyId: "key-a",
iv: "iv-b",
ciphertext: "cipher-b"
}
}
],
tombstones: []
});
const firstPage = store.pull("vault-2", 0, 1);
assert.equal(firstPage.changes.length, 1);
assert.equal(firstPage.hasMore, true);
assert.equal(firstPage.nextSinceServerRevision, 1);
const secondPage = store.pull("vault-2", firstPage.nextSinceServerRevision, 1);
assert.equal(secondPage.changes.length, 1);
assert.equal(secondPage.hasMore, false);
assert.equal(secondPage.nextSinceServerRevision, 2);
assert.equal(secondPage.changes[0].file?.manifest.path, "b.md");
} finally {
rmSync(tempDirectory, { recursive: true, force: true });
}
});
test("device revocation and key rotation state are durable", () => {
const tempDirectory = mkdtempSync(path.join(tmpdir(), "obsidian-sync-device-"));
const storePath = path.join(tempDirectory, "sync-state.json");
try {
const store = createFileSyncStore(storePath);
const desktop = store.registerDevice({
vaultId: "vault-3",
deviceName: "Desktop"
});
const phone = store.registerDevice({
vaultId: "vault-3",
deviceName: "Phone"
});
const rotation = store.rotateVaultKey("vault-3", "key-rotated");
assert.equal(rotation.activeKeyId, "key-rotated");
const revoked = store.revokeDevice("vault-3", phone.deviceId);
assert.equal(revoked.targetDeviceId, phone.deviceId);
assert.equal(store.authenticate("vault-3", phone.deviceId, phone.token), false);
const restartedStore = createFileSyncStore(storePath);
const devices = restartedStore.listDevices("vault-3");
assert.equal(devices.activeKeyId, "key-rotated");
assert.equal(devices.devices.find((device) => device.deviceId === phone.deviceId)?.revokedAt !== undefined, true);
assert.equal(restartedStore.authenticate("vault-3", desktop.deviceId, desktop.token), true);
} finally {
rmSync(tempDirectory, { recursive: true, force: true });
}
});

25
tsconfig.base.json Normal file
View File

@@ -0,0 +1,25 @@
{
"compilerOptions": {
"target": "ES2022",
"lib": [
"ES2022",
"DOM"
],
"strict": true,
"skipLibCheck": true,
"baseUrl": ".",
"paths": {
"@obsidian-sync/sync-protocol": [
"packages/sync-protocol/src"
],
"@obsidian-sync/sync-engine": [
"packages/sync-engine/src"
]
},
"moduleResolution": "Node",
"esModuleInterop": true,
"resolveJsonModule": true,
"forceConsistentCasingInFileNames": true,
"noEmitOnError": true
}
}

17
tsconfig.json Normal file
View File

@@ -0,0 +1,17 @@
{
"files": [],
"references": [
{
"path": "./packages/sync-protocol"
},
{
"path": "./packages/sync-engine"
},
{
"path": "./apps/sync-server"
},
{
"path": "./apps/obsidian-plugin"
}
]
}