Read Apple Notes directly from its SQLite store
Splits the original single-file exporter into a package and adds a query layer, so notes can be searched and read live instead of only exported. Reading the Core Data store rather than driving Notes.app through AppleScript is what makes full-text search practical. Measured on an 818-note store, osascript needs ~0.325s per note body (`first note whose id is` is an unindexed linear scan), so searching everything costs minutes; decoding every gzipped protobuf body here takes 0.11s total. No cache, so results are always current. - store.py read-only SQLite access, folder tree, attachments - parser.py gzip + protobuf decoding and Markdown rendering (moved verbatim) - search.py search, recency listing, single-note reads - export.py Markdown output and the incremental manifest Three correctness details found while building this: - Open with mode=ro, never immutable=1. The immutable flag makes SQLite skip the -wal file and silently serve a stale snapshot; the live store routinely carries megabytes of un-checkpointed WAL. - Recently Deleted is ZFOLDERTYPE = 1, and notes in it keep ZMARKEDFORDELETION = 0, so the existing filter still surfaced them. Matching the folder type is also language-independent. Now excluded by default. - Creation timestamps live in whichever of ZCREATIONDATE..3 is populated, so every candidate column is read. Export no longer writes a CLAUDE.md into the output directory unless asked with --write-claude-md. tests/test_parity.py keeps the original suite byte-for-byte as a regression check on the split; test_search.py covers the new query layer against a synthetic store. 48 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
commit
a8615b3c45
16 files changed
+3421
No files matched your search
@@ -0,0 +1,19 @@
|
||||
name: tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: macos-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.10", "3.13"]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
- run: python -m unittest discover -s tests -v
|
||||
@@ -0,0 +1,8 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.venv/
|
||||
dist/
|
||||
build/
|
||||
*.egg-info/
|
||||
.DS_Store
|
||||
uv.lock
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 ingjieye
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,173 @@
|
||||
# apple-notes-cli
|
||||
|
||||
Search, read, and export Apple Notes from the terminal by reading the local
|
||||
Notes database directly — no AppleScript, no UI automation, no cloud API.
|
||||
|
||||
```console
|
||||
$ apple-notes search "kubernetes" -n 3
|
||||
4821 2026-09-08 14:02 Work/Infra/Cluster upgrade notes (7x)
|
||||
…drain the node before the kubernetes control plane restarts…
|
||||
3907 2026-08-30 09:15 Work/Runbooks/On-call (2x)
|
||||
…kubernetes events are the fastest way to see why a pod is pending…
|
||||
|
||||
$ apple-notes recent --since 2d
|
||||
4830 2026-09-09 15:59 Work/Meeting notes
|
||||
4829 2026-09-09 13:54 Personal/Reading list
|
||||
```
|
||||
|
||||
## Why not AppleScript
|
||||
|
||||
The usual approach is `osascript` against Notes.app. It works, but the cost
|
||||
lands squarely on search. Measured on a real 818-note store:
|
||||
|
||||
| Operation | AppleScript | apple-notes-cli |
|
||||
| --- | --- | --- |
|
||||
| Enumerate folders and count notes | 1.85 s | — |
|
||||
| Fetch one note body | 0.325 s | — |
|
||||
| **Full-text search every note** | **~266 s** (extrapolated) | **0.11 s** |
|
||||
|
||||
`first note whose id is "…"` is a linear scan inside Notes.app with no index
|
||||
behind it, so reading every body costs a third of a second each. That is why
|
||||
AppleScript-based tools either search titles only or cache aggressively and
|
||||
serve you stale results. Reading the SQLite store sidesteps the whole problem:
|
||||
one query, then in-memory decoding of every body.
|
||||
|
||||
## Install
|
||||
|
||||
Requires macOS and Python 3.10+.
|
||||
|
||||
```bash
|
||||
uv tool install git+https://github.com/ingjieye/apple-notes-cli
|
||||
# or
|
||||
pipx install git+https://github.com/ingjieye/apple-notes-cli
|
||||
```
|
||||
|
||||
Run it straight from a clone without installing:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ingjieye/apple-notes-cli && cd apple-notes-cli
|
||||
uv run python -m apple_notes_cli recent
|
||||
```
|
||||
|
||||
macOS may ask you to grant your terminal access to the Notes container the
|
||||
first time. Nothing here needs Full Disk Access on its own.
|
||||
|
||||
## Commands
|
||||
|
||||
```
|
||||
apple-notes search <query> full-text search across every note body
|
||||
apple-notes recent list recently modified notes
|
||||
apple-notes show <id> print one note as Markdown
|
||||
apple-notes folders list folders with note counts
|
||||
apple-notes export <dir> export everything to Markdown files
|
||||
```
|
||||
|
||||
### search
|
||||
|
||||
```bash
|
||||
apple-notes search "OpenTelemetry"
|
||||
apple-notes search "TODO|FIXME" --regex
|
||||
apple-notes search "invoice" --folder Finance --since 1w
|
||||
apple-notes search "postgres" --json | jq -r '.[].title'
|
||||
apple-notes search "draft" --body-only --no-snippets
|
||||
```
|
||||
|
||||
Results rank by match count, then by recency. Titles are searched alongside
|
||||
bodies unless you pass `--body-only`. `--json` emits structured records, which
|
||||
is the useful shape for piping into an LLM or another tool.
|
||||
|
||||
### recent
|
||||
|
||||
```bash
|
||||
apple-notes recent -n 50
|
||||
apple-notes recent --since 6h --folder Work
|
||||
```
|
||||
|
||||
Metadata only — bodies are never inflated, so this stays fast regardless of
|
||||
store size.
|
||||
|
||||
### export
|
||||
|
||||
```bash
|
||||
apple-notes export ~/Downloads/notes
|
||||
apple-notes export ~/Downloads/notes --incremental
|
||||
```
|
||||
|
||||
One Markdown file per note, mirroring your folder tree, with YAML frontmatter
|
||||
(title, ids, dates, folder, links, attachments, and a `notes://` link back to
|
||||
the note). Attachments are copied next to the Markdown under
|
||||
`attachments/<id>/`. `--incremental` keeps a manifest so unchanged notes are
|
||||
skipped and exports of deleted notes are pruned.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
NoteStore.sqlite
|
||||
└── ZICNOTEDATA.ZDATA gzip → protobuf → text runs → Markdown
|
||||
└── ZICCLOUDSYNCINGOBJECT titles, dates, folder tree, attachments
|
||||
```
|
||||
|
||||
| Module | Role |
|
||||
| --- | --- |
|
||||
| `store.py` | read-only SQLite access, folder tree, attachments |
|
||||
| `parser.py` | gzip + protobuf decoding, Markdown rendering |
|
||||
| `search.py` | search, recency listing, single-note reads |
|
||||
| `export.py` | Markdown output and the incremental manifest |
|
||||
|
||||
Three details are worth knowing if you build something similar:
|
||||
|
||||
**Open the database with `mode=ro`, never `immutable=1`.** The `immutable`
|
||||
flag makes SQLite skip the `-wal` file, which silently serves a stale snapshot
|
||||
missing your most recent notes. On a live store the `-wal` file routinely holds
|
||||
megabytes of un-checkpointed data. WAL mode allows concurrent readers, so
|
||||
`mode=ro` is safe while Notes.app is running and never blocks it. Export uses
|
||||
SQLite's `backup()` API instead, for a stable point-in-time copy.
|
||||
|
||||
**There is no FTS index.** The schema has no full-text tables, and bodies are
|
||||
gzipped protobuf blobs, so `WHERE body LIKE …` is not available. Search means
|
||||
inflating every candidate body and matching in memory. At ~800 notes that is
|
||||
about 30 ms, so this tool does no caching and always reflects the live store.
|
||||
|
||||
**Recently Deleted is `ZFOLDERTYPE = 1`.** Notes in the trash keep
|
||||
`ZMARKEDFORDELETION = 0`, so filtering on that column alone still surfaces
|
||||
them. Matching on the folder type is also language-independent, unlike
|
||||
comparing against a localized "Recently Deleted" string. They are excluded by
|
||||
default; pass `--include-trashed` to include them.
|
||||
|
||||
## Exit codes
|
||||
|
||||
`search` follows the `grep` convention: `0` when something matched, `1` when
|
||||
nothing did, `2` on a bad argument or an unreadable database. That makes it
|
||||
usable in a conditional:
|
||||
|
||||
```bash
|
||||
if apple-notes search "release checklist" --no-snippets >/dev/null; then
|
||||
echo "found it"
|
||||
fi
|
||||
```
|
||||
|
||||
## Reading is all it does
|
||||
|
||||
This tool only reads. It never writes to `NoteStore.sqlite` — modifying the
|
||||
store directly would corrupt CloudKit's sync state and risk losing notes. If
|
||||
you need to create or edit notes, drive Notes.app through AppleScript for the
|
||||
write path and use this for reading.
|
||||
|
||||
Password-protected notes are listed but their bodies stay encrypted and
|
||||
unreadable; they are skipped unless you pass `--include-password-protected`.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
uv run python -m unittest discover -s tests
|
||||
```
|
||||
|
||||
`tests/test_parity.py` is the original single-file exporter's suite, kept
|
||||
unchanged apart from neutralized example URLs, so it guards the protobuf
|
||||
decoding against regressions.
|
||||
`tests/test_search.py` builds a synthetic Core Data store to cover the query
|
||||
layer, trash handling, and time filters.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,33 @@
|
||||
[project]
|
||||
name = "apple-notes-cli"
|
||||
version = "0.1.0"
|
||||
description = "Search, read, and export Apple Notes by reading the local database directly — no AppleScript, no UI automation."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
license = { text = "MIT" }
|
||||
authors = [{ name = "ingjieye" }]
|
||||
keywords = ["apple-notes", "macos", "notes", "cli", "search", "markdown", "export"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Environment :: MacOS X",
|
||||
"Intended Audience :: End Users/Desktop",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: MacOS :: MacOS X",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Topic :: Utilities",
|
||||
]
|
||||
dependencies = []
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/ingjieye/apple-notes-cli"
|
||||
Issues = "https://github.com/ingjieye/apple-notes-cli/issues"
|
||||
|
||||
[project.scripts]
|
||||
apple-notes = "apple_notes_cli.cli:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/apple_notes_cli"]
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Read Apple Notes directly from its local Core Data store."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,4 @@
|
||||
from .cli import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,278 @@
|
||||
"""Command line entry point."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from . import search as search_mod
|
||||
from .store import DEFAULT_NOTES_DIR, connect, default_database
|
||||
|
||||
EPILOG = """\
|
||||
examples:
|
||||
apple-notes recent --since 2d
|
||||
apple-notes search "OpenTelemetry" --json
|
||||
apple-notes search "TODO|FIXME" --regex --folder Work
|
||||
apple-notes show 8554
|
||||
apple-notes export ~/Downloads/notes --incremental
|
||||
"""
|
||||
|
||||
|
||||
def add_common(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument(
|
||||
"--notes-dir",
|
||||
type=Path,
|
||||
default=DEFAULT_NOTES_DIR,
|
||||
help=f"Apple Notes group container (default: {DEFAULT_NOTES_DIR})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--database",
|
||||
type=Path,
|
||||
help="explicit NoteStore.sqlite path, overriding --notes-dir",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--include-password-protected",
|
||||
action="store_true",
|
||||
help="include locked notes (their bodies stay encrypted and unreadable)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--include-trashed",
|
||||
action="store_true",
|
||||
help="include notes sitting in Recently Deleted",
|
||||
)
|
||||
|
||||
|
||||
def add_filters(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument(
|
||||
"--since",
|
||||
help="only notes modified since this time (30m, 6h, 2d, 1w, or 2026-09-01)",
|
||||
)
|
||||
parser.add_argument("--folder", help="restrict to folder paths containing this text")
|
||||
parser.add_argument("--json", action="store_true", help="emit JSON instead of text")
|
||||
|
||||
|
||||
def local_time(value: str | None) -> str:
|
||||
"""Render a stored UTC timestamp in the viewer's own timezone.
|
||||
|
||||
Timestamps are kept as UTC ISO-8601 for machine consumers (``--json``,
|
||||
export frontmatter); only human-facing output is localized.
|
||||
"""
|
||||
if not value:
|
||||
return "?"
|
||||
try:
|
||||
return dt.datetime.fromisoformat(value).astimezone().strftime("%Y-%m-%d %H:%M")
|
||||
except ValueError:
|
||||
return value
|
||||
|
||||
|
||||
def open_store(args: argparse.Namespace):
|
||||
database = args.database or default_database(args.notes_dir)
|
||||
return connect(database)
|
||||
|
||||
|
||||
def resolve_since(args: argparse.Namespace):
|
||||
if not getattr(args, "since", None):
|
||||
return None
|
||||
return search_mod.parse_since(args.since)
|
||||
|
||||
|
||||
def cmd_recent(args: argparse.Namespace) -> int:
|
||||
with open_store(args) as conn:
|
||||
notes = search_mod.recent(
|
||||
conn,
|
||||
limit=args.limit,
|
||||
since=resolve_since(args),
|
||||
folder=args.folder,
|
||||
include_password_protected=args.include_password_protected,
|
||||
include_trashed=args.include_trashed,
|
||||
)
|
||||
if args.json:
|
||||
print(json.dumps([n.__dict__ for n in notes], ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if not notes:
|
||||
print("no notes matched")
|
||||
return 0
|
||||
for note in notes:
|
||||
print(f"{note.pk:>7} {local_time(note.modified):<16} {note.folder}/{note.title}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_search(args: argparse.Namespace) -> int:
|
||||
with open_store(args) as conn:
|
||||
hits = search_mod.search(
|
||||
conn,
|
||||
args.query,
|
||||
regex=args.regex,
|
||||
case_sensitive=args.case_sensitive,
|
||||
limit=args.limit,
|
||||
since=resolve_since(args),
|
||||
folder=args.folder,
|
||||
include_password_protected=args.include_password_protected,
|
||||
include_trashed=args.include_trashed,
|
||||
snippet_width=args.snippet_width,
|
||||
max_snippets=0 if args.no_snippets else args.max_snippets,
|
||||
search_titles=not args.body_only,
|
||||
)
|
||||
if args.json:
|
||||
payload = [
|
||||
{
|
||||
**hit.note.__dict__,
|
||||
"match_count": hit.match_count,
|
||||
"snippets": hit.snippets,
|
||||
}
|
||||
for hit in hits
|
||||
]
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if not hits:
|
||||
print(f"no notes matched {args.query!r}")
|
||||
return 1
|
||||
for hit in hits:
|
||||
note = hit.note
|
||||
print(
|
||||
f"{note.pk:>7} {local_time(note.modified):<16} "
|
||||
f"{note.folder}/{note.title} ({hit.match_count}x)"
|
||||
)
|
||||
for snippet in hit.snippets:
|
||||
print(f" {snippet}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_show(args: argparse.Namespace) -> int:
|
||||
with open_store(args) as conn:
|
||||
try:
|
||||
note, body = search_mod.show(conn, args.id, markdown=not args.plain)
|
||||
except KeyError as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
if args.json:
|
||||
print(json.dumps({**note.__dict__, "body": body}, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
print(f"# {note.title}\n")
|
||||
print(f"- id: {note.pk}")
|
||||
print(f"- folder: {note.folder}")
|
||||
print(f"- created: {local_time(note.created)}")
|
||||
print(f"- modified: {local_time(note.modified)}\n")
|
||||
print(body)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_folders(args: argparse.Namespace) -> int:
|
||||
with open_store(args) as conn:
|
||||
rows = search_mod.folders(conn, include_trashed=args.include_trashed)
|
||||
if args.json:
|
||||
print(json.dumps([{"folder": f, "notes": c} for f, c in rows], ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
for path, count in rows:
|
||||
print(f"{count:>6} {path}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_export(args: argparse.Namespace) -> int:
|
||||
from .export import export_notes
|
||||
|
||||
return export_notes(args)
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="apple-notes",
|
||||
description="Search, read, and export Apple Notes by reading the local database directly.",
|
||||
epilog=EPILOG,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_recent = sub.add_parser("recent", help="list recently modified notes")
|
||||
p_recent.add_argument("-n", "--limit", type=int, default=20, help="max notes (default: 20)")
|
||||
add_filters(p_recent)
|
||||
add_common(p_recent)
|
||||
p_recent.set_defaults(func=cmd_recent)
|
||||
|
||||
p_search = sub.add_parser("search", help="full-text search across note bodies")
|
||||
p_search.add_argument("query", help="text to find, or a regex with --regex")
|
||||
p_search.add_argument("-n", "--limit", type=int, help="max results (default: all)")
|
||||
p_search.add_argument("--regex", action="store_true", help="treat query as a regex")
|
||||
p_search.add_argument(
|
||||
"--case-sensitive", action="store_true", help="match case exactly"
|
||||
)
|
||||
p_search.add_argument(
|
||||
"--body-only", action="store_true", help="do not match against titles"
|
||||
)
|
||||
p_search.add_argument(
|
||||
"--max-snippets", type=int, default=3, help="context snippets per note (default: 3)"
|
||||
)
|
||||
p_search.add_argument(
|
||||
"--snippet-width", type=int, default=60, help="context chars each side (default: 60)"
|
||||
)
|
||||
p_search.add_argument("--no-snippets", action="store_true", help="omit context snippets")
|
||||
add_filters(p_search)
|
||||
add_common(p_search)
|
||||
p_search.set_defaults(func=cmd_search)
|
||||
|
||||
p_show = sub.add_parser("show", help="print one note by id")
|
||||
p_show.add_argument("id", type=int, help="note id, as shown by search/recent")
|
||||
p_show.add_argument(
|
||||
"--plain", action="store_true", help="raw extracted text instead of Markdown"
|
||||
)
|
||||
p_show.add_argument("--json", action="store_true", help="emit JSON instead of text")
|
||||
add_common(p_show)
|
||||
p_show.set_defaults(func=cmd_show)
|
||||
|
||||
p_folders = sub.add_parser("folders", help="list folders with note counts")
|
||||
p_folders.add_argument("--json", action="store_true", help="emit JSON instead of text")
|
||||
add_common(p_folders)
|
||||
p_folders.set_defaults(func=cmd_folders)
|
||||
|
||||
p_export = sub.add_parser("export", help="export every note to Markdown files")
|
||||
p_export.add_argument(
|
||||
"output", nargs="?", type=Path, default=Path.cwd() / "exported-notes",
|
||||
help="output directory (default: ./exported-notes)",
|
||||
)
|
||||
p_export.add_argument("--limit", type=int, help="export at most this many notes")
|
||||
p_export.add_argument("--dry-run", action="store_true", help="report without writing")
|
||||
p_export.add_argument(
|
||||
"--incremental", action="store_true",
|
||||
help="skip unchanged notes and prune deleted ones via the manifest",
|
||||
)
|
||||
p_export.add_argument(
|
||||
"--keep-stale", action="store_true",
|
||||
help="with --incremental, do not delete exports of removed notes",
|
||||
)
|
||||
p_export.add_argument(
|
||||
"--no-attachments", action="store_true", help="do not copy attachment files"
|
||||
)
|
||||
p_export.add_argument(
|
||||
"--write-claude-md", action="store_true",
|
||||
help="also drop a CLAUDE.md into the output directory describing the layout",
|
||||
)
|
||||
p_export.add_argument(
|
||||
"--no-snapshot", action="store_true",
|
||||
help="read the live database instead of copying it to a temporary snapshot first",
|
||||
)
|
||||
add_common(p_export)
|
||||
p_export.set_defaults(func=cmd_export)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
return args.func(args)
|
||||
except FileNotFoundError as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 2
|
||||
except ValueError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
except BrokenPipeError:
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,528 @@
|
||||
"""Write notes to disk as Markdown, with incremental-manifest bookkeeping."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .models import ExportResult, Note
|
||||
from .store import copy_live_database, iter_notes, parse_iso_timestamp
|
||||
from .utils import sanitize_filename
|
||||
|
||||
MANIFEST_FILENAME = ".apple_notes_export_manifest.json"
|
||||
MANIFEST_VERSION = 1
|
||||
IGNORED_PRUNE_FILENAMES = {".DS_Store"}
|
||||
|
||||
|
||||
def set_macos_creation_time(path: Path, created: dt.datetime) -> None:
|
||||
setfile = shutil.which("SetFile")
|
||||
if not setfile:
|
||||
return
|
||||
local = created.astimezone()
|
||||
formatted = local.strftime("%m/%d/%Y %H:%M:%S")
|
||||
with contextlib.suppress(OSError, subprocess.SubprocessError):
|
||||
subprocess.run(
|
||||
[setfile, "-d", formatted, str(path)],
|
||||
check=False,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
|
||||
|
||||
def apply_file_timestamps(path: Path, *, created: str | None, modified: str | None) -> None:
|
||||
created_dt = parse_iso_timestamp(created)
|
||||
modified_dt = parse_iso_timestamp(modified) or created_dt
|
||||
if modified_dt is not None:
|
||||
timestamp = modified_dt.timestamp()
|
||||
with contextlib.suppress(OSError):
|
||||
os.utime(path, (timestamp, timestamp))
|
||||
if created_dt is not None and sys.platform == "darwin":
|
||||
set_macos_creation_time(path, created_dt)
|
||||
|
||||
|
||||
|
||||
|
||||
def yaml_scalar(value: str | int | None) -> str:
|
||||
if value is None:
|
||||
return '""'
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
|
||||
|
||||
|
||||
|
||||
def unique_path(path: Path) -> Path:
|
||||
if not path.exists():
|
||||
return path
|
||||
stem = path.stem
|
||||
suffix = path.suffix
|
||||
parent = path.parent
|
||||
counter = 2
|
||||
while True:
|
||||
candidate = parent / f"{stem}-{counter}{suffix}"
|
||||
if not candidate.exists():
|
||||
return candidate
|
||||
counter += 1
|
||||
|
||||
|
||||
def note_filename_stem(title: str, pk: int) -> str:
|
||||
return f"{sanitize_filename(title)}-{pk}"
|
||||
|
||||
|
||||
|
||||
|
||||
def build_markdown(note: Note) -> str:
|
||||
lines = [
|
||||
"---",
|
||||
f"title: {yaml_scalar(note.title)}",
|
||||
f"id: {note.pk}",
|
||||
f"apple_notes_id: {yaml_scalar(note.apple_notes_id)}",
|
||||
f"created: {yaml_scalar(note.created)}",
|
||||
f"modified: {yaml_scalar(note.modified)}",
|
||||
f"folder: {yaml_scalar(note.folder_path)}",
|
||||
]
|
||||
|
||||
if note.attachments:
|
||||
lines.append("attachments:")
|
||||
for attachment in note.attachments:
|
||||
lines.append(f" - filename: {yaml_scalar(attachment.output_name)}")
|
||||
lines.append(f" identifier: {yaml_scalar(attachment.identifier)}")
|
||||
lines.append(f" type: {yaml_scalar(attachment.type_uti)}")
|
||||
else:
|
||||
lines.append("attachments: []")
|
||||
|
||||
if note.links:
|
||||
lines.append("links:")
|
||||
for url in note.links:
|
||||
lines.append(f" - url: {yaml_scalar(url)}")
|
||||
lines.append(f" title: {yaml_scalar(url)}")
|
||||
else:
|
||||
lines.append("links: []")
|
||||
|
||||
lines.append(f"source: notes://showNote?identifier={note.apple_notes_id}")
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
lines.append(note.body.rstrip())
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def file_fingerprint(path: Path | None) -> dict[str, int | str] | None:
|
||||
if path is None or not path.exists():
|
||||
return None
|
||||
stat = path.stat()
|
||||
return {
|
||||
"path": str(path),
|
||||
"size": stat.st_size,
|
||||
"mtime_ns": stat.st_mtime_ns,
|
||||
}
|
||||
|
||||
|
||||
def note_signature(note: Note) -> str:
|
||||
payload = {
|
||||
"markdown": build_markdown(note),
|
||||
"attachments": [
|
||||
{
|
||||
"identifier": attachment.identifier,
|
||||
"type": attachment.type_uti,
|
||||
"output_name": attachment.output_name,
|
||||
"source": file_fingerprint(attachment.source_path),
|
||||
}
|
||||
for attachment in note.attachments
|
||||
],
|
||||
}
|
||||
encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
return hashlib.sha256(encoded).hexdigest()
|
||||
|
||||
|
||||
def path_relative_to_output(output_root: Path, path: Path) -> str:
|
||||
return str(path.resolve().relative_to(output_root.resolve()))
|
||||
|
||||
|
||||
def manifest_file(output_root: Path) -> Path:
|
||||
return output_root / MANIFEST_FILENAME
|
||||
|
||||
|
||||
def load_manifest(output_root: Path) -> dict:
|
||||
path = manifest_file(output_root)
|
||||
if not path.exists():
|
||||
return {"version": MANIFEST_VERSION, "notes": {}}
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError:
|
||||
return {"version": MANIFEST_VERSION, "notes": {}}
|
||||
if not isinstance(data, dict) or not isinstance(data.get("notes"), dict):
|
||||
return {"version": MANIFEST_VERSION, "notes": {}}
|
||||
return data
|
||||
|
||||
|
||||
def manifest_entry(output_root: Path, note: Note, result: ExportResult, signature: str) -> dict:
|
||||
return {
|
||||
"id": note.pk,
|
||||
"apple_notes_id": note.apple_notes_id,
|
||||
"title": note.title,
|
||||
"folder": note.folder_path,
|
||||
"modified": note.modified,
|
||||
"signature": signature,
|
||||
"path": path_relative_to_output(output_root, result.note_path),
|
||||
"attachments": [path_relative_to_output(output_root, path) for path in result.attachment_paths],
|
||||
}
|
||||
|
||||
|
||||
def safe_output_path(output_root: Path, relative_path: str) -> Path | None:
|
||||
if not relative_path or os.path.isabs(relative_path):
|
||||
return None
|
||||
root = output_root.resolve()
|
||||
path = (root / relative_path).resolve()
|
||||
try:
|
||||
path.relative_to(root)
|
||||
except ValueError:
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
def manifest_paths_exist(output_root: Path, entry: dict) -> bool:
|
||||
paths = [entry.get("path"), *entry.get("attachments", [])]
|
||||
for relative_path in paths:
|
||||
if not isinstance(relative_path, str):
|
||||
return False
|
||||
path = safe_output_path(output_root, relative_path)
|
||||
if path is None or not path.exists():
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def path_has_mtime(path: Path, modified: str | None) -> bool:
|
||||
modified_dt = parse_iso_timestamp(modified)
|
||||
if modified_dt is None:
|
||||
return True
|
||||
with contextlib.suppress(OSError):
|
||||
return abs(path.stat().st_mtime - modified_dt.timestamp()) < 1.0
|
||||
return False
|
||||
|
||||
|
||||
def manifest_entry_timestamps_match(output_root: Path, entry: dict) -> bool:
|
||||
modified = entry.get("modified")
|
||||
if not isinstance(modified, str):
|
||||
return True
|
||||
paths = [entry.get("path"), *entry.get("attachments", [])]
|
||||
for relative_path in paths:
|
||||
if not isinstance(relative_path, str):
|
||||
return False
|
||||
path = safe_output_path(output_root, relative_path)
|
||||
if path is None or not path.exists() or not path_has_mtime(path, modified):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def manifest_referenced_paths(notes: dict[str, dict]) -> set[str]:
|
||||
paths: set[str] = set()
|
||||
for entry in notes.values():
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
note_path = entry.get("path")
|
||||
if isinstance(note_path, str):
|
||||
paths.add(note_path)
|
||||
for attachment_path in entry.get("attachments", []):
|
||||
if isinstance(attachment_path, str):
|
||||
paths.add(attachment_path)
|
||||
return paths
|
||||
|
||||
|
||||
def prune_empty_dirs(output_root: Path, start: Path) -> None:
|
||||
root = output_root.resolve()
|
||||
current = start.resolve()
|
||||
while current != root:
|
||||
try:
|
||||
current.relative_to(root)
|
||||
except ValueError:
|
||||
return
|
||||
entries = list(current.iterdir()) if current.exists() else []
|
||||
if entries and all(entry.name in IGNORED_PRUNE_FILENAMES and entry.is_file() for entry in entries):
|
||||
for entry in entries:
|
||||
entry.unlink()
|
||||
try:
|
||||
current.rmdir()
|
||||
except OSError:
|
||||
return
|
||||
current = current.parent
|
||||
|
||||
|
||||
def prune_all_empty_dirs(output_root: Path) -> None:
|
||||
if not output_root.exists():
|
||||
return
|
||||
directories = [path for path in output_root.rglob("*") if path.is_dir()]
|
||||
directories.sort(key=lambda path: len(path.relative_to(output_root).parts), reverse=True)
|
||||
for directory in directories:
|
||||
prune_empty_dirs(output_root, directory)
|
||||
|
||||
|
||||
def remove_manifest_entry_paths(output_root: Path, entry: dict, keep: set[str] | None = None) -> int:
|
||||
keep = keep or set()
|
||||
deleted = 0
|
||||
paths = [entry.get("path"), *entry.get("attachments", [])]
|
||||
for relative_path in paths:
|
||||
if not isinstance(relative_path, str) or relative_path in keep:
|
||||
continue
|
||||
path = safe_output_path(output_root, relative_path)
|
||||
if path is None or not path.exists() or not path.is_file():
|
||||
continue
|
||||
path.unlink()
|
||||
deleted += 1
|
||||
prune_empty_dirs(output_root, path.parent)
|
||||
return deleted
|
||||
|
||||
|
||||
def exported_note_attachment_names(markdown: str) -> list[str]:
|
||||
lines = markdown.splitlines()
|
||||
if not lines or lines[0] != "---":
|
||||
return []
|
||||
|
||||
names: list[str] = []
|
||||
for line in lines[1:]:
|
||||
if line == "---":
|
||||
break
|
||||
stripped = line.strip()
|
||||
if not stripped.startswith("- filename: "):
|
||||
continue
|
||||
raw = stripped.removeprefix("- filename: ").strip()
|
||||
with contextlib.suppress(json.JSONDecodeError):
|
||||
value = json.loads(raw)
|
||||
if isinstance(value, str) and value:
|
||||
names.append(value)
|
||||
return names
|
||||
|
||||
|
||||
def is_exported_note_markdown(path: Path) -> bool:
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return False
|
||||
return text.startswith("---\n") and "\nsource: notes://showNote?identifier=" in text.split("\n---", 1)[0]
|
||||
|
||||
|
||||
def remove_orphaned_export_files(output_root: Path, referenced_paths: set[str]) -> int:
|
||||
if not output_root.exists():
|
||||
return 0
|
||||
|
||||
deleted = 0
|
||||
for note_path in sorted(output_root.rglob("*.md")):
|
||||
relative_note_path = path_relative_to_output(output_root, note_path)
|
||||
if relative_note_path in referenced_paths or not is_exported_note_markdown(note_path):
|
||||
continue
|
||||
|
||||
text = note_path.read_text(encoding="utf-8", errors="replace")
|
||||
attachment_names = exported_note_attachment_names(text)
|
||||
note_path.unlink()
|
||||
deleted += 1
|
||||
|
||||
for attachment_name in attachment_names:
|
||||
attachment_path = safe_output_path(output_root, str(Path(relative_note_path).parent / attachment_name))
|
||||
if attachment_path is None or not attachment_path.exists() or not attachment_path.is_file():
|
||||
continue
|
||||
relative_attachment_path = path_relative_to_output(output_root, attachment_path)
|
||||
if relative_attachment_path in referenced_paths:
|
||||
continue
|
||||
attachment_path.unlink()
|
||||
deleted += 1
|
||||
|
||||
prune_empty_dirs(output_root, note_path.parent)
|
||||
return deleted
|
||||
|
||||
|
||||
def write_manifest(output_root: Path, notes: dict[str, dict]) -> None:
|
||||
output_root.mkdir(parents=True, exist_ok=True)
|
||||
payload = {
|
||||
"version": MANIFEST_VERSION,
|
||||
"generated_at": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
"notes": notes,
|
||||
}
|
||||
manifest_file(output_root).write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
def expected_note_path(output_root: Path, note: Note) -> Path:
|
||||
filename = f"{note_filename_stem(note.title, note.pk)}.md"
|
||||
return output_root / Path(note.folder_path) / filename
|
||||
|
||||
|
||||
def write_note(output_root: Path, note: Note, copy_attachments: bool) -> ExportResult:
|
||||
note_dir = output_root / Path(note.folder_path)
|
||||
note_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
attachment_paths: list[Path] = []
|
||||
if copy_attachments:
|
||||
used_paths: set[str] = set()
|
||||
for attachment in note.attachments:
|
||||
if attachment.source_path is None:
|
||||
continue
|
||||
destination = note_dir / attachment.output_name
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
if attachment.output_name in used_paths:
|
||||
destination = unique_path(destination)
|
||||
used_paths.add(path_relative_to_output(note_dir, destination))
|
||||
shutil.copy2(attachment.source_path, destination)
|
||||
apply_file_timestamps(destination, created=note.created, modified=note.modified)
|
||||
attachment_paths.append(destination)
|
||||
|
||||
path = expected_note_path(output_root, note)
|
||||
path.write_text(build_markdown(note), encoding="utf-8")
|
||||
apply_file_timestamps(path, created=note.created, modified=note.modified)
|
||||
return ExportResult(note_path=path, attachment_paths=attachment_paths)
|
||||
|
||||
|
||||
def write_claude_files(output_root: Path, exported_count: int) -> None:
|
||||
output_root.mkdir(parents=True, exist_ok=True)
|
||||
generated_at = dt.datetime.now().strftime("%Y-%m-%d, %H:%M")
|
||||
content = f"""# Apple Notes Export
|
||||
|
||||
This directory contains Apple Notes exported as markdown files.
|
||||
|
||||
## Structure
|
||||
|
||||
- Each Notes folder is represented as a subdirectory.
|
||||
- Each note is saved as `{{note-title}}-{{id}}.md`.
|
||||
- Attachments are saved in `attachments/{{id}}/` next to the markdown file.
|
||||
- Frontmatter includes title, Apple Notes ID, dates, folder, links, attachments, and the `notes://` source URL.
|
||||
|
||||
## Working With Notes
|
||||
|
||||
1. Preserve `source` links when editing or referencing notes.
|
||||
2. Keep the `attachments/` folder next to the markdown files that reference it.
|
||||
3. Keep folder structure intact when moving exported notes.
|
||||
|
||||
---
|
||||
Generated by `apple-notes export` • Notes exported: {exported_count} • Last updated: {generated_at}
|
||||
"""
|
||||
(output_root / "CLAUDE.md").write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
def export_notes(args: argparse.Namespace) -> int:
|
||||
notes_dir = args.notes_dir.expanduser().resolve()
|
||||
db_path = args.database.expanduser().resolve() if args.database else notes_dir / "NoteStore.sqlite"
|
||||
if not db_path.exists():
|
||||
raise FileNotFoundError(f"Notes database not found: {db_path}")
|
||||
|
||||
temp_db: Path | None = None
|
||||
if args.no_snapshot:
|
||||
read_db = db_path
|
||||
else:
|
||||
temp_db = copy_live_database(db_path)
|
||||
read_db = temp_db
|
||||
|
||||
output_root = args.output.expanduser().resolve()
|
||||
old_manifest = load_manifest(output_root) if args.incremental else {"notes": {}}
|
||||
old_notes: dict[str, dict] = old_manifest.get("notes", {})
|
||||
current_manifest_notes: dict[str, dict] = {}
|
||||
exported = 0
|
||||
written = 0
|
||||
skipped = 0
|
||||
deleted_notes = 0
|
||||
deleted_files = 0
|
||||
try:
|
||||
conn = sqlite3.connect(read_db)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
notes = iter_notes(
|
||||
conn,
|
||||
notes_dir,
|
||||
include_password_protected=args.include_password_protected,
|
||||
limit=args.limit,
|
||||
include_trashed=getattr(args, "include_trashed", False),
|
||||
)
|
||||
for note in notes:
|
||||
exported += 1
|
||||
key = str(note.pk)
|
||||
signature = note_signature(note)
|
||||
expected_relpath = path_relative_to_output(output_root, expected_note_path(output_root, note))
|
||||
old_entry = old_notes.get(key)
|
||||
unchanged = (
|
||||
args.incremental
|
||||
and isinstance(old_entry, dict)
|
||||
and old_entry.get("signature") == signature
|
||||
and old_entry.get("path") == expected_relpath
|
||||
and manifest_paths_exist(output_root, old_entry)
|
||||
and manifest_entry_timestamps_match(output_root, old_entry)
|
||||
)
|
||||
|
||||
if unchanged:
|
||||
skipped += 1
|
||||
current_manifest_notes[key] = old_entry
|
||||
continue
|
||||
|
||||
written += 1
|
||||
if args.dry_run:
|
||||
current_manifest_notes[key] = {
|
||||
"id": note.pk,
|
||||
"apple_notes_id": note.apple_notes_id,
|
||||
"title": note.title,
|
||||
"folder": note.folder_path,
|
||||
"modified": note.modified,
|
||||
"signature": signature,
|
||||
"path": expected_relpath,
|
||||
"attachments": [],
|
||||
}
|
||||
continue
|
||||
if args.incremental and isinstance(old_entry, dict):
|
||||
deleted_files += remove_manifest_entry_paths(
|
||||
output_root,
|
||||
old_entry,
|
||||
keep={expected_relpath},
|
||||
)
|
||||
result = write_note(output_root, note, copy_attachments=not args.no_attachments)
|
||||
new_entry = manifest_entry(output_root, note, result, signature)
|
||||
current_manifest_notes[key] = new_entry
|
||||
if args.incremental and isinstance(old_entry, dict):
|
||||
new_paths = {new_entry["path"], *new_entry.get("attachments", [])}
|
||||
deleted_files += remove_manifest_entry_paths(output_root, old_entry, keep=new_paths)
|
||||
finally:
|
||||
conn.close()
|
||||
finally:
|
||||
if temp_db is not None:
|
||||
temp_db.unlink(missing_ok=True)
|
||||
|
||||
if args.incremental and not args.keep_stale and args.limit is None:
|
||||
stale_keys = set(old_notes) - set(current_manifest_notes)
|
||||
deleted_notes = len(stale_keys)
|
||||
if not args.dry_run:
|
||||
for key in stale_keys:
|
||||
entry = old_notes.get(key)
|
||||
if isinstance(entry, dict):
|
||||
deleted_files += remove_manifest_entry_paths(output_root, entry)
|
||||
|
||||
if not args.dry_run:
|
||||
if args.incremental and not args.keep_stale and args.limit is None:
|
||||
deleted_files += remove_orphaned_export_files(
|
||||
output_root,
|
||||
manifest_referenced_paths(current_manifest_notes),
|
||||
)
|
||||
if getattr(args, "write_claude_md", False):
|
||||
write_claude_files(output_root, exported)
|
||||
write_manifest(output_root, current_manifest_notes)
|
||||
if args.incremental:
|
||||
prune_all_empty_dirs(output_root)
|
||||
print(f"exported_notes={exported}")
|
||||
if args.incremental:
|
||||
print(f"notes_written={written}")
|
||||
print(f"notes_skipped={skipped}")
|
||||
print(f"notes_deleted={deleted_notes}")
|
||||
print(f"files_deleted={deleted_files}")
|
||||
if not args.dry_run:
|
||||
print(f"output={output_root}")
|
||||
return 0
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Dataclasses shared across the parser, store, and exporter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class ProtoField:
|
||||
number: int
|
||||
wire_type: int
|
||||
value: int | bytes
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Attachment:
|
||||
identifier: str
|
||||
type_uti: str | None
|
||||
source_path: Path | None
|
||||
output_name: str
|
||||
|
||||
@property
|
||||
def markdown_reference(self) -> str:
|
||||
reference = self.output_name
|
||||
alt = Path(reference).name.replace("[", "\\[").replace("]", "\\]")
|
||||
if reference.lower().endswith((".png", ".jpg", ".jpeg", ".gif", ".webp")):
|
||||
return f""
|
||||
return f"[{alt}]({reference})"
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Note:
|
||||
pk: int
|
||||
title: str
|
||||
apple_notes_id: str
|
||||
folder_path: str
|
||||
created: str | None
|
||||
modified: str | None
|
||||
body: str
|
||||
attachments: list[Attachment]
|
||||
links: list[str]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class ExportResult:
|
||||
note_path: Path
|
||||
attachment_paths: list[Path]
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class EmbeddedTable:
|
||||
identifier: str
|
||||
markdown: str
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class TextRun:
|
||||
start: int
|
||||
end: int
|
||||
style_type: int | None
|
||||
indent: int
|
||||
block_quote: int | None
|
||||
checklist_done: int | None
|
||||
font_weight: int | None
|
||||
underlined: int | None
|
||||
strikethrough: int | None
|
||||
superscript: int | None
|
||||
link: str | None
|
||||
attachment_identifier: str | None
|
||||
attachment_type: str | None
|
||||
|
||||
@@ -0,0 +1,780 @@
|
||||
"""Pure decoding of the Apple Notes protobuf payload into Markdown.
|
||||
|
||||
Nothing in here touches SQLite or the filesystem: every function takes bytes
|
||||
or text and returns text. Lifted verbatim from the original exporter.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import gzip
|
||||
import re
|
||||
from typing import Iterator
|
||||
|
||||
from .models import ProtoField, TextRun
|
||||
|
||||
URL_RE = re.compile(r"https?://[^\s<>()\[\]\"',。!?;、]+")
|
||||
STYLE_TYPE_TITLE = 0
|
||||
STYLE_TYPE_HEADING = 1
|
||||
STYLE_TYPE_SUBHEADING = 2
|
||||
STYLE_TYPE_MONOSPACED = 4
|
||||
STYLE_TYPE_DOTTED_LIST = 100
|
||||
STYLE_TYPE_DASHED_LIST = 101
|
||||
STYLE_TYPE_NUMBERED_LIST = 102
|
||||
STYLE_TYPE_CHECKBOX = 103
|
||||
STYLE_TYPE_BLOCK_QUOTE = 1
|
||||
FONT_TYPE_BOLD = 1
|
||||
FONT_TYPE_ITALIC = 2
|
||||
FONT_TYPE_BOLD_ITALIC = 3
|
||||
|
||||
|
||||
def read_varint(data: bytes, offset: int) -> tuple[int, int]:
|
||||
value = 0
|
||||
shift = 0
|
||||
while offset < len(data):
|
||||
byte = data[offset]
|
||||
offset += 1
|
||||
value |= (byte & 0x7F) << shift
|
||||
if byte & 0x80 == 0:
|
||||
return value, offset
|
||||
shift += 7
|
||||
if shift > 70:
|
||||
raise ValueError("protobuf varint is too large")
|
||||
raise ValueError("unexpected end of protobuf varint")
|
||||
|
||||
|
||||
def iter_proto_fields(data: bytes) -> Iterator[ProtoField]:
|
||||
offset = 0
|
||||
while offset < len(data):
|
||||
key, offset = read_varint(data, offset)
|
||||
number = key >> 3
|
||||
wire_type = key & 0x07
|
||||
|
||||
if wire_type == 0:
|
||||
value, offset = read_varint(data, offset)
|
||||
yield ProtoField(number, wire_type, value)
|
||||
elif wire_type == 1:
|
||||
if offset + 8 > len(data):
|
||||
raise ValueError("unexpected end of fixed64 field")
|
||||
yield ProtoField(number, wire_type, data[offset : offset + 8])
|
||||
offset += 8
|
||||
elif wire_type == 2:
|
||||
length, offset = read_varint(data, offset)
|
||||
end = offset + length
|
||||
if end > len(data):
|
||||
raise ValueError("unexpected end of length-delimited field")
|
||||
yield ProtoField(number, wire_type, data[offset:end])
|
||||
offset = end
|
||||
elif wire_type == 5:
|
||||
if offset + 4 > len(data):
|
||||
raise ValueError("unexpected end of fixed32 field")
|
||||
yield ProtoField(number, wire_type, data[offset : offset + 4])
|
||||
offset += 4
|
||||
else:
|
||||
raise ValueError(f"unsupported protobuf wire type: {wire_type}")
|
||||
|
||||
|
||||
def first_bytes_field(data: bytes, number: int) -> bytes:
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number == number and field.wire_type == 2 and isinstance(field.value, bytes):
|
||||
return field.value
|
||||
raise ValueError(f"protobuf field {number} was not found")
|
||||
|
||||
|
||||
def first_varint_field(data: bytes, number: int) -> int | None:
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number == number and field.wire_type == 0 and isinstance(field.value, int):
|
||||
return field.value
|
||||
return None
|
||||
|
||||
|
||||
def first_string_field(data: bytes, number: int) -> str | None:
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number != number or field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
with contextlib.suppress(UnicodeDecodeError):
|
||||
return field.value.decode("utf-8")
|
||||
return None
|
||||
|
||||
|
||||
def note_content_from_zdata(zdata: bytes) -> bytes:
|
||||
if zdata.startswith(b"\x1f\x8b"):
|
||||
zdata = gzip.decompress(zdata)
|
||||
document = first_bytes_field(zdata, 2)
|
||||
return first_bytes_field(document, 3)
|
||||
|
||||
|
||||
def find_largest_utf8_field(data: bytes, depth: int = 0) -> str:
|
||||
best = ""
|
||||
if depth > 5:
|
||||
return best
|
||||
with contextlib.suppress(ValueError):
|
||||
for field in iter_proto_fields(data):
|
||||
if field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
with contextlib.suppress(UnicodeDecodeError):
|
||||
text = field.value.decode("utf-8")
|
||||
printable = sum(ch.isprintable() or ch in "\n\r\t" for ch in text)
|
||||
if text and printable / max(len(text), 1) > 0.9 and len(text) > len(best):
|
||||
best = text
|
||||
nested = find_largest_utf8_field(field.value, depth + 1)
|
||||
if len(nested) > len(best):
|
||||
best = nested
|
||||
return best
|
||||
|
||||
|
||||
def extract_note_text(zdata: bytes) -> str:
|
||||
try:
|
||||
content = note_content_from_zdata(zdata)
|
||||
text = first_bytes_field(content, 2).decode("utf-8")
|
||||
except (UnicodeDecodeError, ValueError, OSError):
|
||||
text = find_largest_utf8_field(zdata)
|
||||
|
||||
return normalize_note_text(text)
|
||||
|
||||
|
||||
def parse_paragraph_style(data: bytes) -> tuple[int | None, int, int | None, int | None]:
|
||||
style_type = first_varint_field(data, 1)
|
||||
indent = first_varint_field(data, 4) or 0
|
||||
block_quote = first_varint_field(data, 8)
|
||||
checklist_done = None
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number != 5 or field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
checklist_done = first_varint_field(field.value, 2)
|
||||
break
|
||||
return style_type, indent, block_quote, checklist_done
|
||||
|
||||
|
||||
def parse_text_runs(content: bytes) -> list[TextRun]:
|
||||
runs: list[TextRun] = []
|
||||
offset = 0
|
||||
for field in iter_proto_fields(content):
|
||||
if field.number != 5 or field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
|
||||
length = first_varint_field(field.value, 1)
|
||||
if length is None:
|
||||
continue
|
||||
|
||||
style_type = None
|
||||
indent = 0
|
||||
block_quote = None
|
||||
checklist_done = None
|
||||
font_weight = first_varint_field(field.value, 5)
|
||||
underlined = first_varint_field(field.value, 6)
|
||||
strikethrough = first_varint_field(field.value, 7)
|
||||
superscript = first_varint_field(field.value, 8)
|
||||
link = first_string_field(field.value, 9)
|
||||
attachment_identifier = None
|
||||
attachment_type = None
|
||||
for run_field in iter_proto_fields(field.value):
|
||||
if run_field.wire_type != 2 or not isinstance(run_field.value, bytes):
|
||||
continue
|
||||
if run_field.number == 2:
|
||||
style_type, indent, block_quote, checklist_done = parse_paragraph_style(run_field.value)
|
||||
elif run_field.number == 12:
|
||||
attachment_identifier = first_string_field(run_field.value, 1)
|
||||
attachment_type = first_string_field(run_field.value, 2)
|
||||
|
||||
runs.append(
|
||||
TextRun(
|
||||
offset,
|
||||
offset + length,
|
||||
style_type,
|
||||
indent,
|
||||
block_quote,
|
||||
checklist_done,
|
||||
font_weight,
|
||||
underlined,
|
||||
strikethrough,
|
||||
superscript,
|
||||
link,
|
||||
attachment_identifier,
|
||||
attachment_type,
|
||||
)
|
||||
)
|
||||
offset += length
|
||||
return runs
|
||||
|
||||
|
||||
def run_at_offset(runs: list[TextRun], offset: int) -> TextRun | None:
|
||||
for run in runs:
|
||||
if run.start <= offset < run.end:
|
||||
return run
|
||||
return None
|
||||
|
||||
|
||||
def line_ranges(text: str) -> list[tuple[int, str]]:
|
||||
ranges: list[tuple[int, str]] = []
|
||||
offset = 0
|
||||
for line in text.split("\n"):
|
||||
ranges.append((offset, line))
|
||||
offset += len(line) + 1
|
||||
while ranges and ranges[0][1] == "":
|
||||
ranges.pop(0)
|
||||
while ranges and ranges[-1][1] == "":
|
||||
ranges.pop()
|
||||
return ranges
|
||||
|
||||
|
||||
def overlapping_runs(runs: list[TextRun], start: int, end: int) -> list[TextRun]:
|
||||
return [run for run in runs if run.start < end and run.end > start]
|
||||
|
||||
|
||||
def block_style_key(run: TextRun) -> tuple[int | None, int, int | None, int | None]:
|
||||
return (run.style_type, run.indent, run.block_quote, run.checklist_done)
|
||||
|
||||
|
||||
def dominant_block_run(runs: list[TextRun], start: int, end: int) -> TextRun | None:
|
||||
scores: dict[tuple[int | None, int, int | None, int | None], tuple[int, int, TextRun]] = {}
|
||||
for run in overlapping_runs(runs, start, end):
|
||||
overlap = min(end, run.end) - max(start, run.start)
|
||||
if overlap <= 0:
|
||||
continue
|
||||
key = block_style_key(run)
|
||||
total, _last_start, _representative = scores.get(key, (0, run.start, run))
|
||||
scores[key] = (total + overlap, run.start, run)
|
||||
if not scores:
|
||||
return None
|
||||
return max(scores.values(), key=lambda item: (item[0], item[1]))[2]
|
||||
|
||||
|
||||
def apply_inline_markdown(text: str, run: TextRun, object_replacements: dict[str, str] | None = None) -> str:
|
||||
if not text:
|
||||
return text
|
||||
|
||||
match = re.match(r"^(\s*)(.*?)(\s*)$", text.replace("\x00", "\u2400"), re.DOTALL)
|
||||
if not match:
|
||||
return text
|
||||
leading, result, trailing = match.groups()
|
||||
if not result:
|
||||
return leading + trailing
|
||||
|
||||
if "\ufffc" in result and run.attachment_identifier and object_replacements:
|
||||
replacement = object_replacements.get(run.attachment_identifier)
|
||||
if replacement:
|
||||
result = result.replace("\ufffc", replacement)
|
||||
|
||||
if run.link:
|
||||
label = result.replace("[", "\\[").replace("]", "\\]")
|
||||
url = run.link.replace(")", "%29")
|
||||
result = f"[{label}]({url})"
|
||||
|
||||
if run.font_weight == FONT_TYPE_BOLD:
|
||||
result = f"**{result}**"
|
||||
elif run.font_weight == FONT_TYPE_ITALIC:
|
||||
result = f"*{result}*"
|
||||
elif run.font_weight == FONT_TYPE_BOLD_ITALIC:
|
||||
result = f"***{result}***"
|
||||
|
||||
if run.strikethrough == 1:
|
||||
result = f"~~{result}~~"
|
||||
if run.underlined == 1:
|
||||
result = f"<u>{result}</u>"
|
||||
if run.superscript == 1:
|
||||
result = f"<sup>{result}</sup>"
|
||||
elif run.superscript == -1:
|
||||
result = f"<sub>{result}</sub>"
|
||||
return leading + result + trailing
|
||||
|
||||
|
||||
def add_spacing_after_bold_labels(text: str) -> str:
|
||||
return re.sub(r"(\*\*[^*\n]{1,40}[::]\*\*)(?=\S)", r"\1 ", text)
|
||||
|
||||
|
||||
def fence_for_lines(lines: list[str]) -> str:
|
||||
longest = max((len(match.group(0)) for line in lines for match in re.finditer(r"`+", line)), default=0)
|
||||
return "`" * max(3, longest + 1)
|
||||
|
||||
|
||||
def should_preserve_plain_block(lines: list[str]) -> bool:
|
||||
if len(lines) < 2:
|
||||
return False
|
||||
if any(is_markdown_block_line(line) or has_markdown_inline_markup(line) for line in lines):
|
||||
return False
|
||||
if any(line.startswith("\t") for line in lines):
|
||||
return True
|
||||
return len(lines) >= 3 and any(line.strip() == "↓" for line in lines)
|
||||
|
||||
|
||||
def has_markdown_inline_markup(line: str) -> bool:
|
||||
return "](" in line or "<u>" in line or "</u>" in line
|
||||
|
||||
|
||||
def is_markdown_block_line(line: str) -> bool:
|
||||
return bool(re.match(r"^\s*(#{1,6}\s|[-*+]\s|\d+\.\s|>\s|```|\|)", line))
|
||||
|
||||
|
||||
def replace_leading_tabs(line: str) -> str:
|
||||
tabs = len(line) - len(line.lstrip("\t"))
|
||||
if tabs == 0:
|
||||
return line
|
||||
return " " * tabs + line[tabs:]
|
||||
|
||||
|
||||
def with_hard_breaks(lines: list[str]) -> list[str]:
|
||||
rendered: list[str] = []
|
||||
for index, line in enumerate(lines):
|
||||
line = replace_leading_tabs(line)
|
||||
next_line = lines[index + 1] if index + 1 < len(lines) else None
|
||||
if next_line is not None and not is_markdown_block_line(line) and not is_markdown_block_line(next_line):
|
||||
rendered.append(line.rstrip() + " ")
|
||||
else:
|
||||
rendered.append(line)
|
||||
return rendered
|
||||
|
||||
|
||||
def preserve_plain_structure_blocks(text: str) -> str:
|
||||
groups: list[list[str]] = []
|
||||
current: list[str] = []
|
||||
for line in text.split("\n"):
|
||||
if line == "":
|
||||
if current:
|
||||
groups.append(current)
|
||||
current = []
|
||||
groups.append([])
|
||||
else:
|
||||
current.append(line)
|
||||
if current:
|
||||
groups.append(current)
|
||||
|
||||
rendered: list[str] = []
|
||||
for group in groups:
|
||||
if not group:
|
||||
rendered.append("")
|
||||
continue
|
||||
if should_preserve_plain_block(group):
|
||||
fence = fence_for_lines(group)
|
||||
rendered.extend([fence, *group, fence])
|
||||
else:
|
||||
rendered.extend(with_hard_breaks(group))
|
||||
return "\n".join(rendered)
|
||||
|
||||
|
||||
def inline_style_key(run: TextRun) -> tuple[int | None, int | None, int | None, int | None, str | None, str | None]:
|
||||
return (
|
||||
run.font_weight,
|
||||
run.underlined,
|
||||
run.strikethrough,
|
||||
run.superscript,
|
||||
run.link,
|
||||
run.attachment_identifier,
|
||||
)
|
||||
|
||||
|
||||
def is_plain_inline_key(key: tuple | None) -> bool:
|
||||
return key is None or all(value is None for value in key)
|
||||
|
||||
|
||||
def is_plain_segment(key: tuple | None, text: str) -> bool:
|
||||
return is_plain_inline_key(key) and len(text) == 1 and bool(text.strip())
|
||||
|
||||
|
||||
def merge_leading_style_fragments(
|
||||
segments: list[tuple[tuple | None, TextRun | None, str]],
|
||||
) -> list[tuple[tuple | None, TextRun | None, str]]:
|
||||
merged: list[tuple[tuple | None, TextRun | None, str]] = []
|
||||
index = 0
|
||||
while index < len(segments):
|
||||
key, run, text = segments[index]
|
||||
if (
|
||||
is_plain_segment(key, text)
|
||||
and index + 1 < len(segments)
|
||||
and not is_plain_inline_key(segments[index + 1][0])
|
||||
and segments[index + 1][2]
|
||||
and not segments[index + 1][2][0].isspace()
|
||||
):
|
||||
next_key, next_run, next_text = segments[index + 1]
|
||||
merged.append((next_key, next_run, text + next_text))
|
||||
index += 2
|
||||
continue
|
||||
merged.append((key, run, text))
|
||||
index += 1
|
||||
return merged
|
||||
|
||||
|
||||
def render_inline_markdown(
|
||||
text: str,
|
||||
runs: list[TextRun],
|
||||
start: int,
|
||||
object_replacements: dict[str, str] | None = None,
|
||||
) -> str:
|
||||
if not text:
|
||||
return text
|
||||
|
||||
end = start + len(text)
|
||||
segments: list[tuple[tuple | None, TextRun | None, str]] = []
|
||||
cursor = start
|
||||
for run in overlapping_runs(runs, start, end):
|
||||
if run.start > cursor:
|
||||
segments.append((None, None, text[cursor - start : run.start - start]))
|
||||
segment_start = max(cursor, run.start)
|
||||
segment_end = min(end, run.end)
|
||||
if segment_start < segment_end:
|
||||
segment = text[segment_start - start : segment_end - start]
|
||||
key = inline_style_key(run)
|
||||
if segments and segments[-1][0] == key:
|
||||
previous_key, previous_run, previous_text = segments[-1]
|
||||
segments[-1] = (previous_key, previous_run, previous_text + segment)
|
||||
else:
|
||||
segments.append((key, run, segment))
|
||||
cursor = max(cursor, segment_end)
|
||||
if cursor < end:
|
||||
segments.append((None, None, text[cursor - start :]))
|
||||
segments = merge_leading_style_fragments(segments)
|
||||
|
||||
parts: list[str] = []
|
||||
for _key, run, segment in segments:
|
||||
if run is None:
|
||||
parts.append(segment)
|
||||
else:
|
||||
parts.append(apply_inline_markdown(segment, run, object_replacements))
|
||||
return add_spacing_after_bold_labels("".join(parts))
|
||||
|
||||
|
||||
def line_prefix(style_type: int | None, run: TextRun | None, line_number: int) -> tuple[str, bool]:
|
||||
indent = " " * (run.indent if run else 0)
|
||||
if style_type == STYLE_TYPE_CHECKBOX:
|
||||
checked = "x" if run and run.checklist_done == 1 else " "
|
||||
return f"{indent}- [{checked}] ", True
|
||||
if style_type in {STYLE_TYPE_DOTTED_LIST, STYLE_TYPE_DASHED_LIST}:
|
||||
return f"{indent}- ", True
|
||||
if style_type == STYLE_TYPE_NUMBERED_LIST:
|
||||
return f"{indent}{line_number}. ", True
|
||||
if run and run.block_quote == STYLE_TYPE_BLOCK_QUOTE:
|
||||
return "> " * max(run.indent, 1), False
|
||||
if run and run.indent > 0:
|
||||
return "> " * run.indent, False
|
||||
if style_type == STYLE_TYPE_TITLE:
|
||||
return "### ", True
|
||||
if style_type == STYLE_TYPE_HEADING:
|
||||
return "#### ", True
|
||||
if style_type == STYLE_TYPE_SUBHEADING:
|
||||
return "##### ", True
|
||||
if style_type == STYLE_TYPE_MONOSPACED:
|
||||
return " ", False
|
||||
return "", False
|
||||
|
||||
|
||||
def render_note_text_as_markdown(
|
||||
text: str,
|
||||
runs: list[TextRun],
|
||||
object_replacements: dict[str, str] | None = None,
|
||||
) -> str:
|
||||
if not runs:
|
||||
return preserve_plain_structure_blocks(normalize_note_text(text))
|
||||
|
||||
rendered: list[str] = []
|
||||
ordered_numbers: dict[int, int] = {}
|
||||
for start, line in line_ranges(text):
|
||||
if not line:
|
||||
rendered.append("")
|
||||
ordered_numbers.clear()
|
||||
continue
|
||||
|
||||
first_content = len(line) - len(line.lstrip())
|
||||
line_start = start + first_content
|
||||
line_end = start + len(line)
|
||||
run = dominant_block_run(runs, line_start, line_end)
|
||||
style_type = run.style_type if run else None
|
||||
if style_type in {STYLE_TYPE_TITLE, STYLE_TYPE_HEADING, STYLE_TYPE_SUBHEADING} and len(line.strip()) > 80:
|
||||
style_type = None
|
||||
if style_type == STYLE_TYPE_NUMBERED_LIST:
|
||||
indent = run.indent if run else 0
|
||||
line_number = ordered_numbers.get(indent, 1)
|
||||
prefix, strip_line = line_prefix(style_type, run, line_number)
|
||||
ordered_numbers[indent] = line_number + 1
|
||||
for deeper_indent in [key for key in ordered_numbers if key > indent]:
|
||||
del ordered_numbers[deeper_indent]
|
||||
else:
|
||||
prefix, strip_line = line_prefix(style_type, run, 1)
|
||||
if style_type not in {STYLE_TYPE_CHECKBOX, STYLE_TYPE_DOTTED_LIST, STYLE_TYPE_DASHED_LIST}:
|
||||
ordered_numbers.clear()
|
||||
|
||||
content = line.lstrip() if strip_line else line
|
||||
content_start = start + (first_content if strip_line else 0)
|
||||
rendered.append(prefix + render_inline_markdown(content, runs, content_start, object_replacements))
|
||||
return preserve_plain_structure_blocks("\n".join(rendered))
|
||||
|
||||
|
||||
def extract_note_markdown(zdata: bytes, object_replacements: dict[str, str] | None = None) -> str:
|
||||
try:
|
||||
content = note_content_from_zdata(zdata)
|
||||
text = first_bytes_field(content, 2).decode("utf-8")
|
||||
runs = parse_text_runs(content)
|
||||
return render_note_text_as_markdown(text, runs, object_replacements)
|
||||
except (UnicodeDecodeError, ValueError, OSError):
|
||||
return extract_note_text(zdata)
|
||||
|
||||
|
||||
def normalize_note_text(text: str) -> str:
|
||||
text = text.replace("\r\n", "\n").replace("\r", "\n")
|
||||
text = text.replace("\u2028", "\n").replace("\u2029", "\n")
|
||||
return text.strip("\n")
|
||||
|
||||
|
||||
|
||||
|
||||
def object_id(data: bytes) -> dict[str, int | str]:
|
||||
result: dict[str, int | str] = {}
|
||||
unsigned = first_varint_field(data, 2)
|
||||
string = first_string_field(data, 4)
|
||||
index = first_varint_field(data, 6)
|
||||
if unsigned is not None:
|
||||
result["unsigned"] = unsigned
|
||||
if string is not None:
|
||||
result["string"] = string
|
||||
if index is not None:
|
||||
result["index"] = index
|
||||
return result
|
||||
|
||||
|
||||
def custom_map(data: bytes) -> tuple[int | None, list[tuple[int, dict[str, int | str]]]]:
|
||||
map_type = first_varint_field(data, 1)
|
||||
entries: list[tuple[int, dict[str, int | str]]] = []
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number != 3 or field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
key = first_varint_field(field.value, 1)
|
||||
value = None
|
||||
for entry_field in iter_proto_fields(field.value):
|
||||
if entry_field.number == 2 and entry_field.wire_type == 2 and isinstance(entry_field.value, bytes):
|
||||
value = object_id(entry_field.value)
|
||||
break
|
||||
if key is not None and value is not None:
|
||||
entries.append((key, value))
|
||||
return map_type, entries
|
||||
|
||||
|
||||
def dictionary_elements(data: bytes) -> list[tuple[dict[str, int | str], dict[str, int | str]]]:
|
||||
elements: list[tuple[dict[str, int | str], dict[str, int | str]]] = []
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number != 1 or field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
key = None
|
||||
value = None
|
||||
for element_field in iter_proto_fields(field.value):
|
||||
if element_field.wire_type != 2 or not isinstance(element_field.value, bytes):
|
||||
continue
|
||||
if element_field.number == 1:
|
||||
key = object_id(element_field.value)
|
||||
elif element_field.number == 2:
|
||||
value = object_id(element_field.value)
|
||||
if key is not None and value is not None:
|
||||
elements.append((key, value))
|
||||
return elements
|
||||
|
||||
|
||||
def entry_field(entry: bytes, number: int) -> bytes | None:
|
||||
for field in iter_proto_fields(entry):
|
||||
if field.number == number and field.wire_type == 2 and isinstance(field.value, bytes):
|
||||
return field.value
|
||||
return None
|
||||
|
||||
|
||||
def entry_custom_map(entry: bytes) -> tuple[int | None, list[tuple[int, dict[str, int | str]]]] | None:
|
||||
data = entry_field(entry, 13)
|
||||
if data is None:
|
||||
return None
|
||||
return custom_map(data)
|
||||
|
||||
|
||||
def entry_type(entry: bytes, type_items: list[str]) -> str | None:
|
||||
decoded = entry_custom_map(entry)
|
||||
if decoded is None:
|
||||
return None
|
||||
type_index, _entries = decoded
|
||||
if type_index is None or type_index >= len(type_items):
|
||||
return None
|
||||
return type_items[type_index]
|
||||
|
||||
|
||||
def target_uuid_index(entry: bytes) -> int | None:
|
||||
decoded = entry_custom_map(entry)
|
||||
if decoded is None:
|
||||
return None
|
||||
_type_index, entries = decoded
|
||||
if not entries:
|
||||
return None
|
||||
value = entries[0][1]
|
||||
unsigned = value.get("unsigned")
|
||||
return unsigned if isinstance(unsigned, int) else None
|
||||
|
||||
|
||||
def ordered_set_parts(entry: bytes) -> tuple[list[bytes], list[tuple[dict[str, int | str], dict[str, int | str]]]]:
|
||||
ordered_set = entry_field(entry, 16)
|
||||
if ordered_set is None:
|
||||
return [], []
|
||||
|
||||
ordering = entry_field(ordered_set, 1)
|
||||
if ordering is None:
|
||||
return [], []
|
||||
|
||||
attachments: list[bytes] = []
|
||||
elements: list[tuple[dict[str, int | str], dict[str, int | str]]] = []
|
||||
for field in iter_proto_fields(ordering):
|
||||
if field.wire_type != 2 or not isinstance(field.value, bytes):
|
||||
continue
|
||||
if field.number == 1:
|
||||
for array_field in iter_proto_fields(field.value):
|
||||
if array_field.number != 2 or array_field.wire_type != 2 or not isinstance(array_field.value, bytes):
|
||||
continue
|
||||
for attachment_field in iter_proto_fields(array_field.value):
|
||||
if attachment_field.number == 2 and attachment_field.wire_type == 2 and isinstance(
|
||||
attachment_field.value, bytes
|
||||
):
|
||||
attachments.append(attachment_field.value)
|
||||
elif field.number == 2:
|
||||
elements = dictionary_elements(field.value)
|
||||
return attachments, elements
|
||||
|
||||
|
||||
def index_from_object_id(value: dict[str, int | str]) -> int | None:
|
||||
index = value.get("index")
|
||||
return index if isinstance(index, int) else None
|
||||
|
||||
|
||||
def table_axis_indices(entry: bytes, table_objects: list[bytes], uuid_items: list[bytes]) -> dict[int, int]:
|
||||
attachments, elements = ordered_set_parts(entry)
|
||||
indices: dict[int, int] = {}
|
||||
for position, uuid in enumerate(attachments):
|
||||
with contextlib.suppress(ValueError):
|
||||
indices[uuid_items.index(uuid)] = position
|
||||
|
||||
for key, value in elements:
|
||||
key_index = index_from_object_id(key)
|
||||
value_index = index_from_object_id(value)
|
||||
if key_index is None or value_index is None:
|
||||
continue
|
||||
if key_index >= len(table_objects) or value_index >= len(table_objects):
|
||||
continue
|
||||
key_uuid = target_uuid_index(table_objects[key_index])
|
||||
value_uuid = target_uuid_index(table_objects[value_index])
|
||||
if key_uuid in indices and value_uuid is not None:
|
||||
indices[value_uuid] = indices[key_uuid]
|
||||
return indices
|
||||
|
||||
|
||||
def note_message_to_markdown(note_message: bytes) -> str:
|
||||
text = first_bytes_field(note_message, 2).decode("utf-8")
|
||||
return render_note_text_as_markdown(text, parse_text_runs(note_message)).strip()
|
||||
|
||||
|
||||
def markdown_table_cell(value: str) -> str:
|
||||
value = value.replace("|", "\\|")
|
||||
value = re.sub(r"\n{2,}", "<br><br>", value)
|
||||
return value.replace("\n", "<br>").strip()
|
||||
|
||||
|
||||
def markdown_table(rows: list[list[str]]) -> str:
|
||||
rows = [[markdown_table_cell(cell) for cell in row] for row in rows if row]
|
||||
if not rows:
|
||||
return ""
|
||||
width = max(len(row) for row in rows)
|
||||
normalized = [row + [""] * (width - len(row)) for row in rows]
|
||||
header = normalized[0]
|
||||
separator = ["---"] * width
|
||||
lines = [
|
||||
"| " + " | ".join(header) + " |",
|
||||
"| " + " | ".join(separator) + " |",
|
||||
]
|
||||
for row in normalized[1:]:
|
||||
lines.append("| " + " | ".join(row) + " |")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def extract_markdown_table(mergeable_data: bytes) -> str:
|
||||
if mergeable_data.startswith(b"\x1f\x8b"):
|
||||
mergeable_data = gzip.decompress(mergeable_data)
|
||||
|
||||
data = first_bytes_field(first_bytes_field(mergeable_data, 2), 3)
|
||||
table_objects: list[bytes] = []
|
||||
key_items: list[str] = []
|
||||
type_items: list[str] = []
|
||||
uuid_items: list[bytes] = []
|
||||
for field in iter_proto_fields(data):
|
||||
if field.number == 3 and field.wire_type == 2 and isinstance(field.value, bytes):
|
||||
table_objects.append(field.value)
|
||||
elif field.number == 4 and field.wire_type == 2 and isinstance(field.value, bytes):
|
||||
key_items.append(field.value.decode("utf-8"))
|
||||
elif field.number == 5 and field.wire_type == 2 and isinstance(field.value, bytes):
|
||||
type_items.append(field.value.decode("utf-8"))
|
||||
elif field.number == 6 and field.wire_type == 2 and isinstance(field.value, bytes):
|
||||
uuid_items.append(field.value)
|
||||
|
||||
for table_entry in table_objects:
|
||||
if entry_type(table_entry, type_items) != "com.apple.notes.ICTable":
|
||||
continue
|
||||
|
||||
decoded = entry_custom_map(table_entry)
|
||||
if decoded is None:
|
||||
continue
|
||||
_type_index, map_entries = decoded
|
||||
row_indices: dict[int, int] = {}
|
||||
column_indices: dict[int, int] = {}
|
||||
cell_columns_entry: bytes | None = None
|
||||
|
||||
for key_index, value in map_entries:
|
||||
if key_index >= len(key_items):
|
||||
continue
|
||||
object_index = index_from_object_id(value)
|
||||
if object_index is None or object_index >= len(table_objects):
|
||||
continue
|
||||
key = key_items[key_index]
|
||||
if key == "crRows":
|
||||
row_indices = table_axis_indices(table_objects[object_index], table_objects, uuid_items)
|
||||
elif key == "crColumns":
|
||||
column_indices = table_axis_indices(table_objects[object_index], table_objects, uuid_items)
|
||||
elif key == "cellColumns":
|
||||
cell_columns_entry = table_objects[object_index]
|
||||
|
||||
if not row_indices or not column_indices or cell_columns_entry is None:
|
||||
continue
|
||||
|
||||
rows = [["" for _ in range(max(column_indices.values()) + 1)] for _ in range(max(row_indices.values()) + 1)]
|
||||
dictionary = entry_field(cell_columns_entry, 6)
|
||||
if dictionary is None:
|
||||
continue
|
||||
for column_key, column_value in dictionary_elements(dictionary):
|
||||
column_object_index = index_from_object_id(column_key)
|
||||
row_dictionary_index = index_from_object_id(column_value)
|
||||
if column_object_index is None or row_dictionary_index is None:
|
||||
continue
|
||||
if column_object_index >= len(table_objects) or row_dictionary_index >= len(table_objects):
|
||||
continue
|
||||
column_uuid = target_uuid_index(table_objects[column_object_index])
|
||||
if column_uuid not in column_indices:
|
||||
continue
|
||||
row_dictionary = entry_field(table_objects[row_dictionary_index], 6)
|
||||
if row_dictionary is None:
|
||||
continue
|
||||
for row_key, row_value in dictionary_elements(row_dictionary):
|
||||
row_object_index = index_from_object_id(row_key)
|
||||
cell_index = index_from_object_id(row_value)
|
||||
if row_object_index is None or cell_index is None:
|
||||
continue
|
||||
if row_object_index >= len(table_objects) or cell_index >= len(table_objects):
|
||||
continue
|
||||
row_uuid = target_uuid_index(table_objects[row_object_index])
|
||||
if row_uuid not in row_indices:
|
||||
continue
|
||||
note_message = entry_field(table_objects[cell_index], 10)
|
||||
if note_message is None:
|
||||
continue
|
||||
rows[row_indices[row_uuid]][column_indices[column_uuid]] = note_message_to_markdown(note_message)
|
||||
table = markdown_table(rows)
|
||||
if table:
|
||||
return table
|
||||
return ""
|
||||
|
||||
|
||||
def summary_table_fallback(summary: str | None) -> str:
|
||||
if not summary:
|
||||
return ""
|
||||
lines = [line for line in normalize_note_text(summary).splitlines()]
|
||||
if not lines:
|
||||
return ""
|
||||
return "\n".join(f"> {line}" if line else ">" for line in lines)
|
||||
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
"""Search and browse notes straight from the live database.
|
||||
|
||||
Apple's store keeps note bodies as gzipped protobuf in ``ZICNOTEDATA.ZDATA``,
|
||||
and there is no FTS index anywhere in the schema, so full-text search means
|
||||
inflating every body and matching in memory. That sounds expensive and isn't:
|
||||
a ~800-note store decodes end to end in well under a second, which is why this
|
||||
does no caching and always reports the current state of the database.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import datetime as dt
|
||||
import re
|
||||
import sqlite3
|
||||
|
||||
from .parser import extract_note_markdown, extract_note_text
|
||||
from .store import apple_date, query_folder_paths, trashed_folder_pks
|
||||
|
||||
SINCE_RE = re.compile(r"^(\d+)([smhdw])$")
|
||||
SINCE_UNITS = {"s": "seconds", "m": "minutes", "h": "hours", "d": "days", "w": "weeks"}
|
||||
APPLE_EPOCH_OFFSET = 978307200.0
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class NoteSummary:
|
||||
"""A note's metadata, read without inflating its body."""
|
||||
|
||||
pk: int
|
||||
title: str
|
||||
folder: str
|
||||
created: str | None
|
||||
modified: str | None
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class SearchHit:
|
||||
note: NoteSummary
|
||||
match_count: int
|
||||
snippets: list[str]
|
||||
|
||||
|
||||
def parse_since(value: str) -> dt.datetime:
|
||||
"""Accept a relative offset (``30m``, ``2d``, ``1w``) or an ISO date."""
|
||||
match = SINCE_RE.match(value.strip())
|
||||
if match:
|
||||
amount, unit = int(match.group(1)), match.group(2)
|
||||
delta = dt.timedelta(**{SINCE_UNITS[unit]: amount})
|
||||
return dt.datetime.now(dt.timezone.utc) - delta
|
||||
text = value.strip()
|
||||
try:
|
||||
parsed = dt.datetime.fromisoformat(text)
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"cannot read {value!r} as a time; use 30m/6h/2d/1w or 2026-09-01"
|
||||
) from exc
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.astimezone()
|
||||
return parsed
|
||||
|
||||
|
||||
def _to_apple_time(moment: dt.datetime) -> float:
|
||||
return moment.timestamp() - APPLE_EPOCH_OFFSET
|
||||
|
||||
|
||||
def _build_query(
|
||||
*,
|
||||
since: dt.datetime | None,
|
||||
folder: str | None,
|
||||
include_password_protected: bool,
|
||||
with_body: bool,
|
||||
newest_first: bool,
|
||||
limit: int | None,
|
||||
trashed: set[int],
|
||||
) -> tuple[str, list]:
|
||||
columns = [
|
||||
"n.Z_PK",
|
||||
"n.ZTITLE1",
|
||||
"n.ZFOLDER",
|
||||
# Which creation-date column is populated varies by how and when the
|
||||
# note was created, so every candidate is read and the first non-null
|
||||
# one wins. Matches the exporter's behaviour.
|
||||
"n.ZCREATIONDATE",
|
||||
"n.ZCREATIONDATE1",
|
||||
"n.ZCREATIONDATE2",
|
||||
"n.ZCREATIONDATE3",
|
||||
"n.ZMODIFICATIONDATE1",
|
||||
]
|
||||
if with_body:
|
||||
columns.append("d.ZDATA")
|
||||
where = ["d.ZDATA IS NOT NULL", "IFNULL(n.ZMARKEDFORDELETION, 0) = 0"]
|
||||
params: list = []
|
||||
if not include_password_protected:
|
||||
where.append("IFNULL(n.ZISPASSWORDPROTECTED, 0) = 0")
|
||||
if trashed:
|
||||
placeholders = ", ".join(str(int(pk)) for pk in sorted(trashed))
|
||||
where.append(f"IFNULL(n.ZFOLDER, 0) NOT IN ({placeholders})")
|
||||
if since is not None:
|
||||
where.append("n.ZMODIFICATIONDATE1 >= ?")
|
||||
params.append(_to_apple_time(since))
|
||||
order = "DESC" if newest_first else "ASC"
|
||||
sql = f"""
|
||||
SELECT {", ".join(columns)}
|
||||
FROM ZICCLOUDSYNCINGOBJECT n
|
||||
JOIN ZICNOTEDATA d ON n.Z_PK = d.ZNOTE
|
||||
WHERE {" AND ".join(where)}
|
||||
ORDER BY n.ZMODIFICATIONDATE1 {order}, n.Z_PK {order}
|
||||
"""
|
||||
# A folder filter cannot go in SQL: folder paths are assembled in Python
|
||||
# from the parent chain, so limiting rows here would drop nested matches.
|
||||
if limit is not None and folder is None:
|
||||
sql += " LIMIT ?"
|
||||
params.append(limit)
|
||||
return sql, params
|
||||
|
||||
|
||||
def _created(row: sqlite3.Row) -> str | None:
|
||||
for column in (
|
||||
"ZCREATIONDATE1",
|
||||
"ZCREATIONDATE",
|
||||
"ZCREATIONDATE2",
|
||||
"ZCREATIONDATE3",
|
||||
):
|
||||
with contextlib.suppress(IndexError):
|
||||
stamp = apple_date(row[column])
|
||||
if stamp:
|
||||
return stamp
|
||||
return None
|
||||
|
||||
|
||||
def _summary(row: sqlite3.Row, folder_paths: dict[int, str]) -> NoteSummary:
|
||||
folder = folder_paths.get(row["ZFOLDER"], "Notes") if row["ZFOLDER"] else "Notes"
|
||||
return NoteSummary(
|
||||
pk=int(row["Z_PK"]),
|
||||
title=(row["ZTITLE1"] or "Untitled").strip() or "Untitled",
|
||||
folder=folder,
|
||||
created=_created(row),
|
||||
modified=apple_date(row["ZMODIFICATIONDATE1"]),
|
||||
)
|
||||
|
||||
|
||||
def _folder_matches(candidate: str, wanted: str) -> bool:
|
||||
return wanted.lower() in candidate.lower()
|
||||
|
||||
|
||||
def recent(
|
||||
conn: sqlite3.Connection,
|
||||
*,
|
||||
limit: int | None = 20,
|
||||
since: dt.datetime | None = None,
|
||||
folder: str | None = None,
|
||||
include_password_protected: bool = False,
|
||||
include_trashed: bool = False,
|
||||
) -> list[NoteSummary]:
|
||||
"""List notes newest-first. Reads metadata only — no body inflation."""
|
||||
folder_paths = query_folder_paths(conn)
|
||||
sql, params = _build_query(
|
||||
since=since,
|
||||
folder=folder,
|
||||
include_password_protected=include_password_protected,
|
||||
with_body=False,
|
||||
newest_first=True,
|
||||
limit=limit,
|
||||
trashed=set() if include_trashed else trashed_folder_pks(conn),
|
||||
)
|
||||
results = []
|
||||
for row in conn.execute(sql, params):
|
||||
summary = _summary(row, folder_paths)
|
||||
if folder and not _folder_matches(summary.folder, folder):
|
||||
continue
|
||||
results.append(summary)
|
||||
if limit is not None and len(results) >= limit:
|
||||
break
|
||||
return results
|
||||
|
||||
|
||||
def _snippet(text: str, start: int, end: int, width: int) -> str:
|
||||
left = max(0, start - width)
|
||||
right = min(len(text), end + width)
|
||||
fragment = text[left:right].replace("\n", " ").strip()
|
||||
fragment = re.sub(r"\s+", " ", fragment)
|
||||
prefix = "…" if left > 0 else ""
|
||||
suffix = "…" if right < len(text) else ""
|
||||
return f"{prefix}{fragment}{suffix}"
|
||||
|
||||
|
||||
def search(
|
||||
conn: sqlite3.Connection,
|
||||
query: str,
|
||||
*,
|
||||
regex: bool = False,
|
||||
case_sensitive: bool = False,
|
||||
limit: int | None = None,
|
||||
since: dt.datetime | None = None,
|
||||
folder: str | None = None,
|
||||
include_password_protected: bool = False,
|
||||
include_trashed: bool = False,
|
||||
snippet_width: int = 60,
|
||||
max_snippets: int = 3,
|
||||
search_titles: bool = True,
|
||||
) -> list[SearchHit]:
|
||||
"""Full-text search across every note body.
|
||||
|
||||
Every candidate body is inflated and matched in memory; results are ranked
|
||||
by match count, then by recency.
|
||||
"""
|
||||
flags = 0 if case_sensitive else re.IGNORECASE
|
||||
pattern = re.compile(query if regex else re.escape(query), flags)
|
||||
|
||||
folder_paths = query_folder_paths(conn)
|
||||
sql, params = _build_query(
|
||||
since=since,
|
||||
folder=folder,
|
||||
include_password_protected=include_password_protected,
|
||||
with_body=True,
|
||||
newest_first=True,
|
||||
limit=None,
|
||||
trashed=set() if include_trashed else trashed_folder_pks(conn),
|
||||
)
|
||||
|
||||
hits: list[SearchHit] = []
|
||||
for row in conn.execute(sql, params):
|
||||
summary = _summary(row, folder_paths)
|
||||
if folder and not _folder_matches(summary.folder, folder):
|
||||
continue
|
||||
try:
|
||||
body = extract_note_text(row["ZDATA"])
|
||||
except Exception:
|
||||
# A single unreadable note must not abort the whole search.
|
||||
continue
|
||||
haystack = f"{summary.title}\n{body}" if search_titles else body
|
||||
matches = list(pattern.finditer(haystack))
|
||||
if not matches:
|
||||
continue
|
||||
snippets = [
|
||||
_snippet(haystack, m.start(), m.end(), snippet_width)
|
||||
for m in matches[:max_snippets]
|
||||
]
|
||||
hits.append(SearchHit(note=summary, match_count=len(matches), snippets=snippets))
|
||||
|
||||
# ISO-8601 strings sort chronologically, so one pass ranks by match count
|
||||
# and breaks ties by recency.
|
||||
hits.sort(key=lambda h: (h.match_count, h.note.modified or ""), reverse=True)
|
||||
return hits[:limit] if limit is not None else hits
|
||||
|
||||
|
||||
def show(
|
||||
conn: sqlite3.Connection,
|
||||
pk: int,
|
||||
*,
|
||||
markdown: bool = True,
|
||||
) -> tuple[NoteSummary, str]:
|
||||
"""Fetch one note's metadata and body by primary key."""
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT n.Z_PK, n.ZTITLE1, n.ZFOLDER, n.ZCREATIONDATE,
|
||||
n.ZCREATIONDATE1, n.ZCREATIONDATE2, n.ZCREATIONDATE3,
|
||||
n.ZMODIFICATIONDATE1, d.ZDATA
|
||||
FROM ZICCLOUDSYNCINGOBJECT n
|
||||
JOIN ZICNOTEDATA d ON n.Z_PK = d.ZNOTE
|
||||
WHERE n.Z_PK = ? AND d.ZDATA IS NOT NULL
|
||||
""",
|
||||
(pk,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise KeyError(f"no note with id {pk}")
|
||||
summary = _summary(row, query_folder_paths(conn))
|
||||
body = (
|
||||
extract_note_markdown(row["ZDATA"]) if markdown else extract_note_text(row["ZDATA"])
|
||||
)
|
||||
return summary, body
|
||||
|
||||
|
||||
def folders(
|
||||
conn: sqlite3.Connection, *, include_trashed: bool = False
|
||||
) -> list[tuple[str, int]]:
|
||||
"""Every folder path with its note count, ordered by path."""
|
||||
folder_paths = query_folder_paths(conn)
|
||||
trashed = set() if include_trashed else trashed_folder_pks(conn)
|
||||
counts: dict[str, int] = {}
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT n.ZFOLDER
|
||||
FROM ZICCLOUDSYNCINGOBJECT n
|
||||
JOIN ZICNOTEDATA d ON n.Z_PK = d.ZNOTE
|
||||
WHERE d.ZDATA IS NOT NULL AND IFNULL(n.ZMARKEDFORDELETION, 0) = 0
|
||||
"""
|
||||
)
|
||||
for row in rows:
|
||||
if row["ZFOLDER"] and int(row["ZFOLDER"]) in trashed:
|
||||
continue
|
||||
path = folder_paths.get(row["ZFOLDER"], "Notes") if row["ZFOLDER"] else "Notes"
|
||||
counts[path] = counts.get(path, 0) + 1
|
||||
return sorted(counts.items())
|
||||
@@ -0,0 +1,395 @@
|
||||
"""Read-only access to the Apple Notes Core Data store.
|
||||
|
||||
Two ways in:
|
||||
|
||||
* ``connect()`` opens the live database with ``mode=ro``. SQLite's WAL mode
|
||||
supports concurrent readers, so this is safe while Notes.app is running and
|
||||
it always sees the newest committed data.
|
||||
* ``snapshot()`` copies the database via ``backup()`` first. Slower, but gives
|
||||
a stable point-in-time view for long operations like a full export.
|
||||
|
||||
Never open the store with ``immutable=1``: that flag makes SQLite skip the
|
||||
``-wal`` file entirely, silently serving a stale snapshot that is missing the
|
||||
most recent notes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import datetime as dt
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Iterator
|
||||
|
||||
from .models import Attachment, EmbeddedTable, Note
|
||||
from .parser import (
|
||||
URL_RE,
|
||||
extract_markdown_table,
|
||||
extract_note_markdown,
|
||||
summary_table_fallback,
|
||||
)
|
||||
from .utils import sanitize_filename
|
||||
|
||||
APPLE_EPOCH = dt.datetime(2001, 1, 1, tzinfo=dt.timezone.utc)
|
||||
DEFAULT_NOTES_DIR = Path.home() / "Library/Group Containers/group.com.apple.notes"
|
||||
|
||||
|
||||
def default_database(notes_dir: Path | None = None) -> Path:
|
||||
return (notes_dir or DEFAULT_NOTES_DIR) / "NoteStore.sqlite"
|
||||
|
||||
|
||||
def connect(database: Path) -> sqlite3.Connection:
|
||||
"""Open the live store read-only. Sees WAL content; does not block Notes.app."""
|
||||
if not database.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Apple Notes database not found at {database}.\n"
|
||||
"Notes.app must have been launched at least once on this machine."
|
||||
)
|
||||
conn = sqlite3.connect(f"file:{database}?mode=ro", uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def apple_date(value: float | int | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
numeric = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if numeric <= 0:
|
||||
return None
|
||||
return (APPLE_EPOCH + dt.timedelta(seconds=numeric)).isoformat()
|
||||
|
||||
|
||||
def parse_iso_timestamp(value: str | None) -> dt.datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
with contextlib.suppress(ValueError):
|
||||
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=dt.timezone.utc)
|
||||
return parsed
|
||||
return None
|
||||
|
||||
|
||||
|
||||
|
||||
def query_folder_paths(conn: sqlite3.Connection) -> dict[int, str]:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT Z_PK, ZTITLE2, ZPARENT
|
||||
FROM ZICCLOUDSYNCINGOBJECT
|
||||
WHERE ZTITLE2 IS NOT NULL
|
||||
"""
|
||||
).fetchall()
|
||||
titles = {int(row["Z_PK"]): row["ZTITLE2"] or "Notes" for row in rows}
|
||||
parents = {int(row["Z_PK"]): row["ZPARENT"] for row in rows}
|
||||
cache: dict[int, str] = {}
|
||||
|
||||
def build(folder_id: int, seen: set[int] | None = None) -> str:
|
||||
if folder_id in cache:
|
||||
return cache[folder_id]
|
||||
if seen is None:
|
||||
seen = set()
|
||||
if folder_id in seen:
|
||||
return sanitize_filename(titles.get(folder_id, "Notes"))
|
||||
seen.add(folder_id)
|
||||
title = sanitize_filename(titles.get(folder_id, "Notes"))
|
||||
parent = parents.get(folder_id)
|
||||
if parent is None or int(parent) not in titles:
|
||||
cache[folder_id] = title
|
||||
else:
|
||||
cache[folder_id] = str(Path(build(int(parent), seen)) / title)
|
||||
return cache[folder_id]
|
||||
|
||||
for folder_id in titles:
|
||||
build(folder_id)
|
||||
return cache
|
||||
|
||||
|
||||
def trashed_folder_pks(conn: sqlite3.Connection) -> set[int]:
|
||||
"""Folder primary keys for the trash, plus anything nested inside it.
|
||||
|
||||
``ZFOLDERTYPE == 1`` marks Recently Deleted independently of the UI
|
||||
language, so this beats matching a localized folder name. Notes sitting in
|
||||
the trash keep ``ZMARKEDFORDELETION = 0``, which is why filtering on that
|
||||
column alone still surfaces them.
|
||||
"""
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT Z_PK, ZPARENT, ZFOLDERTYPE
|
||||
FROM ZICCLOUDSYNCINGOBJECT
|
||||
WHERE ZTITLE2 IS NOT NULL
|
||||
"""
|
||||
).fetchall()
|
||||
parents = {int(row["Z_PK"]): row["ZPARENT"] for row in rows}
|
||||
roots = {int(row["Z_PK"]) for row in rows if row["ZFOLDERTYPE"] == 1}
|
||||
trashed = set(roots)
|
||||
for folder_pk in parents:
|
||||
chain: set[int] = set()
|
||||
current: int | None = folder_pk
|
||||
while current is not None and current not in chain:
|
||||
chain.add(current)
|
||||
if current in roots:
|
||||
trashed |= chain
|
||||
break
|
||||
parent = parents.get(current)
|
||||
current = int(parent) if parent is not None else None
|
||||
return trashed
|
||||
|
||||
|
||||
def find_media_file(notes_dir: Path, media_id: str | None, generation: str | None, filename: str | None) -> Path | None:
|
||||
if not media_id:
|
||||
return None
|
||||
accounts_dir = notes_dir / "Accounts"
|
||||
if not accounts_dir.exists():
|
||||
return None
|
||||
for account_dir in accounts_dir.iterdir():
|
||||
media_dir = account_dir / "Media" / media_id
|
||||
if not media_dir.exists():
|
||||
continue
|
||||
candidates: list[Path] = []
|
||||
if generation and filename:
|
||||
candidates.append(media_dir / generation / filename)
|
||||
if filename:
|
||||
candidates.extend(media_dir.glob(f"*/{filename}"))
|
||||
candidates.extend(path for path in media_dir.glob("*/*") if path.is_file())
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
|
||||
def query_embedded_tables(conn: sqlite3.Connection, note_pk: int) -> list[EmbeddedTable]:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT
|
||||
ZIDENTIFIER AS table_identifier,
|
||||
ZMERGEABLEDATA1 AS mergeable_data,
|
||||
ZMERGEABLEDATA AS legacy_mergeable_data,
|
||||
ZSUMMARY AS summary
|
||||
FROM ZICCLOUDSYNCINGOBJECT
|
||||
WHERE ZNOTE = ?
|
||||
AND ZTYPEUTI = 'com.apple.notes.table'
|
||||
ORDER BY Z_PK
|
||||
""",
|
||||
(note_pk,),
|
||||
).fetchall()
|
||||
|
||||
tables: list[EmbeddedTable] = []
|
||||
for row in rows:
|
||||
markdown = ""
|
||||
data = row["mergeable_data"] or row["legacy_mergeable_data"]
|
||||
if data:
|
||||
with contextlib.suppress(ValueError, OSError, UnicodeDecodeError):
|
||||
markdown = extract_markdown_table(data)
|
||||
if not markdown:
|
||||
markdown = summary_table_fallback(row["summary"])
|
||||
if markdown:
|
||||
tables.append(EmbeddedTable(identifier=row["table_identifier"] or "", markdown=markdown))
|
||||
return tables
|
||||
|
||||
|
||||
def query_attachments(conn: sqlite3.Connection, notes_dir: Path, note_pk: int) -> list[Attachment]:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT
|
||||
a.ZIDENTIFIER AS attachment_identifier,
|
||||
a.ZTYPEUTI AS attachment_type,
|
||||
m.ZIDENTIFIER AS media_identifier,
|
||||
m.ZFILENAME AS media_filename,
|
||||
m.ZGENERATION1 AS media_generation
|
||||
FROM ZICCLOUDSYNCINGOBJECT a
|
||||
LEFT JOIN ZICCLOUDSYNCINGOBJECT m ON a.ZMEDIA = m.Z_PK
|
||||
WHERE a.ZNOTE = ?
|
||||
AND a.ZMEDIA IS NOT NULL
|
||||
ORDER BY a.Z_PK
|
||||
""",
|
||||
(note_pk,),
|
||||
).fetchall()
|
||||
|
||||
attachments: list[Attachment] = []
|
||||
for row in rows:
|
||||
attachment_id = row["attachment_identifier"] or row["media_identifier"] or "attachment"
|
||||
source = find_media_file(
|
||||
notes_dir,
|
||||
row["media_identifier"],
|
||||
row["media_generation"],
|
||||
row["media_filename"],
|
||||
)
|
||||
suffix = ""
|
||||
if row["media_filename"]:
|
||||
suffix = Path(row["media_filename"]).suffix
|
||||
output_name = sanitize_filename(f"{attachment_id.lower()}{suffix}", fallback="attachment")
|
||||
attachments.append(
|
||||
Attachment(
|
||||
identifier=attachment_id,
|
||||
type_uti=row["attachment_type"],
|
||||
source_path=source,
|
||||
output_name=output_name,
|
||||
)
|
||||
)
|
||||
return attachments
|
||||
|
||||
|
||||
def attachment_reference_path(note_id: int, output_name: str) -> str:
|
||||
return str(Path("attachments") / str(note_id) / output_name)
|
||||
|
||||
|
||||
def nest_attachments_under_note_id(attachments: list[Attachment], note_id: int) -> list[Attachment]:
|
||||
return [
|
||||
dataclasses.replace(
|
||||
attachment,
|
||||
output_name=attachment_reference_path(note_id, Path(attachment.output_name).name),
|
||||
)
|
||||
for attachment in attachments
|
||||
]
|
||||
|
||||
|
||||
def replace_attachment_placeholders(body: str, attachments: list[Attachment]) -> str:
|
||||
result = body
|
||||
inserted = 0
|
||||
for attachment in attachments:
|
||||
if "\ufffc" not in result:
|
||||
break
|
||||
result = result.replace("\ufffc", attachment.markdown_reference, 1)
|
||||
inserted += 1
|
||||
|
||||
remaining = attachments[inserted:]
|
||||
if remaining:
|
||||
lines = ["", "## Attachments", ""]
|
||||
lines.extend(f"- {attachment.markdown_reference}" for attachment in remaining)
|
||||
result = result.rstrip() + "\n" + "\n".join(lines)
|
||||
return result
|
||||
|
||||
|
||||
def replace_table_placeholders(body: str, tables: list[EmbeddedTable]) -> str:
|
||||
result = body
|
||||
inserted = 0
|
||||
for table in tables:
|
||||
if "\ufffc" not in result:
|
||||
break
|
||||
result = result.replace("\ufffc", "\n\n" + table.markdown + "\n\n", 1)
|
||||
inserted += 1
|
||||
|
||||
remaining = tables[inserted:]
|
||||
if remaining:
|
||||
lines = ["", "## Tables", ""]
|
||||
for table in remaining:
|
||||
lines.extend([table.markdown, ""])
|
||||
result = result.rstrip() + "\n" + "\n".join(lines)
|
||||
return result
|
||||
|
||||
|
||||
def extract_links(body: str) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
links: list[str] = []
|
||||
for match in URL_RE.finditer(body):
|
||||
url = match.group(0).rstrip(".,;:!?)]}")
|
||||
if url and url not in seen:
|
||||
seen.add(url)
|
||||
links.append(url)
|
||||
return links
|
||||
|
||||
|
||||
|
||||
|
||||
def copy_live_database(source: Path) -> Path:
|
||||
temp = tempfile.NamedTemporaryFile(prefix="apple-notes-", suffix=".sqlite", delete=False)
|
||||
temp_path = Path(temp.name)
|
||||
temp.close()
|
||||
|
||||
src = sqlite3.connect(f"file:{source}?mode=ro", uri=True)
|
||||
try:
|
||||
dst = sqlite3.connect(temp_path)
|
||||
try:
|
||||
src.backup(dst)
|
||||
finally:
|
||||
dst.close()
|
||||
finally:
|
||||
src.close()
|
||||
return temp_path
|
||||
|
||||
|
||||
def iter_notes(
|
||||
conn: sqlite3.Connection,
|
||||
notes_dir: Path,
|
||||
include_password_protected: bool = False,
|
||||
limit: int | None = None,
|
||||
include_trashed: bool = False,
|
||||
) -> Iterator[Note]:
|
||||
folder_paths = query_folder_paths(conn)
|
||||
where = [
|
||||
"d.ZDATA IS NOT NULL",
|
||||
"IFNULL(n.ZMARKEDFORDELETION, 0) = 0",
|
||||
]
|
||||
if not include_password_protected:
|
||||
where.append("IFNULL(n.ZISPASSWORDPROTECTED, 0) = 0")
|
||||
if not include_trashed:
|
||||
trashed = trashed_folder_pks(conn)
|
||||
if trashed:
|
||||
placeholders = ", ".join(str(int(pk)) for pk in sorted(trashed))
|
||||
where.append(f"IFNULL(n.ZFOLDER, 0) NOT IN ({placeholders})")
|
||||
sql = f"""
|
||||
SELECT
|
||||
n.Z_PK,
|
||||
n.ZTITLE1,
|
||||
n.ZFOLDER,
|
||||
n.ZIDENTIFIER,
|
||||
n.ZCREATIONDATE,
|
||||
n.ZCREATIONDATE1,
|
||||
n.ZCREATIONDATE2,
|
||||
n.ZCREATIONDATE3,
|
||||
n.ZMODIFICATIONDATE1,
|
||||
d.ZDATA
|
||||
FROM ZICCLOUDSYNCINGOBJECT n
|
||||
JOIN ZICNOTEDATA d ON n.Z_PK = d.ZNOTE
|
||||
WHERE {" AND ".join(where)}
|
||||
ORDER BY n.ZMODIFICATIONDATE1 ASC, n.Z_PK ASC
|
||||
"""
|
||||
if limit is not None:
|
||||
sql += " LIMIT ?"
|
||||
rows = conn.execute(sql, (limit,))
|
||||
else:
|
||||
rows = conn.execute(sql)
|
||||
|
||||
for row in rows:
|
||||
pk = int(row["Z_PK"])
|
||||
tables = query_embedded_tables(conn, pk)
|
||||
body = extract_note_markdown(row["ZDATA"])
|
||||
body = replace_table_placeholders(body, tables)
|
||||
attachments = query_attachments(conn, notes_dir, pk)
|
||||
title = note_title(row["ZTITLE1"], body)
|
||||
attachments = nest_attachments_under_note_id(attachments, pk)
|
||||
body = replace_attachment_placeholders(body, attachments)
|
||||
folder_path = folder_paths.get(row["ZFOLDER"], "Notes") if row["ZFOLDER"] else "Notes"
|
||||
yield Note(
|
||||
pk=pk,
|
||||
title=title,
|
||||
apple_notes_id=row["ZIDENTIFIER"] or str(pk),
|
||||
folder_path=folder_path,
|
||||
created=apple_date(row["ZCREATIONDATE1"])
|
||||
or apple_date(row["ZCREATIONDATE"])
|
||||
or apple_date(row["ZCREATIONDATE2"])
|
||||
or apple_date(row["ZCREATIONDATE3"]),
|
||||
modified=apple_date(row["ZMODIFICATIONDATE1"]),
|
||||
body=body,
|
||||
attachments=attachments,
|
||||
links=extract_links(body),
|
||||
)
|
||||
|
||||
|
||||
|
||||
def note_title(row_title: str | None, body: str) -> str:
|
||||
if row_title and row_title.strip():
|
||||
return row_title.strip()
|
||||
for line in body.splitlines():
|
||||
if line.strip():
|
||||
return line.strip()
|
||||
return "Untitled"
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Filename helpers shared by the store and the exporter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
FORBIDDEN_FILENAME_CHARS = re.compile(r"[/\\:*?\"<>|\x00-\x1f]")
|
||||
|
||||
|
||||
def sanitize_filename(value: str, fallback: str = "Untitled", max_length: int = 180) -> str:
|
||||
value = value.replace("\n", " ").replace("\r", " ").strip()
|
||||
value = FORBIDDEN_FILENAME_CHARS.sub("-", value)
|
||||
value = re.sub(r"\s+", " ", value).strip(" .")
|
||||
if not value:
|
||||
value = fallback
|
||||
if len(value) > max_length:
|
||||
value = value[:max_length].rstrip(" .")
|
||||
return value or fallback
|
||||
|
||||
|
||||
@@ -0,0 +1,541 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
import datetime as dt
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
from apple_notes_cli import export, models, parser, store, utils
|
||||
|
||||
# These tests were written against the original single-file exporter. The code
|
||||
# is now split across modules, so this flattens the public names back into one
|
||||
# namespace and the assertions below stay byte-for-byte identical -- which is
|
||||
# exactly what makes them useful as a regression check on the split.
|
||||
export_apple_notes = types.SimpleNamespace()
|
||||
for _module in (models, parser, utils, store, export):
|
||||
for _name in dir(_module):
|
||||
if not _name.startswith("_"):
|
||||
setattr(export_apple_notes, _name, getattr(_module, _name))
|
||||
|
||||
|
||||
def varint(value: int) -> bytes:
|
||||
out = bytearray()
|
||||
while True:
|
||||
byte = value & 0x7F
|
||||
value >>= 7
|
||||
if value:
|
||||
out.append(byte | 0x80)
|
||||
else:
|
||||
out.append(byte)
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def field(number: int, wire_type: int, payload: bytes | int) -> bytes:
|
||||
key = varint((number << 3) | wire_type)
|
||||
if wire_type == 0:
|
||||
assert isinstance(payload, int)
|
||||
return key + varint(payload)
|
||||
assert isinstance(payload, bytes)
|
||||
return key + varint(len(payload)) + payload
|
||||
|
||||
|
||||
def note_payload(text: str, attribute_runs: list[bytes] | None = None) -> bytes:
|
||||
content = field(2, 2, text.encode("utf-8"))
|
||||
for attribute_run in attribute_runs or []:
|
||||
content += field(5, 2, attribute_run)
|
||||
document = field(1, 0, 0) + field(2, 0, 0) + field(3, 2, content)
|
||||
top = field(1, 0, 0) + field(2, 2, document)
|
||||
return gzip.compress(top)
|
||||
|
||||
|
||||
def checklist_attribute_run(length: int, done: int, indent: int = 0) -> bytes:
|
||||
checklist = field(1, 2, b"uuid") + field(2, 0, done)
|
||||
paragraph_style = field(1, 0, export_apple_notes.STYLE_TYPE_CHECKBOX)
|
||||
if indent:
|
||||
paragraph_style += field(4, 0, indent)
|
||||
paragraph_style += field(5, 2, checklist)
|
||||
return field(1, 0, length) + field(2, 2, paragraph_style)
|
||||
|
||||
|
||||
def paragraph_attribute_run(
|
||||
length: int,
|
||||
style_type: int | None,
|
||||
indent: int = 0,
|
||||
block_quote: bool = False,
|
||||
) -> bytes:
|
||||
paragraph_style = b""
|
||||
if style_type is not None:
|
||||
paragraph_style += field(1, 0, style_type)
|
||||
if indent:
|
||||
paragraph_style += field(4, 0, indent)
|
||||
if block_quote:
|
||||
paragraph_style += field(8, 0, export_apple_notes.STYLE_TYPE_BLOCK_QUOTE)
|
||||
return field(1, 0, length) + field(2, 2, paragraph_style)
|
||||
|
||||
|
||||
def inline_attribute_run(
|
||||
length: int,
|
||||
*,
|
||||
font_weight: int | None = None,
|
||||
underlined: int | None = None,
|
||||
strikethrough: int | None = None,
|
||||
superscript: int | None = None,
|
||||
link: str | None = None,
|
||||
) -> bytes:
|
||||
payload = field(1, 0, length)
|
||||
if font_weight is not None:
|
||||
payload += field(5, 0, font_weight)
|
||||
if underlined is not None:
|
||||
payload += field(6, 0, underlined)
|
||||
if strikethrough is not None:
|
||||
payload += field(7, 0, strikethrough)
|
||||
if superscript is not None:
|
||||
payload += field(8, 0, superscript)
|
||||
if link is not None:
|
||||
payload += field(9, 2, link.encode("utf-8"))
|
||||
return payload
|
||||
|
||||
|
||||
class AppleNotesExportTests(unittest.TestCase):
|
||||
def test_extract_note_text_from_gzipped_notestore_proto(self):
|
||||
payload = note_payload("Title\n\nbody line")
|
||||
|
||||
self.assertEqual(export_apple_notes.extract_note_text(payload), "Title\n\nbody line")
|
||||
|
||||
def test_extract_note_markdown_renders_checklists(self):
|
||||
payload = note_payload(
|
||||
"\nTask\nDone\nNested\n",
|
||||
[
|
||||
checklist_attribute_run(1, 0),
|
||||
checklist_attribute_run(5, 0),
|
||||
checklist_attribute_run(5, 1),
|
||||
checklist_attribute_run(7, 0, indent=1),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"- [ ] Task\n- [x] Done\n - [ ] Nested",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_renders_block_styles(self):
|
||||
payload = note_payload(
|
||||
"\nTitle\nHeading\nSub\nCode\nQuote\nBullet\nOne\nTwo\n",
|
||||
[
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_TITLE),
|
||||
paragraph_attribute_run(6, export_apple_notes.STYLE_TYPE_TITLE),
|
||||
paragraph_attribute_run(8, export_apple_notes.STYLE_TYPE_HEADING),
|
||||
paragraph_attribute_run(4, export_apple_notes.STYLE_TYPE_SUBHEADING),
|
||||
paragraph_attribute_run(5, export_apple_notes.STYLE_TYPE_MONOSPACED),
|
||||
paragraph_attribute_run(6, None, indent=1, block_quote=True),
|
||||
paragraph_attribute_run(7, export_apple_notes.STYLE_TYPE_DOTTED_LIST),
|
||||
paragraph_attribute_run(4, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(4, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"### Title\n#### Heading\n##### Sub\n Code\n> Quote\n- Bullet\n1. One\n2. Two",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_does_not_wrap_trailing_space_in_bold(self):
|
||||
payload = note_payload(
|
||||
"产量: 约占全球咖啡产量",
|
||||
[
|
||||
inline_attribute_run(4, font_weight=export_apple_notes.FONT_TYPE_BOLD),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"**产量:** 约占全球咖啡产量",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_adds_space_after_bold_label(self):
|
||||
payload = note_payload(
|
||||
"产量:约占全球咖啡产量",
|
||||
[
|
||||
inline_attribute_run(3, font_weight=export_apple_notes.FONT_TYPE_BOLD),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"**产量:** 约占全球咖啡产量",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_preserves_tab_indented_plain_blocks(self):
|
||||
payload = note_payload("RC Account\n\t0:N WABA\n\t\t1:N Phone")
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"```\nRC Account\n\t0:N WABA\n\t\t1:N Phone\n```",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_preserves_arrow_flow_blocks(self):
|
||||
payload = note_payload("Meta\n↓\nTIG -> Verification\n↓\nNECA")
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"```\nMeta\n↓\nTIG -> Verification\n↓\nNECA\n```",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_preserves_plain_line_breaks(self):
|
||||
payload = note_payload("SW: https://example.com\nBUS: test/test\nLogInsight: http://log")
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"SW: https://example.com \nBUS: test/test \nLogInsight: http://log",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_does_not_fence_mixed_markdown_blocks(self):
|
||||
payload = note_payload("Heading\n[https://example.com](https://example.com)\n\tIndented")
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"Heading \n[https://example.com](https://example.com) \n Indented",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_merges_leading_plain_style_fragment(self):
|
||||
payload = note_payload(
|
||||
"主要产地: 巴西",
|
||||
[
|
||||
paragraph_attribute_run(1, None),
|
||||
inline_attribute_run(4, font_weight=export_apple_notes.FONT_TYPE_BOLD),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"**主要产地:** 巴西",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_uses_dominant_block_style(self):
|
||||
payload = note_payload(
|
||||
"产量:约",
|
||||
[
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_HEADING),
|
||||
paragraph_attribute_run(4, export_apple_notes.STYLE_TYPE_DOTTED_LIST),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(export_apple_notes.extract_note_markdown(payload), "- 产量:约")
|
||||
|
||||
def test_extract_note_markdown_preserves_numbering_across_nested_bullets(self):
|
||||
payload = note_payload(
|
||||
"南美洲\n巴西豆\n中美洲\n哥斯达\n非洲\n埃塞",
|
||||
[
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_HEADING),
|
||||
paragraph_attribute_run(3, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(3, export_apple_notes.STYLE_TYPE_DOTTED_LIST, indent=1),
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_DOTTED_LIST, indent=1),
|
||||
paragraph_attribute_run(3, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(2, export_apple_notes.STYLE_TYPE_DOTTED_LIST, indent=1),
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_DOTTED_LIST, indent=1),
|
||||
paragraph_attribute_run(2, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(1, export_apple_notes.STYLE_TYPE_NUMBERED_LIST),
|
||||
paragraph_attribute_run(2, export_apple_notes.STYLE_TYPE_DOTTED_LIST, indent=1),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"1. 南美洲\n - 巴西豆\n2. 中美洲\n - 哥斯达\n3. 非洲\n - 埃塞",
|
||||
)
|
||||
|
||||
def test_extract_note_markdown_renders_inline_styles(self):
|
||||
payload = note_payload(
|
||||
"BoldItalicBothUnderStrikeSupLink",
|
||||
[
|
||||
inline_attribute_run(4, font_weight=export_apple_notes.FONT_TYPE_BOLD),
|
||||
inline_attribute_run(6, font_weight=export_apple_notes.FONT_TYPE_ITALIC),
|
||||
inline_attribute_run(4, font_weight=export_apple_notes.FONT_TYPE_BOLD_ITALIC),
|
||||
inline_attribute_run(5, underlined=1),
|
||||
inline_attribute_run(6, strikethrough=1),
|
||||
inline_attribute_run(3, superscript=1),
|
||||
inline_attribute_run(4, link="https://example.com"),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_note_markdown(payload),
|
||||
"**Bold***Italic****Both***<u>Under</u>~~Strike~~<sup>Sup</sup>[Link](https://example.com)",
|
||||
)
|
||||
|
||||
def test_markdown_table_escapes_cells(self):
|
||||
table = export_apple_notes.markdown_table([["A|B", "C"], ["one\ntwo", "three"]])
|
||||
|
||||
self.assertEqual(table, "| A\\|B | C |\n| --- | --- |\n| one<br>two | three |")
|
||||
|
||||
def test_replace_table_placeholders_in_order(self):
|
||||
tables = [
|
||||
export_apple_notes.EmbeddedTable("T1", "| A |\n| --- |\n| 1 |"),
|
||||
export_apple_notes.EmbeddedTable("T2", "| B |\n| --- |\n| 2 |"),
|
||||
]
|
||||
|
||||
body = export_apple_notes.replace_table_placeholders("before\n\ufffc\nmiddle\n\ufffc\nafter", tables)
|
||||
|
||||
self.assertIn("| A |\n| --- |\n| 1 |", body)
|
||||
self.assertIn("| B |\n| --- |\n| 2 |", body)
|
||||
|
||||
def test_extract_links_matches_urls_before_chinese_punctuation(self):
|
||||
body = "https://wiki.example.com/a 账号\nhttps://git.example.com/x?ref_type=heads,/ai"
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.extract_links(body),
|
||||
[
|
||||
"https://wiki.example.com/a",
|
||||
"https://git.example.com/x?ref_type=heads",
|
||||
],
|
||||
)
|
||||
|
||||
def test_sanitize_filename_matches_alto_style_for_url_like_titles(self):
|
||||
title = "https://platform.example.com/external-channels"
|
||||
|
||||
self.assertEqual(
|
||||
export_apple_notes.sanitize_filename(title),
|
||||
"https---platform.example.com-external-channels",
|
||||
)
|
||||
|
||||
def test_build_markdown_frontmatter_shape(self):
|
||||
note = export_apple_notes.Note(
|
||||
pk=42,
|
||||
title="Example",
|
||||
apple_notes_id="ABC-123",
|
||||
folder_path="Notes",
|
||||
created="2026-06-10T00:00:00+00:00",
|
||||
modified="2026-06-10T01:00:00+00:00",
|
||||
body="Example\n\nbody",
|
||||
attachments=[],
|
||||
links=["https://example.com"],
|
||||
)
|
||||
|
||||
markdown = export_apple_notes.build_markdown(note)
|
||||
|
||||
self.assertTrue(markdown.startswith("---\n"))
|
||||
self.assertIn("title: \"Example\"\n", markdown)
|
||||
self.assertIn("id: 42\n", markdown)
|
||||
self.assertIn("apple_notes_id: \"ABC-123\"\n", markdown)
|
||||
self.assertIn("attachments: []\n", markdown)
|
||||
self.assertIn("source: notes://showNote?identifier=ABC-123\n", markdown)
|
||||
self.assertTrue(markdown.endswith("Example\n\nbody\n"))
|
||||
|
||||
def test_note_signature_changes_when_body_changes(self):
|
||||
note = export_apple_notes.Note(
|
||||
pk=1,
|
||||
title="Example",
|
||||
apple_notes_id="A",
|
||||
folder_path="Notes",
|
||||
created=None,
|
||||
modified=None,
|
||||
body="old",
|
||||
attachments=[],
|
||||
links=[],
|
||||
)
|
||||
changed = export_apple_notes.dataclasses.replace(note, body="new")
|
||||
|
||||
self.assertNotEqual(export_apple_notes.note_signature(note), export_apple_notes.note_signature(changed))
|
||||
|
||||
def test_manifest_roundtrip_and_safe_delete(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
note_file = root / "Notes" / "Example-1.md"
|
||||
attachment_file = root / "Notes" / "image.png"
|
||||
outside_file = root.parent / "outside-apple-notes-export-test.txt"
|
||||
note_file.parent.mkdir()
|
||||
note_file.write_text("note", encoding="utf-8")
|
||||
attachment_file.write_text("attachment", encoding="utf-8")
|
||||
outside_file.write_text("outside", encoding="utf-8")
|
||||
self.addCleanup(lambda: outside_file.unlink(missing_ok=True))
|
||||
|
||||
entry = {
|
||||
"path": "Notes/Example-1.md",
|
||||
"attachments": ["Notes/image.png", "../outside-apple-notes-export-test.txt"],
|
||||
}
|
||||
|
||||
deleted = export_apple_notes.remove_manifest_entry_paths(root, entry)
|
||||
|
||||
self.assertEqual(deleted, 2)
|
||||
self.assertFalse(note_file.exists())
|
||||
self.assertFalse(attachment_file.exists())
|
||||
self.assertTrue(outside_file.exists())
|
||||
|
||||
def test_write_and_load_manifest(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
export_apple_notes.write_manifest(root, {"1": {"path": "Notes/Example-1.md"}})
|
||||
|
||||
manifest = export_apple_notes.load_manifest(root)
|
||||
|
||||
self.assertEqual(manifest["version"], export_apple_notes.MANIFEST_VERSION)
|
||||
self.assertEqual(manifest["notes"]["1"]["path"], "Notes/Example-1.md")
|
||||
|
||||
def test_prune_empty_dirs_removes_ds_store_only_directories(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
old_dir = root / "Old Folder" / "Nested"
|
||||
old_dir.mkdir(parents=True)
|
||||
(old_dir / ".DS_Store").write_text("metadata", encoding="utf-8")
|
||||
|
||||
export_apple_notes.prune_all_empty_dirs(root)
|
||||
|
||||
self.assertFalse((root / "Old Folder").exists())
|
||||
|
||||
def test_prune_empty_dirs_keeps_directories_with_regular_files(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
kept_dir = root / "Old Folder"
|
||||
kept_dir.mkdir()
|
||||
(kept_dir / ".DS_Store").write_text("metadata", encoding="utf-8")
|
||||
(kept_dir / "manual.md").write_text("manual", encoding="utf-8")
|
||||
|
||||
export_apple_notes.prune_all_empty_dirs(root)
|
||||
|
||||
self.assertTrue(kept_dir.exists())
|
||||
self.assertTrue((kept_dir / "manual.md").exists())
|
||||
|
||||
def test_remove_orphaned_export_files_deletes_stale_note_and_attachment(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
old_dir = root / "10 - 工作" / "12 - RC"
|
||||
old_dir.mkdir(parents=True)
|
||||
stale_note = old_dir / "Example-1.md"
|
||||
stale_attachment = old_dir / "image.png"
|
||||
stale_note.write_text(
|
||||
"\n".join([
|
||||
"---",
|
||||
'title: "Example"',
|
||||
"id: 1",
|
||||
'apple_notes_id: "A"',
|
||||
'folder: "10 - 工作/12 - RC"',
|
||||
"attachments:",
|
||||
' - filename: "image.png"',
|
||||
' identifier: "ATTACHMENT"',
|
||||
' type: "public.png"',
|
||||
"links: []",
|
||||
"source: notes://showNote?identifier=A",
|
||||
"---",
|
||||
"",
|
||||
"body",
|
||||
"",
|
||||
]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
stale_attachment.write_text("attachment", encoding="utf-8")
|
||||
|
||||
deleted = export_apple_notes.remove_orphaned_export_files(root, set())
|
||||
|
||||
self.assertEqual(deleted, 2)
|
||||
self.assertFalse((root / "10 - 工作").exists())
|
||||
|
||||
def test_remove_orphaned_export_files_keeps_manual_markdown(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
old_dir = root / "10 - 工作"
|
||||
old_dir.mkdir()
|
||||
manual = old_dir / "manual.md"
|
||||
manual.write_text("# manual\n", encoding="utf-8")
|
||||
|
||||
deleted = export_apple_notes.remove_orphaned_export_files(root, set())
|
||||
|
||||
self.assertEqual(deleted, 0)
|
||||
self.assertTrue(manual.exists())
|
||||
|
||||
def test_write_note_applies_modified_time_to_markdown(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
modified = "2026-06-10T01:00:00+00:00"
|
||||
note = export_apple_notes.Note(
|
||||
pk=7,
|
||||
title="Timestamped",
|
||||
apple_notes_id="T",
|
||||
folder_path="Notes",
|
||||
created="2026-06-09T01:00:00+00:00",
|
||||
modified=modified,
|
||||
body="body",
|
||||
attachments=[],
|
||||
links=[],
|
||||
)
|
||||
|
||||
result = export_apple_notes.write_note(root, note, copy_attachments=True)
|
||||
|
||||
expected = dt.datetime.fromisoformat(modified).timestamp()
|
||||
self.assertAlmostEqual(result.note_path.stat().st_mtime, expected, delta=1.0)
|
||||
|
||||
def test_manifest_entry_timestamps_match_detects_stale_mtime(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
note_path = root / "Notes" / "Example-1.md"
|
||||
note_path.parent.mkdir()
|
||||
note_path.write_text("note", encoding="utf-8")
|
||||
modified = "2026-06-10T01:00:00+00:00"
|
||||
|
||||
self.assertFalse(
|
||||
export_apple_notes.manifest_entry_timestamps_match(
|
||||
root,
|
||||
{"path": "Notes/Example-1.md", "attachments": [], "modified": modified},
|
||||
)
|
||||
)
|
||||
|
||||
export_apple_notes.apply_file_timestamps(note_path, created=None, modified=modified)
|
||||
|
||||
self.assertTrue(
|
||||
export_apple_notes.manifest_entry_timestamps_match(
|
||||
root,
|
||||
{"path": "Notes/Example-1.md", "attachments": [], "modified": modified},
|
||||
)
|
||||
)
|
||||
|
||||
def test_attachment_references_are_nested_under_note_id(self):
|
||||
attachment = export_apple_notes.Attachment(
|
||||
identifier="ATTACHMENT",
|
||||
type_uti="public.png",
|
||||
source_path=None,
|
||||
output_name="image.png",
|
||||
)
|
||||
|
||||
nested = export_apple_notes.nest_attachments_under_note_id([attachment], 1)
|
||||
body = export_apple_notes.replace_attachment_placeholders("before\n\ufffc\nafter", nested)
|
||||
|
||||
self.assertEqual(nested[0].output_name, "attachments/1/image.png")
|
||||
self.assertIn("", body)
|
||||
|
||||
def test_write_note_copies_attachments_to_same_named_folder(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
source = root / "source.png"
|
||||
source.write_text("attachment", encoding="utf-8")
|
||||
attachment = export_apple_notes.Attachment(
|
||||
identifier="ATTACHMENT",
|
||||
type_uti="public.png",
|
||||
source_path=source,
|
||||
output_name="attachments/7/image.png",
|
||||
)
|
||||
note = export_apple_notes.Note(
|
||||
pk=7,
|
||||
title="Timestamped",
|
||||
apple_notes_id="T",
|
||||
folder_path="Notes",
|
||||
created=None,
|
||||
modified=None,
|
||||
body="",
|
||||
attachments=[attachment],
|
||||
links=[],
|
||||
)
|
||||
|
||||
result = export_apple_notes.write_note(root, note, copy_attachments=True)
|
||||
markdown = result.note_path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertTrue((root / "Notes" / "Timestamped-7.md").exists())
|
||||
self.assertTrue((root / "Notes" / "attachments" / "7" / "image.png").exists())
|
||||
self.assertIn('filename: "attachments/7/image.png"', markdown)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,251 @@
|
||||
"""Tests for the query layer, built on a synthetic Core Data store."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import gzip
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from apple_notes_cli import search as search_mod
|
||||
from apple_notes_cli.store import trashed_folder_pks
|
||||
|
||||
APPLE_EPOCH_OFFSET = 978307200.0
|
||||
|
||||
|
||||
def varint(value: int) -> bytes:
|
||||
out = bytearray()
|
||||
while True:
|
||||
byte = value & 0x7F
|
||||
value >>= 7
|
||||
if value:
|
||||
out.append(byte | 0x80)
|
||||
else:
|
||||
out.append(byte)
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def field(number: int, wire_type: int, payload: bytes | int) -> bytes:
|
||||
key = varint((number << 3) | wire_type)
|
||||
if wire_type == 0:
|
||||
return key + varint(payload)
|
||||
return key + varint(len(payload)) + payload
|
||||
|
||||
|
||||
def note_payload(text: str) -> bytes:
|
||||
content = field(2, 2, text.encode("utf-8"))
|
||||
document = field(1, 0, 0) + field(2, 0, 0) + field(3, 2, content)
|
||||
return gzip.compress(field(1, 0, 0) + field(2, 2, document))
|
||||
|
||||
|
||||
def apple_time(moment: dt.datetime) -> float:
|
||||
return moment.timestamp() - APPLE_EPOCH_OFFSET
|
||||
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE ZICCLOUDSYNCINGOBJECT (
|
||||
Z_PK INTEGER PRIMARY KEY,
|
||||
ZTITLE1 VARCHAR,
|
||||
ZTITLE2 VARCHAR,
|
||||
ZFOLDER INTEGER,
|
||||
ZPARENT INTEGER,
|
||||
ZFOLDERTYPE INTEGER,
|
||||
ZMARKEDFORDELETION INTEGER,
|
||||
ZISPASSWORDPROTECTED INTEGER,
|
||||
ZIDENTIFIER VARCHAR,
|
||||
ZCREATIONDATE TIMESTAMP,
|
||||
ZCREATIONDATE1 TIMESTAMP,
|
||||
ZCREATIONDATE2 TIMESTAMP,
|
||||
ZCREATIONDATE3 TIMESTAMP,
|
||||
ZMODIFICATIONDATE1 TIMESTAMP,
|
||||
ZNOTEDATA INTEGER
|
||||
);
|
||||
CREATE TABLE ZICNOTEDATA (
|
||||
Z_PK INTEGER PRIMARY KEY,
|
||||
ZNOTE INTEGER,
|
||||
ZDATA BLOB
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
class QueryLayerTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.db_path = Path(self.tmp.name) / "NoteStore.sqlite"
|
||||
self.conn = sqlite3.connect(self.db_path)
|
||||
self.conn.row_factory = sqlite3.Row
|
||||
self.conn.executescript(SCHEMA)
|
||||
|
||||
now = dt.datetime.now(dt.timezone.utc)
|
||||
# Folders: 10 = Work, 11 = Work/Deep, 20 = trash, 21 = a folder nested
|
||||
# inside the trash (Notes does not create these itself, but the parent
|
||||
# walk has to survive one).
|
||||
self._folder(10, "Work", None, 0)
|
||||
self._folder(11, "Deep", 10, 0)
|
||||
self._folder(20, "Recently Deleted", None, 1)
|
||||
self._folder(21, "Nested", 20, 0)
|
||||
|
||||
self._note(1, "Alpha", 10, "telemetry is here", now - dt.timedelta(hours=1))
|
||||
self._note(2, "Beta", 11, "telemetry telemetry twice", now - dt.timedelta(days=5))
|
||||
self._note(3, "Gamma", 10, "nothing to see", now - dt.timedelta(minutes=10))
|
||||
self._note(4, "Trashed", 20, "telemetry in the bin", now - dt.timedelta(minutes=5))
|
||||
self._note(5, "NestedTrash", 21, "telemetry nested", now - dt.timedelta(minutes=5))
|
||||
self._note(6, "Locked", 10, "telemetry locked", now, protected=1)
|
||||
self._note(7, "Deleted", 10, "telemetry deleted", now, marked=1)
|
||||
self.conn.commit()
|
||||
|
||||
def _folder(self, pk: int, title: str, parent: int | None, folder_type: int) -> None:
|
||||
self.conn.execute(
|
||||
"INSERT INTO ZICCLOUDSYNCINGOBJECT (Z_PK, ZTITLE2, ZPARENT, ZFOLDERTYPE)"
|
||||
" VALUES (?, ?, ?, ?)",
|
||||
(pk, title, parent, folder_type),
|
||||
)
|
||||
|
||||
def _note(
|
||||
self,
|
||||
pk: int,
|
||||
title: str,
|
||||
folder: int,
|
||||
body: str,
|
||||
modified: dt.datetime,
|
||||
*,
|
||||
protected: int = 0,
|
||||
marked: int = 0,
|
||||
) -> None:
|
||||
note_pk = 1000 + pk
|
||||
self.conn.execute(
|
||||
"INSERT INTO ZICCLOUDSYNCINGOBJECT (Z_PK, ZTITLE1, ZFOLDER, ZMARKEDFORDELETION,"
|
||||
" ZISPASSWORDPROTECTED, ZIDENTIFIER, ZCREATIONDATE1, ZMODIFICATIONDATE1, ZNOTEDATA)"
|
||||
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
note_pk,
|
||||
title,
|
||||
folder,
|
||||
marked,
|
||||
protected,
|
||||
f"uuid-{pk}",
|
||||
apple_time(modified),
|
||||
apple_time(modified),
|
||||
pk,
|
||||
),
|
||||
)
|
||||
self.conn.execute(
|
||||
"INSERT INTO ZICNOTEDATA (Z_PK, ZNOTE, ZDATA) VALUES (?, ?, ?)",
|
||||
(pk, note_pk, note_payload(body)),
|
||||
)
|
||||
|
||||
# --- trash handling -------------------------------------------------
|
||||
|
||||
def test_trashed_folder_pks_includes_nested_folders(self):
|
||||
self.assertEqual(trashed_folder_pks(self.conn), {20, 21})
|
||||
|
||||
def test_search_excludes_trashed_notes_by_default(self):
|
||||
titles = {hit.note.title for hit in search_mod.search(self.conn, "telemetry")}
|
||||
self.assertEqual(titles, {"Alpha", "Beta"})
|
||||
|
||||
def test_search_can_opt_into_trashed_notes(self):
|
||||
titles = {
|
||||
hit.note.title
|
||||
for hit in search_mod.search(self.conn, "telemetry", include_trashed=True)
|
||||
}
|
||||
self.assertEqual(titles, {"Alpha", "Beta", "Trashed", "NestedTrash"})
|
||||
|
||||
def test_recent_excludes_trashed_notes(self):
|
||||
self.assertEqual(
|
||||
[n.title for n in search_mod.recent(self.conn)], ["Gamma", "Alpha", "Beta"]
|
||||
)
|
||||
|
||||
def test_locked_and_deleted_notes_are_excluded(self):
|
||||
titles = {
|
||||
hit.note.title
|
||||
for hit in search_mod.search(self.conn, "telemetry", include_trashed=True)
|
||||
}
|
||||
self.assertNotIn("Locked", titles)
|
||||
self.assertNotIn("Deleted", titles)
|
||||
|
||||
# --- search behaviour -----------------------------------------------
|
||||
|
||||
def test_hits_rank_by_match_count(self):
|
||||
hits = search_mod.search(self.conn, "telemetry")
|
||||
self.assertEqual([h.note.title for h in hits], ["Beta", "Alpha"])
|
||||
self.assertEqual([h.match_count for h in hits], [2, 1])
|
||||
|
||||
def test_title_is_searched_unless_body_only(self):
|
||||
self.assertEqual(len(search_mod.search(self.conn, "Gamma")), 1)
|
||||
self.assertEqual(
|
||||
len(search_mod.search(self.conn, "Gamma", search_titles=False)), 0
|
||||
)
|
||||
|
||||
def test_regex_search(self):
|
||||
hits = search_mod.search(self.conn, r"tele\w+try", regex=True)
|
||||
self.assertEqual({h.note.title for h in hits}, {"Alpha", "Beta"})
|
||||
|
||||
def test_plain_query_is_not_treated_as_regex(self):
|
||||
self.assertEqual(search_mod.search(self.conn, "tele.metry"), [])
|
||||
|
||||
def test_case_sensitivity(self):
|
||||
self.assertEqual(len(search_mod.search(self.conn, "TELEMETRY")), 2)
|
||||
self.assertEqual(
|
||||
len(search_mod.search(self.conn, "TELEMETRY", case_sensitive=True)), 0
|
||||
)
|
||||
|
||||
def test_since_filters_older_notes(self):
|
||||
since = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=1)
|
||||
titles = {h.note.title for h in search_mod.search(self.conn, "telemetry", since=since)}
|
||||
self.assertEqual(titles, {"Alpha"})
|
||||
|
||||
def test_folder_filter_matches_nested_paths(self):
|
||||
hits = search_mod.search(self.conn, "telemetry", folder="Work/Deep")
|
||||
self.assertEqual([h.note.title for h in hits], ["Beta"])
|
||||
|
||||
def test_snippets_carry_context_and_respect_cap(self):
|
||||
hit = search_mod.search(self.conn, "telemetry", max_snippets=1)[0]
|
||||
self.assertEqual(len(hit.snippets), 1)
|
||||
self.assertIn("telemetry", hit.snippets[0])
|
||||
|
||||
# --- folders and show ------------------------------------------------
|
||||
|
||||
def test_folders_reports_counts_without_trash(self):
|
||||
self.assertEqual(
|
||||
search_mod.folders(self.conn), [("Work", 3), ("Work/Deep", 1)]
|
||||
)
|
||||
|
||||
def test_show_returns_body_and_metadata(self):
|
||||
note, body = search_mod.show(self.conn, 1001)
|
||||
self.assertEqual(note.title, "Alpha")
|
||||
self.assertEqual(note.folder, "Work")
|
||||
self.assertIn("telemetry", body)
|
||||
|
||||
def test_show_raises_for_unknown_id(self):
|
||||
with self.assertRaises(KeyError):
|
||||
search_mod.show(self.conn, 999999)
|
||||
|
||||
|
||||
class ParseSinceTests(unittest.TestCase):
|
||||
def test_relative_offsets(self):
|
||||
now = dt.datetime.now(dt.timezone.utc)
|
||||
self.assertAlmostEqual(
|
||||
(now - search_mod.parse_since("2d")).total_seconds(), 172800, delta=5
|
||||
)
|
||||
self.assertAlmostEqual(
|
||||
(now - search_mod.parse_since("30m")).total_seconds(), 1800, delta=5
|
||||
)
|
||||
self.assertAlmostEqual(
|
||||
(now - search_mod.parse_since("1w")).total_seconds(), 604800, delta=5
|
||||
)
|
||||
|
||||
def test_iso_date_is_localized(self):
|
||||
parsed = search_mod.parse_since("2026-09-01")
|
||||
self.assertEqual((parsed.year, parsed.month, parsed.day), (2026, 9, 1))
|
||||
self.assertIsNotNone(parsed.tzinfo)
|
||||
|
||||
def test_invalid_value_raises_value_error(self):
|
||||
with self.assertRaises(ValueError):
|
||||
search_mod.parse_since("last tuesday")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user