Lore
A data file that explains itself. Parquet that carries its own meaning, privacy flags and an AI-readable brief. In development. Python, Apache-2.0. Runs on…
What it is
A Lore file is a normal Parquet file that also carries, inside it, what the data means: each column's meaning and role, which columns are personal, where the data came from, what it is like, how it was tested, and a short brief an AI can read before touching the data.
Every tool that reads Parquet reads a Lore file unchanged. Tools that know Lore also read its meaning.
Why
When data leaves the system it was made in, its meaning is lost. AI agents then either guess, or read real rows to find out, which can leak personal data. Lore keeps the meaning with the data, with privacy rules a checker can test.
See it in one query
With plain DuckDB, no extension:
SELECT decode(value) AS brief
FROM parquet_kv_metadata('examples/tpch-region.lore')
WHERE decode(key) = 'lore.brief';
And the data is still just Parquet: SELECT * FROM read_parquet('examples/tpch-region.lore').
What it includes
- The specification, with a JSON Schema for the manifest.
- A Python package to write, read and check Lore files (it needs only pyarrow), and a
lorecommand:lore inspect,lore brief,lore check. - Personal columns masked for you when writing, and checked when reading.
- Conformance files: good and deliberately broken examples, with the verdict each must get.
Status
Draft 0.1, open for comment. Extension .lore (or .lore.parquet where a system needs .parquet). The specification and package are not published yet.