9:41

Lakegres: Vector search on existing Open/Hudi/Iceberg tables

, . . .

, . , , , , . . .

. , , , . 100 1,000 . , . . .

, , , . , .

. , .

. , , .

. ( , ), . 1.1 × (4 × + 8 × ) . 247 1,024 . 16, 1.15 , 46 10 .

, . , . . .

2021. 350 6 . 500- , . . .

. . .

, , . . . .

, . , . . , .

(). . , .

; .

, . , .

. . , , . . .

. . .

. : , . . , .

, . . 1 8 , 4 . 2, .

, . , 1,000 4,000 , 512 . , . .

Index storage layout: original vectors remain in the base table; metadata contains generation state, centroids, cluster segments, and posting blocks with six packed array sections.
The index is derived metadata; creating or dropping it leaves the base data files unchanged. Each posting block is one metadata record containing six arrays, decoded selectively after the block is fetched.

, , , , . . . , .

. . . , . .

Vector query path: resolve snapshot and index freshness, select nearest clusters, read blocks and deltas, prune and refine candidates, validate record keys, and optionally re-rank using full vectors.
Candidate discovery uses the compact index. Optional exact re-ranking reads the full vectors for the shortlist from the same table snapshot.

, . . . . . .

. , . , . . . .

, . . . , .

. , . . . , . . .

. 247,154,006 , 247 , 1,024- 3. 1 3.

. , . 320 768 , 240 1,024 .

, . 200 10 , . .

Lakegres versus storage-optimized Databricks AI Search latency on 247 million vectors. Warm p50: 63.5 versus 655 ms; p90: 67.9 versus 876 ms; p99: 75.3 versus 1,209 ms. Cold p50: 536.2 versus 8,242 ms.
Lakegres: five executors plus one driver. Lower latency is better. Warm and cold results use separate scales, both starting at zero.

, 63.5 50, 67.9 90 75.3 99. 655 , 876 1,209 , . 50 536.2 8,242 . @10 0.95 0.952 .

. , . .

. , .

8.4 8.4 . , . ’ . . , 10- , 70,000 ; .

10 15 . $5.79/ , ; $7.68/. . .

. . , . , . . .

, 18 3 10 . , . .

. , , .

. .

, . . , .

. 20 25 , , . , , . . .

. . , , .

: , , . , , .

. .

, . , .

, , , .

@, @, @, @, @0405, @13, @, @, @ @ , .