Xiangpeng's blog

A system programmer’s guide to LLM inference

Mon, 08 Jun 2026 00:00:00 GMT

Acknowledgments

Let’s take a moment to thank my PhD sponsors: InfluxData, Bauplan, SpiralDB, and the taxpayers of the State of Wisconsin and the federal government.

LLMs have become so important that I (probably you as well) want to understand them better, and the best way to learn is to build one.

In this blog post, I’ll share what I’ve learned about LLM inference, from the perspective of a systems programmer.

I pick the model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf so that it runs on most machines but is also complex enough to count as a “modern LLM” ¹. This is the only model to support here. By the end, we’ll be able to run prefill at 100 tokens/s and decode at 15 tokens/s, not so bad on a CPU-only machine.

¹ The model was released in April 2026.

This blog covers most of the important parts of a local LLM inference engine:

The LLM architecture
Quantization
Fast matrix multiplication
KV cache

But does not cover:

GPU acceleration, this is a pure CPU inference engine. I’ll probably do a GPU follow up later.
No MTP (speculative decoding), because we are not GPU yet.
Things that are {vendor-specific | closed-source}, e.g., CUDA, are not covered

How to read the name? Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

Qwen (pronounced /kwɛn/, like “when” with a “kw”) is the model family name; it comes from Alibaba, a not so popular Chinese company (in tech world).

3.6 is the version number; previous versions were 3.5 and 3, so the family has been around for a while.

35B is the model size. 35 Billion is the number of parameters; if each parameter were 8 bits (fp8 or int8), the model would be about 35 GB.

A3B means the model activates 3 Billion parameters to generate a token. It also implies the model is an MoE (Mixture of Experts) model — unlike dense models where all parameters are activated for every token.

UD-Q4_K_M means the model is quantized to 4 bits (Q4), and K_M is the quantization scheme (there are many ways to quantize a model). UD stands for Unsloth Dynamic quantization; Unsloth is a company that produces these quantized variants.

gguf is the model file format. Unlike safetensors, gguf is a self-contained format: a single file holds everything you need to run the model. In practice, it’s just metadata plus the model weights.

What’s inside the file?

The file format is pretty boring (as intended): some metadata describing the model, followed by pairs of tensor info and tensor data, as shown in Figure 1.

Figure 1: The GGUF file layout

The tensor info is stored at the beginning of the file, and points to the actual tensor data through the offset and len fields:

struct TensorInfo {
    name: String, // e.g., `blk.3.attn_norm_weight`
    shape: Vec<u64>,
    dtype: u32,
    offset: u64,
    len: u64,
}

Note that dtype is a per-tensor field, which means two tensors in the same file may use different data types (i.e., different quantization schemes). This lets us store important tensors (e.g., the embedding weights) with more bits for higher precision, while quantizing the rest more aggressively. As you might guess, per-tensor quantization is a fine art.

A closer look at the quantization schemes shows that the model has 6 different types of tensors, more than half of the bytes are quantized to Q4_K:

  BF16      2 tensors         1.00 MB
  F32     368 tensors        99.78 MB
  Q4_K     82 tensors     11808.00 MB
  Q5_K     38 tensors      6688.00 MB
  Q6_K      4 tensors      1027.85 MB
  Q8_0    259 tensors      1978.38 MB
  ------ ---- ---------    ----------
  total   753 tensors     21603.01 MB

The metadata also encodes the model architecture: the number of layers, the model family, and so on.

The model architecture

The model belongs to the Qwen3-Next family; for background on the architecture lineage, see Sebastian Raschka’s “Big LLM Architecture Comparison”.

Figure 2 is a simplified view of the architecture and its weight distribution. For every input token, the model first converts it into an embedding vector (with dimension 2048 for this model), then runs it through several layers of computation to produce the final output.

Qwen3.6 is not a typical transformer: it mixes the so-called DeltaNet layers with conventional Attention layers at a 3:1 ratio. We will get to the details later; the high-level intuition is that attention’s per-token state (the KV cache) grows linearly with sequence length and its compute is quadratic, while DeltaNet’s state is fixed-size — this helps the model scale to longer context.

All Qwen3.6 family models share this architecture; larger members of the family scale by adding more layers and using a wider hidden dimension (e.g., the Qwen3-Next-80B variant has 48 layers and a 2048 hidden dim).²

² Qwen3.6-35B-A3B has 40 layers organized as 10 repetitions of (3 DeltaNet + 1 Attention) blocks, so 30 DeltaNet layers and 10 Attention layers. See the apxml spec page.

Figure 2: The Qwen3.6-35B-A3B model architecture and its weights distribution

Table below shows the more detailed weights distribution:

  Group                            Params         Stored    Share
-----------------------------------------------------------------
  Token embedding                508.56 M     515.31 MiB    2.44%
  DeltaNet layers                  1.01 B       1.01 GiB    4.92%
  Attention layers               272.66 M     276.35 MiB    1.31%
  MoE blocks                      32.36 B      18.43 GiB   89.44%
  Final output                   508.56 M     397.86 MiB    1.89%
-----------------------------------------------------------------
  Reported total                  34.66 B      20.60 GiB  100.00%

MoE blocks dominate: they account for 89.4% of model size, and each MoE block is roughly 469 MB. However, each MoE block only activates 1/32 of its parameters (8 of 256 experts per token), while the other components fire for every token. So each token activates approximately 0.508B + 1.01B + 273M + 32.36B / 32 + 0.508B ≈ 3.3B parameters, which rounds to the “3B” in “A3B”.³

³ The official “3B active” figure is computed slightly differently — see the Qwen3.6 model card. The ballpark is the same.

There are two stages of LLM inference: prefill and decode. Prefill processes the input prompt (potentially thousands of tokens) in a single forward pass and populates the per-layer state (KV cache, DeltaNet state). It is usually compute-bound. Decode generates tokens one at a time, and on each step the full weight matrix must be streamed through the FPUs again, so it is usually memory-bound. We’ll come back to this distinction later.

Token <–> String

A token is an LLM-specific representation of a string. It isn’t fundamentally different from other representations like ASCII or Unicode, where 42 means * (in ASCII) and U+1F3D4 means 🏔 (in Unicode).

The first thing to look at is the vocabulary table, stored in the tokenizer.ggml.tokens metadata field as a list of strings. The first 8 tokens of this model are: [!, \, #, $, %, &, ', (]. So token id 0 refers to !, id 7 refers to (, and so on.

Note that different models may have different vocabulary tables. A smaller vocabulary means more tokens are needed to represent the same string. Modern LLMs tend to have larger vocabularies — this model has 151,936 vocabulary tokens, which exceeds u16::MAX (65,535), so the token index must be u32.⁴

⁴ Anthropic changed their tokenizer in Opus 4.7, which produces 1.46× more tokens on the same text input — a hidden ~46% price hike at the same per-token rate.

From token to string

Token-to-string is easy: look up each token in the vocabulary and concatenate the strings.

The one caveat is streaming output: a single token may represent only a partial UTF-8 sequence, so the decoder needs to buffer bytes until a complete codepoint can be emitted. This is especially common with non-ASCII scripts and emoji.

From string to token

String-to-token is more involved, but conceptually similar to how dictionary-based compression works. We start with the property that every byte in the input string has a single-token representation — by design — which guarantees that any input is representable.

Then we apply byte-pair encoding (BPE) merge rules; the goal is to greedily merge adjacent tokens into the longest sequences the vocabulary supports, which reduces the total token count.

Actually there are more details to consider

Four mistakes during my PhD

Thu, 23 Apr 2026 00:00:00 GMT

I’m currently having an end-of-PhD crisis, where I keep thinking about what to do next. It’s a good time to slow down and reflect, to avoid repeating the same mistakes. I have made many mistakes during my PhD. Here, I list four major ones I’m currently aware of.

Mistake 1: Giving up too late

Since childhood, I was told to never give up, stick to it, and I would eventually win. Maybe that’s true, but I definitely took it too far.

It took me a while to learn that never giving up is easy; giving up at the right time is hard.

I have two examples.

My first advisor was definitely not a fit for me, and I knew it after three months of working with him. But it took me almost three years to finally make the decision to switch. It has been one of the most important decisions I have made in my life, and I wish I had made it much earlier.

Why did it take so long? Because I wasn’t sure if it was my problem or my advisor’s problem. I wasn’t sure if a new advisor would actually be better, and I wasn’t sure if it was worth the disruption. I was told that a PhD is about suffering, so maybe it was just a normal part of the process.

My first paper was submitted six times and rejected six times over three years until it was finally abandoned. It was clearly a failed project, and I knew early on that it wasn’t worth publishing.

Why did it take so long to abandon it? I had invested so much time and energy into the project. Every resubmission required more physical and mental effort. The more I resubmitted, the harder it was to give up.

It was also an ego issue: I felt that if I gave up, I would be admitting that I wasted so much time on a failed project. Is this a skill issue? What if this happens again with the next project?

In both examples, I always knew the right thing to do. But I was held back by the momentum of keeping the same direction. It was very hard to make a change.

Lessons learned: practice low ego; build the courage to give up.

Mistake 2: Getting real too late

As a junior PhD student, it’s very tempting to work on simple topics like key-value stores, indexes, graph processing, and concurrency control. Here, “simple” means well-defined, classic, and narrowly scoped problems. It’s not that these topics are unimportant, but they are often the lazy way to do research: 1. Someone has already defined the problem exceptionally well, and the research community easily understands and appreciates it. 2. They have well-defined interfaces, meaning you don’t have to work on a real codebase; you can just benchmark an isolated component. 3. There are always ways to improve them by combining them with new workloads, new hardware, or AI.

My biased observation is that almost nobody outside academia cares about these topics, and the resulting papers are largely products of an academic echo chamber. Ironically, the more detached a project is from real-world use, the more it creates the illusion that you’re working on something profound that “only academia can do.”

Unfortunately, I wasted way too much time on these feel-good projects. I fooled myself by thinking, “neural networks were ignored for decades before changing the world, so maybe my project will be useful someday too.” I could find an infinite number of examples where “forward-looking” research eventually pays off, and that my project is like one of them. It’s beyond logical to argue whether a research project is worth doing (see mistake 4), the key is not to find more examples but to be honest with yourself.

Getting real is painful. It involves very complex systems, diverse use cases, and energy-consuming interactions with real people. It also doesn’t seem immediately rewarding: you invest a massive amount of time without a guaranteed return. But not getting real is an existential threat to the entire academic system. Tax payers cut the funding, industry ignores research work, and fewer and fewer students want to be in the field.

Getting real means to optimize from keyboard to screen, rather than to optimize the layer you like. There’s a stupid culture in academia that people categorize themselves as a “database” person, a “filesystem” person, or a “networking” person; they align themselves with topics, not problems. This self-constraint makes paper publishing easy but great work hard (see mistake 3).

Lessons learned: I created a rule for all my future projects. Before starting, I must find three practitioners who are willing to use my system if it succeeds, write down their names, and talk to them regularly.

Mistake 3: Care too much about paper/research community

Every PhD student needs to graduate with a few papers, usually three. That’s the single most important metric. Once you have three papers, everything else can be worked out. Similarly, the achievements of a professor are often measured by the number of publications in top-tier conferences; everything else is just a footnote on their CV.

This is how we benchmark the community, nobody should be blamed for optimizing for this metric.

But it’s dangerous for a community to optimize for a goal that is only indirectly connected to the rest of the world. I’m sure many publications will eventually translate to real-world impact, but in my humble opinion, the conversion loss is surprisingly high.

I was “smart enough” to adopt a paper-centric approach. Before I had anything, I first needed to have a recipe for top-tier conference papers. First, I needed a novel idea. That goes into the abstract, and it must be novel enough to justify a paper. Then, I would frame a convincing story about why it is important, and that goes into the introduction. Finally, I would show that my design is better than existing systems, and that goes into the evaluation.

This paper-centric research approach works surprisingly well for publishing papers. Because no effort is wasted, every minute of research goes directly toward the final manuscript.

But this is a dangerous paradigm because it distorts your judgment and misaligns with how novel ideas actually emerge. What if your novel idea doesn’t actually work, and you only realize this when the paper is already written up to the evaluation section? What should you do then? Find a new workload? Find a new hardware setup? Change your storyline a little bit? The absolute last thing you want to do is change the idea itself. Your idea is so beautiful, clean, and elegant; it fits perfectly into a paper and provides an amazing storyline. Except that it doesn’t work.

Naturally, you are incentivized to “make it work.” This doesn’t necessarily mean cheating, but rather finding logically correct, yet practically nonsensical, approaches to cover up the flaws (see mistake 4). And it feels perfectly fine to do this because it’s completely logical and everyone else does it.

I believe the true nature of research is to “go and see.” You start with a general direction, keep an open mind, explore, break things, learn enough to get into the deep details, and see if you find anything valuable, and then generalize.

This is difficult because it takes a tremendous amount of time to “learn enough to get into the details.” ¹ When you aren’t close enough to the details, it’s very tempting to superficially conclude that “A is essentially B,” leading you to miss interesting problems to work on.

¹ How to Do Great Work.

Lessons learned: do the right thing from first principles; don’t worry about what others are doing, competition is for losers.

I’d like to conclude this mistake with a quote from Steve Jobs:

It’s the disease of thinking that a really great idea is 90% of the work. And if you just tell all these other people, ‘Here’s this great idea,’ then of course they can go off and make it happen… As you evolve that great idea, it changes and grows. It never comes out like it starts because you learn a lot more as you get into the subtleties of it.

Mistake 4: Lost in the Church of Reason

This might be the most controversial, yet perhaps the biggest mistake I made.

One of the primary goals of PhD training is developing the ability of critical thinking and logical reasoning. I liked this so much. I enjoyed wielding the analytical knife to make my thoughts crystal clear. I enjoyed understanding everything I wanted to understand, and I enjoyed learning from great minds who interpret the world through simple, effective rules.

But after the initial excitement, it soon became boring. The sharp analytical knife quickly turned into muscle memory that we consciously or unconsciously apply to everything. It became a template that regulated my thinking, shielding me from the fear of truly free thinking. Using this framework always felt like the “correct” way to do things, and was a satisfying indicator of being well-educated.

Over time, I learned enough intellectual tools to be able to argue for anything. The power of logical reasoning is that you can always find an angle to frame your work as useful, novel, and important. The more deeply I understood a topic, the easier it was to invent arguments supporting my perspective. Academic research devolved into a purely intellectual exercise of crafting arguments, and I became lost in the church of reason ².

² The Church of Reason is a metaphor from the book Zen and the Art of Motorcycle Maintenance

³ Math’s Fundamental Flaw

I soon fell into logical and moral nihilism, losing the ability to judge what is truly right and wrong. Furthermore, I learned that pure logic itself is fundamentally unsound, incomplete, and undecidable³. This mindset had profound consequences on my life. I lost interest in doing anything, because, from a strictly logical standpoint, doing “great” research was entirely indistinguishable from doing “bad” research.

The cure was finding my leap of faith⁴—the boundary of logical reasoning where you must unconditionally accept something as true. That final leap of faith boils down to a single question: what is good, and what is not good?

⁴ Leap of Faith

My leap of faith, after many revisions, is this: to add something unique to humanity. I don’t necessarily know exactly what that thing is yet, but I know what it isn’t. If I don’t do a project, but someone else will, then it’s not a true addition to humanity. In those cases, I will just let other people do it, and I will focus on creating value that only I can do.

Conclusion

I’d like to conclude with my fifth mistake: not being ambitious enough. I wasted too much time stuck in local minima—trying to satisfy my ego, maintain a sense of security, and avoid being seen as a failure.

I haven’t yet found the wisdom and courage to navigate my way out, but I hope to eventually leave something valuable to humanity.

parquet-linter: A better Parquet is Parquet itself

Sun, 22 Feb 2026 00:00:00 GMT

Acknowledgments

My work is supported by funding from InfluxData, Bauplan, SpiralDB, and the taxpayers of the State of Wisconsin and the federal government. Much appreciation!

Try it out here: https://github.com/XiangpengHao/parquet-linter

> cargo install parquet-linter-cli

> parquet-linter data.parquet

warning low-compression-ratio
  --> column[2]("pid")
  aggregated compression ratio is 1.09 (SNAPPY) across 1/1 row groups; data is nearly incompressible
  fix: set column pid compression uncompressed

warning dictionary-encoding-cardinality
  --> column[1]("ts_ns")
  dictionary data pages fell back to PLAIN in 1/1 row groups; estimated cardinality is moderate (~132048 distinct / 380089 non-null = 35%), dictionary page size may be too small
  fix: set column ts_ns dictionary_page_size_limit 2097152

Parquet is a spec, not an implementation

When we talk about Parquet, we often refer to one of its readers or writers, likely the Java, C++ (used in PyArrow), Rust, or DuckDB implementation.

When we complain that Parquet is not great for our use case, we’re actually complaining that the Parquet file generated by a specific implementation with a specific set of parameters is not good enough.

As a spec, Parquet is flexible; it doesn’t really restrict much on how the data is stored, and it allows very fine-grained configuration on almost every aspect of the file format. There’s no single best “Parquet” for all use cases.

Configuring the right parameters for the right use case is not easy. It often requires a deep understanding of the cascading effects of encodings, compression, layouts, data types, and page/row group sizes. It’s likely that no single person on the planet can get it right on the first try.

We need different kinds of Parquet

You might think, why not just use the best practices? Because it’s not easy to define what is “best”.

Use case 1: long term archival

Probably the most common use case for Parquet is to archive data for long-term storage. In this use case, we don’t care about decoding speed; we want the data to be as compact as possible. Smaller Parquet files means lower S3 bill.

Use case 2: fast query performance

Parquet is the backbone of lakehouses like Iceberg, Delta Lake, and DuckLake. In such use cases, we want to maximize query performance. As a recent study shows (Liquid Cache), decoding Parquet takes a significant amount of query time. To minimize decoding time, we want to use lightweight compression, or even no compression at all (as in Databricks’ local cache).

Use case 3: ecosystem compatibility

Parquet is an open-direct-access format; Spark, Trino, DuckDB, DataFusion, etc., all support it. But not all readers support the same set of Parquet features. In this use case, we’d like to use the most conservative set of Parquet features that are supported by all readers.

Vision: Parquet Linter

Introducing Parquet Linter: https://github.com/XiangpengHao/parquet-linter, a tool to check compression, decompression, and compatibility issues in your Parquet files.

Parquet Linter transforms your Parquet file into a better Parquet file

I envision three levels of Parquet Linter.

Level 1: pure gain

In this level, Parquet Linter checks for issues that make Parquet files worse for all use cases and suggests better configurations.

For example, if a column’s compression ratio is 1.0 using a compression algorithm (this happens surprisingly often!), then we should just remove the compression because it is pure overhead.

Level 2: trade-offs guided

In this level, we allow the user to specify the trade-offs they want to prioritize, e.g., choosing compression ratio over decoding speed. Then Parquet Linter checks the configuration and suggests a better set of configurations that aligns with the user’s trade-offs.

Level 3: towards intelligence

The Parquet Linter is essentially a model; it takes workload features as input and outputs a set of configurations to store the data.

Currently, this model is a set of empirical rules, biased by the experience of a poor database PhD student. Just like what’s happening everywhere else in the world, we can replace the set of rules with a model, maybe even a large language model. I believe this can be a very interesting research direction going forward.

Does it work?

I picked 7 files (somewhat randomly) from HuggingFace’s datasets and ran the Parquet Linter (Level 1) on them.

We track two metrics separately:

decode time (ms): time to convert a Parquet file into Arrow RecordBatches
file size (MB): size of the Parquet file on disk

Even with just simple heuristics from Level 1, Parquet Linter can achieve significant reduction in both file size (up to -19.2%) and decode time (up to -29.8%). Level 2 and 3 are currently on the roadmap. With the collective wisdom of the community, I’m sure we can do much better.

File Size Leaderboard (MB, lower is better)

	File 0	File 1	File 2	File 3	File 4	File 5	File 6	Total
HuggingFace default	171.38	137.88	105.55	88.78	145.97	62.59	247.95	960.10
parquet-linter	171.35 (-0.02%)	128.31 (-6.94%)	90.10 (-14.64%)	71.70 (-19.24%)	141.41 (-3.12%)	58.08 (-7.21%)	244.73 (-1.30%)	905.68 (-5.67%)

Decode Time Leaderboard (ms, lower is better)

	File 0	File 1	File 2	File 3	File 4	File 5	File 6	Total
HuggingFace default	243.59	302.44	272.20	27.34	285.08	84.62	377.25	1592.53
parquet-linter	237.92 (-2.33%)	212.03 (-29.89%)	213.47 (-21.58%)	38.43 (+40.56%)	205.84 (-27.80%)	51.36 (-39.31%)	322.11 (-14.61%)	1281.15 (-19.55%)

Benchmark `parquet-linter`

You can verify the results by running the benchmark yourself:

cargo build --release --package parquet-linter-leaderboard
./target/release/parquet-leaderboard --from-linter --iterations 3

Vector search using only Parquet and DataFusion

Mon, 09 Feb 2026 00:00:00 GMT

Acknowledgments

My work is supported by funding from InfluxData, Bauplan, SpiralDB, and the taxpayers of the State of Wisconsin and the federal government. Much appreciation!

To search vector embeddings, the standard advice is almost always “spin up a vector database.”

Tools like Pinecone, Milvus, or specialized formats like Lance are fantastic pieces of engineering.

But they come with a hidden cost: complexity. New infrastructure to manage, new file formats to learn, and new data to coordinate.

But what if you didn’t need any of that? What if you could just keep your data in Parquet, the format you’re already using?

This post explores a little experiment: implementing efficient, native vector search directly inside Parquet files. Introducing pq-vector: https://github.com/XiangpengHao/pq-vector

Wait, Isn’t Parquet Terrible for Random Access?

“But Parquet is a columnar format!” I can hear you screaming. “It’s designed for heavy scans, not point lookups!”

You’re not wrong. The common wisdom is that Parquet is ill-suited for random access because it stores column values in compressed pages. To retrieve a single value—like a specific vector embedding—you typically have to decompress the entire page it resides in. This is wasteful if you only need that one item.

But let’s look at the actual numbers.

A typical vector embedding—say, from OpenAI’s text-embedding-3-small model—has 1,536 dimensions. That’s about 6KB of data. Now, guess how big a standard Parquet page is? Usually a few KB as well.

If we simply configure the Parquet writer to match the page size with the embedding size, we can effectively force each embedding into its own page. With this, “decompressing a page” just means “reading the one vector we want.”

We don’t change the file format; we just tune it.

// Configure parquet to store one embedding per page
let props = WriterProperties::builder()
    .set_data_page_size_limit(vector_size)     // one vector per page
    .set_column_compression(embedding_col, Compression::None)
    .build();

With this simple configuration change, we’ve effectively turned Parquet into a random-access friendly format for our vectors.

By simply configuring the page size to match the embedding size, we can effectively turn Parquet into a random-access friendly format for our vectors.

Zero-copy Vector Indexing

Of course, fast random access isn’t enough. If you have to scan every single row to calculate distances (O(N)), it doesn’t matter how fast you can read an individual page — it’ll still be slow. We need an index.

But here’s the challenge: how do we add an index without breaking compatibility? We don’t want to create a “custom Parquet” that DuckDB or Spark can’t read.

The solution is surprisingly elegant. Parquet allows you to embed arbitrary metadata in the file footer ¹ . Standard readers will happily ignore it, but our specialized reader can look for it.

¹ This is not a hack, it a elegant Parquet feature that allows you to extend the file format, more details here

We chose the Inverted File (IVF) index for this prototype. It works by partitioning the vector space into clusters (centroids). When we want to search, we figure out which clusters are close to our query, and then we only look at the vectors in those clusters.

The best part? It’s zero-copy.

Some vector stores (HNSW-based) force you to duplicate your data into their internal structures. Our index is just a lightweight list of pointers (row IDs) and cluster centroids. The actual heavy vector data stays right where it is — in the Parquet data pages.

In our experiments with ~5,000 academic papers each with 4096-dimensional embeddings (we used qwen/qwen3-embedding-8b), the index added a negligible 0.21 MB to a 68 MB file. That’s an overhead of just 0.3%.

The vector index references to the embedding vectors in Parquet files, without duplicating the data.

In-Place Index Creation

Now we understand vector search in Parquet is not only feasible, but also fast.

You might ask: doesn’t this mean I need to rewrite my entire Parquet file just to create that index? But here’s the kicker: creating a vector index on Parquet doesn’t require a rewrite.

Wait, isn’t Parquet immutable? How can you add an index to a Parquet file without rewriting it?

Enter another beautiful part of Parquet files: The Footer. At the end of each Parquet file is its metadata. We can simply append our vector index after the end of the metadata, and then copy that old metadata to the end of the vector index.

This way we create a perfectly legit Parquet file without rewriting any data pages².

² This assumes the embedding column is already optimized for random access. There are likely clever ways to rewrite only the embedding column, though.

The index is appended after the end of the old metadata

DataFusion Integration

An index is useless if you can’t use it. DataFusion lacks “official” support for custom indexes. But its modularity is its superpower: anyone can extend it.

We implemented a DataFusion query optimizer which will automatically rewrite the query plan to use the index, if the Parquet file has one.

pq-vector automatically rewrites the query plan to use the index

How It Looks in Code

We implemented a proof-of-concept using Rust, using parquet and datafusion.

1. Building the Index

The builder reads your existing Parquet file, trains K-means centroids, and writes the index to the file footer.

use pq_vector::IndexBuilder;

IndexBuilder::new(
    "data/combined.parquet",
    "embedding",
)
.n_clusters(100)
.build_inplace()?;

2. Searching with SQL (DataFusion)

To make this actually useful, we hooked it into DataFusion’s optimizer rules. This lets you run vector searches using standard SQL syntax.

use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::SessionContext;
use pq_vector::df_vector::{PqVectorSessionBuilderExt, VectorTopKOptions};

// Enable the pq-vector extension in DataFusion
let state = SessionStateBuilder::new()
    .with_default_features()
    .with_pq_vector(VectorTopKOptions {
        nprobe: 10,
        ..Default::default()
    })
    .build();
let ctx = SessionContext::new_with_state(state);

ctx.register_parquet("t", "data/combined.parquet", Default::default())
    .await?;

// Standard SQL vector search!
let df = ctx
    .sql("SELECT * FROM t ORDER BY array_distance(embedding, [0.1, 0.2, 0.3]) LIMIT 10")
    .await?;

Notice that we can mix the vector search with normal SQL filters, one of the key features of Vector Databases. This is the power of keeping everything in one engine. You don’t need a vector search engine.

Does It Actually Work?

We benchmarked this on our dataset of 4,886 vectors (4096 dimensions).

Operation	Time	Speedup	Recall@10
Brute force	100ms	1x	1.00
IVF search (nprobe=1)	3.4ms	29x	0.83
IVF search (nprobe=5)	17.7ms	5.7x	0.96
IVF search (nprobe=10)	32.4ms	3.1x	1.00

The results are pretty clear. Even with a simple IVF index, we see massive speedups compared to a full scan. With nprobe=5, we’re getting 96% recall at 5.7x the speed of a brute force scan.

The latency is almost entirely dominated by random I/O, which validates our theory: if you tune the page size correctly, Parquet handles random access just fine.

Need your help!

pq-vector is very early; it’s definitely not as polished as commercial vector databases now, but we have just shown that there’s nothing we can’t do with standard open source tools like Parquet and DataFusion. Here are a few things we’re planning to do next:

No HNSW: We used IVF because it’s simple and compact. Graph-based indexes like HNSW are probably more accurate, but they have much higher space overhead.
Multi-Parquet index: We often query many Parquet files, not just one file. We plan to support querying multiple Parquet files seamlessly.
Better filter support: Like DuckDB’s VSS, our implementation currently has limited support for filters. We plan to allow users to write SQL like: SELECT * FROM t WHERE col_1=5 ORDER BY array_distance(embedding, [0.1, 0.2, 0.3]) LIMIT 10.

github.com/XiangpengHao/pq-vector

Stop building systems for agents

Mon, 02 Feb 2026 00:00:00 GMT

Acknowledgments

My work is supported by funding from InfluxData, Bauplan, SpiralDB, and the taxpayers of the State of Wisconsin and the federal government. Much appreciation!

The Wrong Direction

Everyone knows today’s systems are sub-optimal for agents. They waste context, confuse models with unstructured output, and prioritize vision-based human interaction over text-based LLMs.

So we desperately need to build agent systems that treat LLMs as first-class citizens. I’m sure you can generate a hundred papers or spawn a hundred startups in this direction.

This approach surely works, but it is not fundamental: if humans can live with it, LLMs can eventually catch up. Maybe next month, maybe next year, but eventually LLMs can easily do what humans can do, and then your mission-critical agent-native system becomes a nice-to-have.

In the LLM era, we need to ask not what humans can do for LLMs, but what LLMs can do for humans.

So what is the fundamental challenge here? The velocity at which a human can take responsibility for an agent’s actions.

The Unaccountable Machine

In human society, every functional system relies on an accountability chain. When things break, we need a person to blame, and to fix it.

For the last few decades, we have maintained a subtle balance where the velocity of building systems roughly equals the velocity of accountability.

LLMs changed this by making it 1000x faster to write code, but our ability to take accountability for a system barely changed.

This creates a perfect accountability sink. When a vibe-system fails, we can only hope LLMs will fix it by themselves; they are unaccountable. But unaccountable systems are useless, and will eventually collapse.

Most LLM-vibed systems look fancy at first, but without a human owning their behavior, they are useless at best, and burdensome to society at worst.

Axiom: AI systems, no matter how capable, must be held accountable by a human.

Therefore, the real bottleneck of agent adoption is the Time to Accountability: how quickly a human operator can understand, diagnose, and own the system’s behavior?

The “Happen-to-Work” Foundation

(Un)Surprisingly, this bottleneck has almost nothing to do with AI.

You join a new team. How long does it take for you to be held accountable for a production system? It happens when you can observe the system interactions, reproduce regressions and failures, and explain what’s going on.

You may think, isn’t this what we have been preparing for? Hasn’t computer science been working on this for decades? Isn’t this a solved problem?

Unfortunately, the answer is no. We lack the infrastructure to understand systems from a third-party perspective; instead, the wisdom is hidden in the brain of the engineer who built it. From a user’s perspective, everything just happens-to-work.

When we compile software, it just happens-to-compile. We hope we are using the right compiler version, compiling the right dependencies, executing in the correct parallel order, and persisting files without corrupting data (in fact, we don’t even know if the data is corrupted or persisted).

When we issue a database transaction, it just happens-to-work. We hope the data transferred from disk returns in a deterministic order, that transactions respect isolation levels, and that our access controls actually work.

LLM Agents Shake Our Fragile Foundation

How could it be that our modern society is based on something so fragile that almost nobody understands? You might ask.

The answer is simple: we have only spent minimal effort to make it work most of the time. When rare edge cases happen, we just retry and hope they don’t appear. No one is able to reproduce a rare production bug, no one is able to fully explain why performance jitters, and no one is able to fully understand how sub-systems interact.

It has been fine, you may argue. But no, it’s not fine. As LLMs elevate humanity to the next level, what used to be ok is no longer ok.

We’re now capable of operating tremendously more systems; tremendous amounts of work will be cheaply digitized, and a tremendous number of tasks will be cheaply performed by LLMs. When the scale grows 1000 times, our fragile foundation becomes the problem.

The Real Agent-Native Systems

So what defines agent-native systems? They are human-centric:

Radically observable.
Radically deterministic.

Radically observable means we must see exactly how the system performs: who invokes a system call, how threads interleave, which function writes to a memory address, etc. The system should act like a glass box—observing system behavior from all levels of the stack. If performance regresses, we should see the exact reason, whether the CPU core is overheating or the SSD firmware is garbage collecting.

Radically deterministic means that given the same input, the program produces exactly the same output (randomness is controlled via seeds). This applies not only to a single function, but to the entire multi-threaded, distributed system. Threads must interleave in exactly the same way, hash tables must iterate in exactly the same order, and environmental noise (like noisy neighbors) must never alter the software’s execution.

Stop building systems tailored for agents, build systems that are easier for humans to own, make them observable and deterministic.

What is LiquidCache?

Mon, 24 Nov 2025 00:00:00 GMT

Acknowledgments

This work is supported by funding from:
1. InfluxData, Bauplan, and SpiralDB.
2. The taxpayers of the State of Wisconsin and the federal government.

Your support for science is greatly appreciated!

LiquidCache is a caching layer that unifies the design goals of compute and storage¹.

¹ Check out our research paper (VLDB 2025) for more technical details.

It accelerates query performance without needing to leave Parquet.

It addresses this fundamental tension:

Storage systems want to optimize for ecosystem compatibility, and long-term, stable, open governance; industry thus gravitated towards Parquet as the de facto columnar format.
Query engines want the data to be optimized for flexible layouts, rapid evolution, and performance-first optimizations.

Instead of squeezing the last bits of performance from Parquet², or trying to create future-proof file formats³, LiquidCache addresses this problem through a new abstraction: the caching layer.

² Great paper on Parquet selection pushdown: Selection Pushdown in Column Stores using Bit Manipulation Instructions.

³ AnyBlox: A Framework for Self-Decoding Datasets, and F3: The Open-Source Data File Format for the Future

At a glance, LiquidCache is a distributed caching service: it supports all object storage backends (S3, GCS, Azure Blob Storage, etc.), and serves all kinds of applications (knowledge bases, dashboards, etc.) deployed on all kinds of compute (Kubernetes, Lambda, etc.).
Under the hood, LiquidCache caches Parquet as liquid data, which is ultra-optimized for compute pushdown, compressed execution, modern storage, and network‑efficient data transfer.

LiquidCache overview. It caches different object store sources and serves different analytical applications.

It is built on open standards: Parquet for data storage, DataFusion as the query engine, and Arrow Flight for data transfer. This makes LiquidCache highly composable – you can easily integrate it into your existing analytics stack.

Why LiquidCache?

We like S3

Simple durability: 11 nines of durability—you never have to worry about data loss.
Simple scalability: virtually unlimited space and throughput.

But S3 is slow and expensive

≈100 ms first‑byte latency plus transfer latency; this quickly adds up when multiple round‑trips are needed to fetch data.⁴
Storage, request, and data‑transfer/egress costs; prices have remained largely unchanged for a decade even as underlying hardware has become ~20× cheaper.

⁴ Exploiting Cloud Object Storage for High-Performance Analytics

S3 prices have barely changed for a decade, despite ~20× reductions in underlying hardware costs, credit to Andrew Lamb

LiquidCache: foundation of diskless architectures

Caches are everywhere⁵: compute‑local caches (e.g., Snowflake/Databricks local NVMe, Spark host caches)⁶, shared‑nothing caches, and cache services⁷.
DLC trilemma: among durability, low latency, and low cost, you can only choose two⁸.

⁵ The Five-Minute Rule for the Cloud: Caching in Analytics Systems

⁶ DuckDB’s external file cache

⁷ ClickHouse’s distributed cache for S3

⁸ The Cloud Storage Triad: Latency, Cost, Durability

How LiquidCache Works

We like Parquet

All major query engines support it (DataFusion, Spark, Trino, DuckDB, Snowflake, BigQuery, and more).
It is battle‑tested and keeps evolving (e.g., page indexes, new encodings).
It is under open, stable governance (Apache Software Foundation), so your data is in good hands.

But sometimes we want more aggressive performance

There are better encodings and compression schemes out there.
Parquet is critical data infrastructure: it evolves cautiously to keep your data safe and stable—it can’t try new research today and abandon your data tomorrow.

LiquidCache: cache-only, pushdown-optimized data representation

LiquidCache uses state‑of‑the‑art encodings and compression chosen by the workload.⁹
Liquid data is invisible to the rest of the ecosystem: it is cache‑only. This means it can freely change its layout, adding or removing encodings without breaking any user code.
LiquidCache transparently, progressively, and selectively transcodes Parquet data to the liquid format.
Liquid data is designed for efficient pushdown to save both compute and network resources.

⁹ The liquid format is heavily inspired by Vortex. We plan to support a Vortex backend in the future.

Without any changes to Parquet, LiquidCache takes care of the performance optimizations.

Conclusions

LiquidCache is the one‑stop shop for diskless, serverless, and pushdown‑native analytics.

It is built on open standards (Parquet, Arrow Flight, DataFusion) for easy integration and stable governance.

LiquidCache caches Parquet as liquid data, which is ultra-optimized for compute pushdown, compressed execution, modern storage, and network‑efficient data transfer.

Who are we?

LiquidCache started as a research project led by Xiangpeng Hao at UW‑Madison ADSL.
It was made possible by a research gift from InfluxData. One year later, SpiralDB and Bauplan also joined the journey.¹⁰
LiquidCache will remain a public‑benefit project in appreciation of the support from taxpayers, research gifts, and the open‑source community.

¹⁰ Support our research here!

On the Nature of Strings

Wed, 13 Aug 2025 00:00:00 GMT

Acknowledgments

This work was supported by funding from:

InfluxData, Bauplan, and SpiralDB.
The taxpayers of the State of Wisconsin and the federal government.

Your support for science is greatly appreciated!

Strings are the most common data type we encounter in analytics. When we built LiquidCache – a distributed pushdown cache for DataFusion – we discovered that the way strings are stored and processed has an outsized impact on performance and memory. This post distills key points from my “On the Nature of Strings” talk at DC Systems (slides here; recording coming soon) and explains how we rethink string storage to make LiquidCache fast and lean.

TL;DR

We analyzed the PublicBI dataset (46 tables, 710 queries, 386 GB of data) and found that strings are the most important data type in the dataset: 61 % of bytes are strings, and around 40 % of projected columns are string columns.
There are four types of strings, categorized by distinct ratio and average length:
Dictionary encoding is crucial for storing string data:
Different trade-offs for representing variable-length strings:
Inlined prefixes are critical for efficient comparisons:
Null handling is essential:
Compression is important:

Conclusion

Strings are the most important data type in analytics, and we need to take them seriously.
Strings have many subtypes and should be treated differently.
If you’re not sure, try LiquidCache – it has a state-of-the-art string-handling pipeline and delivers best-in-class string performance with easy integration.

Build your own S3-Select in 400 lines of Rust

Sun, 23 Mar 2025 00:00:00 GMT

Acknowledgement

This blog post was made possible by funding support from:

InfluxData
Taxpayers of the state of Wisconsin and the federal government.

Your support for science is greatly appreciated!

TL;DR

S3-Select can filter S3 data before sending it back to you, significantly saving network bandwidth and time. Unfortunately, AWS killed this feature in 2024.

Good news: you can build your own S3-Select with all open-source tools and open standards! This blog post shows you how to do it in 400 lines of Rust¹ using the FDAP stack².

¹ Without blank lines and comments.

² FDAP: Apache Arrow Flight, Apache DataFusion, Apache Arrow, Apache Parquet

Complete code is available in this repository.

Looking for a more complete solution? Check out LiquidCache, a open-source/open-standard, push-down enabled storage system built on the same principles.

Comparing S3 (left) with S3-Select (right). S3-Select evaluates the filter before returning the data to you, saving network bandwidth.

Architecture

As we can’t change S3, we assume that the data is stored in our own storage system (often as a cache to S3).

Our architecture consists of three main components: a storage server, a compute client, and the Arrow Flight protocol connecting them.

The compute server is a full-fledged DataFusion server that evaluates user queries. The storage server contains Parquet files and also embeds a DataFusion instance for filter evaluation. The compute and storage communicate with each other via Arrow Flight.

Compute

The compute (client) is a full-fledged DataFusion node that automatically pushes down filter expressions to the storage server. Unlike S3-Select which requires manual SQL filter crafting, our system automatically decides what can be evaluated at the storage layer, while handling complex operations (joins, aggregations) in the compute node.

Storage

The storage (server) stores Parquet³ files and evaluates the filters pushed down from the compute node. We leverage DataFusion’s highly optimized query engine for filter evaluation, which includes efficient Parquet reading with filter pushdown, parallel filter evaluation across multiple threads, and streaming filtered results back to the compute node.

³ Parquet is the industry standard columnar storage format. CSV/JSON/etc. are also easily supported, but you should really consider using a real format like Parquet.

Communication

Communication between compute and storage happens via the Arrow Flight protocol – a high-performance, open-standard protocol for efficient data exchange. Unlike S3-Select which transmits data in CSV/JSON format, we keep data in columnar format throughout the entire pipeline. This means the compute node can directly operate on the data without heavy deserialization, and any system that speaks Arrow Flight can read from our storage.

Life of a query

Let’s walk through how a query flows through our system:

The life of a query. The compute node pushes down the filter expression to the storage node. The storage node evaluates the filter and sends the data back to the compute node. The compute node evaluates the rest of the query and sends the result back to the user.

Schema Resolution: When a user submits a query, the compute node first resolves the table schema from the storage node via Arrow Flight.
Query Planning: The compute node generates a query plan and decides which parts should be evaluated locally versus pushed down to the storage node.
Filter Pushdown: The compute node sends filter predicates to the storage node for evaluation close to the data.
Filter Evaluation: The storage node evaluates these filters and streams only the matching data back to the compute node.
Final Processing: The compute node handles the computation-heavy parts (aggregations, joins, etc.) and returns the final result to the user.

Note that only filters are pushed down to the storage node, computation-heavy operations happen at the compute layer. This prevents overloading the storage node’s processing capabilities while reducing network traffic.

Implementing the Storage Server

Now that we understand the architecture, let’s build the storage server. Surprisingly, we can implement it in less than 100 lines of code thanks to DataFusion!

Our storage server is defined as:

struct StorageServer {
    execution_plans: Mutex<HashMap<u64, Arc<dyn ExecutionPlan>>>,
    next_id: atomic::AtomicU64,
    ctx: SessionContext,
}

Let’s break down what each field does:

execution_plans: Maps query IDs to their execution plans
next_id: Generates unique query IDs⁴
ctx: The DataFusion session context (we use a single context for all queries for simplicity)

⁴ In production, you should use a proper unique ID generator like uuid.

The StorageServer implements the FlightSqlService trait with three key methods:

#[tonic::async_trait]
impl FlightSqlService for StorageServer {
    type FlightService = StorageServer;

    async fn get_flight_info_schemas(
        &self,
        query: CommandGetDbSchemas,
        _request: Request<FlightDescriptor>,
    ) -> Result<Response<FlightInfo>, Status> {
        todo!()
    }


    async fn get_flight_info_statement(
        &self,
        cmd: CommandStatementQuery,
        _request: Request<FlightDescriptor>,
    ) -> Result<Response<FlightInfo>, Status> {
        todo!()
    }

    async fn do_get_fallback(
        &self,
        _request: Request<Ticket>,
        message: Any,
    ) -> Result<Response<<Self as FlightService>::DoGetStream>, Status> {
        todo!()
    }
}

Let’s implement each method one by one.

1. Schema Resolution

First, we need to implement get_flight_info_schemas to handle schema retrieval:

async fn get_flight_info_schemas(
    &self,
    query: CommandGetDbSchemas,
    _request: Request<FlightDescriptor>,
) -> Result<Response<FlightInfo>, Status> {
    let table_url = query.catalog.unwrap();
    let table_name = query.db_schema_filter_pattern.unwrap();
    _ = self
        .ctx
        .register_parquet(&table_name, table_url, Default::default())
        .await;
    let schema = self.ctx.table_provider(&table_name).await.unwrap().schema();
    let info = FlightInfo::new().try_with_schema(&schema).unwrap();
    Ok(Response::new(info))
}

This method:

Registers the Parquet file with DataFusion
Gets the schema from the registered table
Returns the schema as a FlightInfo response

2. Query Planning

Next, we implement get_flight_info_statement to handle query planning:

async fn get_flight_info_statement(
    &self,
    cmd: CommandStatementQuery,
    _request: Request<FlightDescriptor>,
) -> Result<Response<FlightInfo>, Status> {
    let query = cmd.query.as_str();
    let (state, logical_plan) = self.ctx.sql(query).await.unwrap().into_parts();
    let plan = state.optimize(&logical_plan).unwrap();
    let physical_plan = state.create_physical_plan(&plan).await.unwrap();
    let partition_count = physical_plan.output_partitioning().partition_count();
    let schema = physical_plan.schema();
    let id = self.next_id.fetch_add(1, atomic::Ordering::Relaxed);
    self.execution_plans
        .lock()
        .unwrap()
        .insert(id, physical_plan);
    let mut info = FlightInfo::new().try_with_schema(&schema).unwrap();
    for partition in 0..partition_count {
        let fetch = FetchResults {
            handle: id,
            partition: partition as u32,
        };
        let buf = fetch.as_any().encode_to_vec().into();
        let ticket = Ticket { ticket: buf };
        let endpoint = FlightEndpoint::new().with_ticket(ticket.clone());
        info = info.with_endpoint(endpoint);
    }
    Ok(Response::new(info))
}

This method:

Parses the SQL into a logical plan
Optimizes the logical plan and converts it to a physical plan
Returns partition info and query ID to the compute node

3. Data Streaming

Finally, we implement do_get_fallback to handle data streaming:

async fn do_get_fallback(
    &self,
    _request: Request<Ticket>,
    message: Any,
) -> Result<Response<<Self as FlightService>::DoGetStream>, Status> {
    let fetch_results: FetchResults = message.unpack().unwrap().unwrap();
    let plan_lock = self.execution_plans.lock().unwrap();
    let physical_plan = plan_lock.get(&fetch_results.handle).unwrap().clone();
    let stream = physical_plan
        .execute(fetch_results.partition as usize, self.ctx.task_ctx())
        .unwrap()
        .map_err(|e| arrow_flight::error::FlightError::ExternalError(Box::new(e)));
    let encoder = FlightDataEncoderBuilder::new().build(stream);
    let response_stream =
        encoder.map(|result| result.map_err(|e| Status::internal(e.to_string())));
    Ok(Response::new(Box::pin(response_stream)))
}

This method:

Retrieves the physical plan for that query
Executes the plan on the specified partition
Returns the stream to the compute node

That’s it for the storage server! Just these three methods are enough to implement the core functionality⁵.

⁵ Of course, a production server needs way more: error handling, logging, authentication, etc. But these concerns are orthogonal to our core implementation.

Implementing the Compute Node

The compute node has more work to do, as it needs to:

Communicate with the storage server to get data
Decide which parts of the query to send to storage
Process the filtered data to produce the final result

We’ll implement this through two main components:

FlightTable: Decides what gets pushed down to storage
FlightExec: Handles the data streaming from storage

Here’s how they fit together in the query plan:

An example query plan showing how FlightTable and FlightExec connect. The ParquetExec and FilterExec⁶ are executed on the storage server, while the rest of the plan runs on the compute node. The two plans are connected via the FlightExec node.

⁶ Strictly speaking, when filter pushdown is enabled, the FilterExec is merged into ParquetExec in the storage server.

Implementing FlightTable

Let’s start with FlightTable, which implements DataFusion’s TableProvider trait:

pub struct FlightTable {
    channel: Channel,
    server: String,
    table_name: TableReference,
    output_schema: SchemaRef,
}

#[async_trait]
impl TableProvider for FlightTable {
    async fn scan(
        &self,
        _state: &dyn Session,
        projection: Option<&Vec<usize>>,
        filters: &[Expr],
        limit: Option<usize>,
    ) -> Result<Arc<dyn ExecutionPlan>> {
        todo!()
    }

    fn supports_filters_pushdown(
        &self,
        filters: &[&Expr],
    ) -> Result<Vec<TableProviderFilterPushDown>> {
       todo!() 
    }

    // ... other trait methods
}

The FlightTable has two important methods:

scan: Creates a FlightExec node that pulls data from storage
supports_filters_pushdown: Tells DataFusion which filters can be pushed down

Let’s implement the scan method first:

FlightTable unparses filters, projections, and limits back to SQL and sends them to the storage server.

async fn scan(
    &self,
    _state: &dyn Session,
    projection: Option<&Vec<usize>>,
    filters: &[Expr],
    limit: Option<usize>,
) -> Result<Arc<dyn ExecutionPlan>> {
    let unparsed_sql = {
        // we don't care about actual source for the purpose of unparsing the sql.
        let empty_table_provider = empty::EmptyTable::new(self.schema().clone());
        let table_source = Arc::new(DefaultTableSource::new(Arc::new(empty_table_provider)));

        let logical_plan = TableScan {
            table_name: self.table_name.clone(),
            source: table_source,
            projection: projection.map(|p| p.to_vec()),
            filters: filters.to_vec(),
            fetch: limit,
            projected_schema: Arc::new(self.schema().as_ref().clone().to_dfschema().unwrap()),
        };
        let unparser = Unparser::new(&PostgreSqlDialect {});
        let unparsed_sql = unparser
            .plan_to_sql(&LogicalPlan::TableScan(logical_plan))
            .unwrap();
        unparsed_sql.to_string()
    };

    println!("SQL send to cache: \n{}", unparsed_sql);

    let mut client = FlightSqlServiceClient::new(self.channel.clone());
    let info = client.execute(unparsed_sql, None).await.unwrap();

    Ok(Arc::new(FlightExec::try_new(
        self.schema.clone(),
        info,
        projection,
        &self.server,
    )?))
}

The magic happens in the unparsing step⁷. We leverage DataFusion’s query planner to:

⁷ We can also send DataFusion’s physical plan (like Ballista), or even Substrait to the storage server.

Create a logical plan containing the filters, projections, and limits
Unparse this plan back to SQL using DataFusion’s unparser
Send this SQL to the storage server for evaluation
Create a FlightExec node that will stream data from storage

Next, let’s implement the filter pushdown support:

fn supports_filters_pushdown(
    &self,
    filters: &[&Expr],
) -> Result<Vec<TableProviderFilterPushDown>> {
    let filter_push_down: Vec<TableProviderFilterPushDown> = filters
        .iter()
        .map(
            |f| match Unparser::new(&PostgreSqlDialect {}).expr_to_sql(f) {
                Ok(_) => TableProviderFilterPushDown::Exact,
                Err(_) => TableProviderFilterPushDown::Unsupported,
            },
        )
        .collect();
    Ok(filter_push_down)
}

Our rule is simple but effective: if a filter can be unparsed to SQL, we push it down to storage.

Implementing FlightExec

Now let’s implement FlightExec, which is responsible for streaming data from storage:

pub struct FlightExec {
    server: String,
    partitions: Arc<[FlightPartition]>,
    plan_properties: PlanProperties,
}

impl ExecutionPlan for FlightExec {
    fn execute(
        &self,
        partition: usize,
        _context: Arc<TaskContext>,
    ) -> Result<SendableRecordBatchStream> {
        let future_stream = flight_stream(self.partitions[partition].clone(), self.schema());
        Ok(Box::pin(FlightStream {
            state: FlightStreamState::Init,
            future_stream: Some(Box::pin(future_stream)),
            schema: self.schema(),
        }))
    }

    // ... other trait methods
}

FlightExec is mainly a wrapper around FlightStream, which handles the async streaming of data from storage:

struct FlightStream {
    state: FlightStreamState,
    future_stream: Option<BoxFuture<'static, Result<SendableRecordBatchStream>>>,
    schema: SchemaRef,
}

impl Stream for FlightStream {
    type Item = Result<RecordBatch>;

    fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
        let result: Poll<Option<Result<RecordBatch>>> = loop {
            match &mut self.state {
                FlightStreamState::Init => {
                    self.state = FlightStreamState::GetStream(self.future_stream.take().unwrap());
                    continue;
                }
                FlightStreamState::GetStream(fut) => {
                    let stream = ready!(fut.as_mut().poll(cx)).unwrap();
                    self.state = FlightStreamState::Processing(stream);
                    continue;
                }
                FlightStreamState::Processing(stream) => {
                    let result = stream.as_mut().poll_next(cx);
                    break result;
                }
            }
        };
        match result {
            Poll::Ready(Some(Ok(batch))) => Poll::Ready(Some(Ok(batch))),
            Poll::Ready(None) => Poll::Ready(None),
            Poll::Ready(Some(Err(e))) => {
                panic!("Error reading flight stream: {}", e);
            }
            _ => Poll::Pending,
        }
    }
}

The implementation is a bit complex due to Rust’s lack of native async iterators, requiring us to implement a state machine manually. However, the core concept is straightforward – it pulls data from storage and returns it as a stream.

Putting It All Together

Now let’s assemble our components into a working system!

Server Binary

The server binary is simple – it just starts a Flight service with our StorageServer:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let addr = "127.0.0.1:50051".parse()?;
    Server::builder()
        .add_service(FlightServiceServer::new(StorageServer::default()))
        .serve(addr)
        .await?;
    Ok(())
}

Client Binary

The client binary configures DataFusion, registers our FlightTable, and runs the query:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut session_config = SessionConfig::from_env()?;
    session_config
        .options_mut()
        .execution
        .parquet
        .pushdown_filters = true;
    let ctx = Arc::new(SessionContext::new_with_config(session_config));

    let cache_server = "http://localhost:50051";
    let table_name = "aws-edge-locations";
    let table_url = "./aws-edge-locations.parquet";
    let sql = format!(
        "SELECT DISTINCT \"city\" FROM \"{table_name}\" WHERE \"country\" = 'United States'"
    );

    let table = FlightTable::create(cache_server, table_name, table_url).await;
    ctx.register_table(table_name, Arc::new(table))?;
    ctx.sql(&sql).await?.show().await?;
    Ok(())
}

Running the System

To run our S3-Select alternative:

cargo run --bin server
cargo run --bin client

You should see output like:

```bash
SQL to run: 
-------
SELECT DISTINCT "city" FROM "aws-edge-locations" WHERE "country" = 'United States'
-------

SQL to pushdown: 
-------
SELECT "aws-edge-locations"."city" FROM "aws-edge-locations" WHERE ("aws-edge-locations"."country" = 'United States')
-------

+----------------+
| city           |
+----------------+
| Boston         |
| Chicago        |
| Portland       |
| New York       |
| Newark         |
| Detroit        |
...

Notice that only the filter (WHERE "country" = 'United States') was pushed down to storage, while the aggregation (DISTINCT) was evaluated by the compute node. This is exactly what we wanted!

What’s Next: LiquidCache

Congratulations! You’ve built a working S3-Select alternative in just 400 lines of Rust. However, this blog post only scratches the surface of what’s possible with this architecture.

To take this concept further, I’m excited to announce LiquidCache – a modern, open-source, push-down enabled storage system built on the same principles. LiquidCache extends what we’ve built here (to 15k loc) with advanced features like:

Advanced caching strategies
Advanced filter evaluation techniques
Enhanced reliability and error handling
Performance optimizations

Check out our research paper for technical details and consider contributing to the project!

Conclusion

Building a functional S3-Select alternative is surprisingly simple once you leverage the right building blocks. The FDAP stack (Flight, DataFusion, Arrow, Parquet) provides powerful primitives that handle most of the heavy lifting for us.

The real challenge – and fun – lies in understanding what these components do and how to thread them together effectively. As demonstrated by the imports alone, we’re standing on the shoulders of giants:

use arrow::{
    array::RecordBatch,
    datatypes::{SchemaRef, ToByteSlice},
};
use arrow_flight::{
    FlightClient, FlightEndpoint, FlightInfo, Ticket,
    flight_service_client::FlightServiceClient,
    sql::{CommandGetDbSchemas, client::FlightSqlServiceClient},
};
use datafusion::{
    catalog::{Session, TableProvider},
    common::{ToDFSchema, project_schema},
    datasource::{DefaultTableSource, TableType, empty},
    error::{DataFusionError, Result},
    execution::{RecordBatchStream, SendableRecordBatchStream, TaskContext},
    logical_expr::{LogicalPlan, TableProviderFilterPushDown, TableScan},
    physical_expr::EquivalenceProperties,
    physical_plan::{
        DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties,
        execution_plan::{Boundedness, EmissionType},
        stream::RecordBatchStreamAdapter,
    },
    prelude::*,
    sql::{
        TableReference,
        unparser::{Unparser, dialect::PostgreSqlDialect},
    },
};
use futures::{Stream, TryStreamExt, future::BoxFuture};
use std::task::{Context, Poll, ready};
use std::{any::Any, pin::Pin, sync::Arc};
use tonic::{async_trait, transport::Channel};

The power of open-source tools and open standards isn’t just a fallback — it’s a superior approach that liberates us from vendor lock-in and service discontinuations. We’ve demonstrated that not only can we rebuild essential services like S3-Select after their commercial versions are discontinued, but we can create more powerful⁸, customizable, and cost-effective alternatives.

⁸ Azure also has filter pushdown, but they don’t support Parquet files.

Efficient Filter Pushdown in Parquet

Wed, 12 Mar 2025 00:00:00 GMT

Acknowledgement

This work (the PR, this blog post, parquet-viewer, and LiquidCache) was made possible by funding support from:

InfluxData
Taxpayers of the state of Wisconsin and the federal government.

Your support for science is greatly appreciated!

In the previous post, we discussed how DataFusion prunes Parquet files to skip irrelevant files/row_groups (sometimes also pages).

This post discusses how Parquet readers skip irrelevant rows while scanning data.

Why filter pushdown in Parquet?

Below is a query that reads sensor data with filters on date_time and location:

SELECT val, location 
FROM sensor_data 
WHERE date_time > '2025-03-12' AND location = 'office';

Parquet pruning skips irrelevant files/row_groups, while filter pushdown skips irrelevant rows. Without filter pushdown, all rows from location, val, and date_time columns are decoded before location='office' is evaluated. Filter pushdown is especially useful when the filter is selective, i.e., removes many rows.

In our setup, sensor data is aggregated by date — each day has its own Parquet file. DataFusion prunes the unneeded Parquet files, i.e., 2025-03-10/11.parquet.

Once the files to read are located, the current default implementation reads all the projected columns (sensor_id, val, and location) into Arrow RecordBatches, then applies the filters over location to get the final set of rows.

A better approach is filter pushdown, which evaluates filter conditions first and only decodes data that passes these conditions. In practice, this works by first processing only the filter columns (like location), building a boolean mask of rows that satisfy our conditions, then using this mask to selectively decode only the relevant rows from other columns (sensor_id, val). This eliminates the waste of decoding rows that will be filtered out.

While simple in theory, practical implementations often make performance worse.

Why slower?

At a high level, the Parquet reader first builds a filter mask – essentially a boolean array indicating which rows meet the filter criteria – and then uses this mask to selectively decode only the needed rows from the remaining columns in the projection.

Let’s dig into details of how filter pushdown is implemented in the current Rust implementation of Parquet readers.

Implementation of filter pushdown in Rust Parquet readers – the first phase builds the filter mask, the second phase applies the filter mask to the other columns

The filter pushdown has two phases:

Build the filter mask (steps 1-3)
Apply the filter mask to the other columns (steps 4-7)

Within each phase, it takes three steps from Parquet to Arrow:

Decompress the Parquet pages using generic decompression algorithms like LZ4, Zstd, etc. (steps 1, 4, 6)
Decode the page content into Arrow format (steps 2, 5, 7)
Evaluate the filter over Arrow data (step 3)

In the figure above, we can see that location is decompressed and decoded twice, first when building the filter mask (steps 1, 2), and second when building the output (steps 4, 5). This happens for all columns that appear both in the filter and output.

The table below shows the corresponding CPU time on the ClickBench query 22:

Decompress	Decode	Apply filter	Others
206 ms	117 ms	22 ms	48 ms

Clearly, decompress/decode operations dominate the time spent. With filter pushdown, we need to decompress/decode three times; but without filter pushdown, we only need to do this twice. This explains why filter pushdown is slower.

Note

Highly selective filters may skip the entire page; but as long as we read one row from the page, we need to decompress/decode the entire page.

Attempt: cache filter columns

Intuitively, caching the filter columns and reusing them later could help.

But caching consumes prohibitively high memory:

We need to cache Arrow arrays, which are on average 4x larger than Parquet data.
We need to cache the entire column in memory, because in Phase 1 we build filters over the entire column, and only use it in Phase 2.
The memory usage is proportional to the number of filter columns, which can be unboundedly high.

Worse, caching filter columns means we need to read partially from Parquet and partially from cache, which is complex to implement and requires a radical change to the current implementation.

Note

Feel the complexity: consider building a cache that properly handles nested columns, multiple filters, and filters with multiple columns.

Real solution

We need a solution that:

Is simple to implement, i.e., doesn’t require thousands of lines of code.
Incurs minimal memory overhead.

This section describes my <700 LOC PR (with lots of comments and tests) that reduces total ClickBench time by 15%, with up to 2x lower latency for some queries, no obvious regression on other queries, and caches at most 2 pages (~2MB) per column in memory.

New decoding pipeline, building filter mask and output columns are interleaved in a single pass, allowing us to cache minimal pages for minimal amount of time

The new pipeline interleaves the previous two phases into a single pass, so that:

The page being decompressed is immediately used to build filter masks and output columns.
We cache the decompressed page for minimal time; after one pass (steps 1-6), the cache memory is released for the next pass.

This allows the cache to only hold 1 page at a time, and to immediately discard the previous page after it’s used, significantly reducing the memory requirement for caching.

What pages are cached?

You may have noticed that only location is cached, not val, because val is only used for output. More generally, only columns that appear both in the filter and output are cached, and at most 1 page is cached for each such column.

More examples:

SELECT val 
FROM sensor_data 
WHERE date_time > '2025-03-12' AND location = 'office';

In this case, we don’t cache any columns, because val is not used for filtering.

SELECT COUNT(*) 
FROM sensor_data 
WHERE date_time > '2025-03-12' AND location = 'office';

In this case, we also don’t cache any columns, because the output projection is empty after query plan optimization.

Then why cache 2 pages/column instead of 1?

This is another real-world nuance regarding how Parquet layouts the pages.

Parquet by default encodes data using dictionary encoding, which writes a dictionary page as the first page of a column chunk, followed by the keys referencing the dictionary.

You can see this in action using parquet-viewer:

Parquet viewer shows the page layout of a column chunk

This means that to decode a page of data, we actually need to reference two pages: the dictionary page and the data page.

This is why we cache 2 pages per column: one dictionary page and one data page. The data page slot will move forward as we read the data; but the dictionary page slot always references the first page.

Cached two pages, one for dictionary (pinned), one for data (moves as we read the data)

How does it perform?

Here are my results on ClickBench on my AMD 9900X machine. The total time is reduced by 15%, with Q23 being 2.24x faster, and queries that get slower are likely due to noise.

┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃ no-pushdown ┃ new-pushdown ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │      0.47ms │       0.43ms │ +1.10x faster │
│ QQuery 1     │     51.10ms │      50.10ms │     no change │
│ QQuery 2     │     68.23ms │      64.49ms │ +1.06x faster │
│ QQuery 3     │     90.68ms │      86.73ms │     no change │
│ QQuery 4     │    458.93ms │     458.59ms │     no change │
│ QQuery 5     │    522.06ms │     478.50ms │ +1.09x faster │
│ QQuery 6     │     49.84ms │      49.94ms │     no change │
│ QQuery 7     │     55.09ms │      55.77ms │     no change │
│ QQuery 8     │    565.26ms │     556.95ms │     no change │
│ QQuery 9     │    575.83ms │     575.05ms │     no change │
│ QQuery 10    │    164.56ms │     178.23ms │  1.08x slower │
│ QQuery 11    │    177.20ms │     191.32ms │  1.08x slower │
│ QQuery 12    │    591.05ms │     569.92ms │     no change │
│ QQuery 13    │    861.06ms │     848.59ms │     no change │
│ QQuery 14    │    596.20ms │     580.73ms │     no change │
│ QQuery 15    │    554.96ms │     548.77ms │     no change │
│ QQuery 16    │   1175.08ms │    1146.07ms │     no change │
│ QQuery 17    │   1150.45ms │    1121.49ms │     no change │
│ QQuery 18    │   2634.75ms │    2494.07ms │ +1.06x faster │
│ QQuery 19    │     90.15ms │      89.24ms │     no change │
│ QQuery 20    │    620.15ms │     591.67ms │     no change │
│ QQuery 21    │    782.38ms │     703.15ms │ +1.11x faster │
│ QQuery 22    │   1927.94ms │    1404.35ms │ +1.37x faster │
│ QQuery 23    │   8104.11ms │    3610.76ms │ +2.24x faster │
│ QQuery 24    │    360.79ms │     330.55ms │ +1.09x faster │
│ QQuery 25    │    290.61ms │     252.54ms │ +1.15x faster │
│ QQuery 26    │    395.18ms │     362.72ms │ +1.09x faster │
│ QQuery 27    │    891.76ms │     959.39ms │  1.08x slower │
│ QQuery 28    │   4059.54ms │    4137.37ms │     no change │
│ QQuery 29    │    235.88ms │     228.99ms │     no change │
│ QQuery 30    │    564.22ms │     584.65ms │     no change │
│ QQuery 31    │    741.20ms │     757.87ms │     no change │
│ QQuery 32    │   2652.48ms │    2574.19ms │     no change │
│ QQuery 33    │   2373.71ms │    2327.10ms │     no change │
│ QQuery 34    │   2391.00ms │    2342.15ms │     no change │
│ QQuery 35    │    700.79ms │     694.51ms │     no change │
│ QQuery 36    │    151.51ms │     152.93ms │     no change │
│ QQuery 37    │    108.18ms │      86.03ms │ +1.26x faster │
│ QQuery 38    │    114.64ms │     106.22ms │ +1.08x faster │
│ QQuery 39    │    260.80ms │     239.13ms │ +1.09x faster │
│ QQuery 40    │     60.74ms │      73.29ms │  1.21x slower │
│ QQuery 41    │     58.75ms │      67.85ms │  1.15x slower │
│ QQuery 42    │     65.49ms │      68.11ms │     no change │
└──────────────┴─────────────┴──────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary           ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (no-pushdown)    │ 38344.79ms │
│ Total Time (new-pushdown)   │ 32800.50ms │
│ Average Time (no-pushdown)  │   891.74ms │
│ Average Time (new-pushdown) │   762.80ms │
│ Queries Faster              │         13 │
│ Queries Slower              │          5 │
│ Queries with No Change      │         25 │
└─────────────────────────────┴────────────┘

Conclusion

Despite being simple in theory, filter pushdown in Parquet is non-trivial to implement. It requires understanding both the Parquet format and reader implementation details. The challenge lies in efficiently navigating through the dynamics of decoding, filter evaluation, and memory management.

Where are we now, system researchers?

Sun, 09 Mar 2025 00:00:00 GMT

We, as system researchers, face an existential crisis. We find ourselves questioning our unique strengths and fundamental purpose where industry increasingly outpaces academic innovation.

Research is a privilege

Research is paid for by taxpayers, but we often forget this is a gift. We think we deserve this money because we’re smart or have fancy degrees, without asking if we really earned it.

I think spending money on research is good for everyone, but that doesn’t mean we can just take the money without being responsible to the public.

When someone pull the plug on the research funding, we could easily blame the person, and name a few research achievements that would be impossible without those funding. But from the bottom of my heart, I often have the same question that whether some of the research funding would be better spent on something else.

I’m not saying we should only fund research that promises clear results – that would go against the whole point of exploring new ideas. What worries me is research funding decisions are unaccountable to the public. The people who decide where money goes are usually academics themselves, who are often detached and highly alienated with what regular people actually need or care about. We often talk only to each other, praising work that looks good on paper but doesn’t help real people. Over time, with no one making us prove our worth, our research drifts further from what actually matters.

But the public has one last ultimate weapon — cutting off the money entirely. This tough move hurts, but sometimes it’s the only way to make researchers listen: we need to do work that actually matters to people.

Research is a privilege, not an entitlement. We must question whether our work truly justifies the public investment we receive and delivers meaningful value to society.

System research is irrelevant

System research is irrelevant. Industry has become the better place for meaningful systems work. Most impactful and innovative systems today come from companies, not universities.

Industry has the money and patience to build complete systems. But most importantly, industry systems are accountable – systems that don’t deliver value get shut down quickly. This accountability creates a natural selection process. Industry systems must stay relevant or die. They evolve to meet real needs or disappear.

Due to the unaccountable funding agencies mentioned above, research systems are often one-shot projects —- they are immediately abandoned right after publication. We are frequently impressed by the great systems coming from industry (the most recent example being the infrastructure at DeepSeek), and we’ve realized that we are far from competing with them. As a result, we seem to constrain ourselves to a few narrow research topics, solving problems that are difficult to connect with even a single real user (essentially imaginary problems). Or we simply adjust our goal from research to education, preparing students for their industry jobs, so they can continue research there. But if that is the case, why would we need research funding at all?

We are unqualified

PhD students are typically only a few years older than undergraduates. How can they possibly compete with senior industry practitioners who have been working on the same problems for decades?

Yet we are not even trying to compete. We don’t like “engineering problems”, because we researchers are supposed to work on “research problems”.

We like research problems for two simple reasons:

Feeling “researchy” makes us feel good about ourselves; simply thinking about fancy terminology makes us feel fundamental and important.
We don’t know how to code.

We waste too much time babbling about knowledge we learn from papers – how to schedule a million machines, how to train a billion parameters, how to design infinitely scalable systems. Just thinking about these problems makes us feel important as researchers, although most of us have never deployed a service in the cloud, never used the techniques we proposed, and never worked with the filesystems, kernels, compilers, networks, or databases we studied. We waste time on these theoretical discussions because we don’t know how to code and are unwilling to practice. As Feynman said, “What I cannot create, I do not understand.” Simply knowing how a system works from 1000 feet doesn’t mean we can build it. The nuances of real systems often explain why they’re built in particular ways. Without diving into these details, we’re merely scratching the surface.

Here are my bold claims:

Your opinion doesn’t matter until you write >50k lines of system code.
Your novel idea is 💩 if it’s not implemented in a system with real users.

(Writing code does not make you a good researcher, but not writing code makes you a bad one.)

The system research community does not need more novel solutions – novel solutions are essentially combinations of existing techniques. When we need to solve a problem, most of us would figure out a similar solution, and what matters is the execution of the ideas.

Instead, we need more people willing to sit down and code, build real systems, and talk to real users. Be a solid practitioner, don’t be a feel-good researcher.

We are pushing young researchers away

WE ARE TOO SLOW because we waste too much time on procedural overhead rather than actual scientific exploration.

Paper publishing takes too much time. We spend too much effort arguing what’s new and what’s hard, instead of focusing on doing the actual research. Writing a paper already takes too much time, and then we need to anonymize artifacts, register abstracts, wait for reviews, write rebuttals, revise the paper, and can still be rejected for arbitrary reasons. The turnaround time for a single submission can be up to 6 months.

There is also a dark side to paper publishing. The entire publication process can be exploited by determined actors, and we are unwilling to address this issue, probably because many big names are involved. Young researchers see low-quality papers being accepted, yet their own carefully crafted systems are rejected for arbitrary reasons.

All of these burn our time and energy, pushing young researchers away from the community. After all, research doesn’t have to happen in academia.

Our measurement is wrong

All of our research projects start with the two questions of what’s new and what’s hard, but unfortunately, these two questions have guided us toward irrelevancy. Many one-shot papers claim novelty and then disappear, preventing future research projects from making progress. They take credit for being the first to introduce an idea, even when the implementation doesn’t work or is completely wrong. Yet, all future researchers are required to compare their work against these papers.

Most of the time, the code is terribly implemented or overly simplified, making fair comparisons impossible. But reviewers don’t care – they see the two papers as informationally equivalent, viewing the same idea from a 1000-foot perspective, and lazily question authors about what’s new and what’s hard.

The real difference between papers often lies in numerous small details that sound trivial but are actually essential for relevance. In most cases, figuring out these details takes much more time and demonstrates more novelty than coming up with the initial idea itself.

The reviewers – often just a few years older than PhD students, don’t know how to code, unable to delve into implementation details, and incapable of appreciating real system nuances – will likely reject the paper for lack of novelty.

System research is knowing

Despite all previous arguments, I still believe there is value in systems research, and I still believe we can be relevant, but that’s for another post.

System research is about knowing – understanding systems deeply and completely, contrasting with AI research that embraces uncertainty and probabilistic approaches.

We study the complex interactions of systems, and practice to know them better. We fight against the seemingly infinite complexity of systems, and uncover the underlying principles.

Knowing requires a lot of practicing – hard work, patience, and a lot of coding – and we need a culture that cultivates this.

Online Parquet Viewer with Rust

Xiangpeng Hao — Tue, 12 Nov 2024 00:00:00 GMT

Quick look

Online here: https://parquet-viewer.xiangpeng.systems

Source code: parquet-viewer

Run SQL queries

Visualize query plans

Examine metadata

How it works

It compiles Parquet/Arrow/DataFusion to webassembly which runs in the browser.

Specifically, it uses trunk to pack wasm files, leptos to build reactive UI components, and Tailwind CSS for styling.

And most importantly, LLM wrote most of the code.

Why do I need this?

Parquet files are not human-readable, you can’t just open them like CSV or JSON files.

Of course, there are many CLI tools that allow you to inspect file content – if you enjoy deciphering cryptic CLI args.

There are also Java/C#/Windows applications that provide GUI options – if you’re nostalgia for the 90s.

But with this tool, everything happens inside your favorite browser, written in your favorite language, using your favorite tech stack.

Simply drag and drop the file, and all the important information is in front of you.

Rust for frontend?

The good

Leptos is surprisingly intuitive and easy-to-use.
DataFusion/Arrow/Parquet compiles to wasm with almost no modification.
There’s surprisingly small amount of code needed to build an interactive UI.

The bad

Build size is large, debug build is 100MB, release build is 40MB. Poor dev experience due to laggy edit-build-run cycle. (User experience is fine, though)
Not every crate is wasm-compatible, and it’s very difficult to triangle down the root cause.
I haven’t figure out how to nicely use JavaScript yet – sometimes you do need JS.

Caching in DataFusion

Xiangpeng Hao — Sun, 27 Oct 2024 00:00:00 GMT

📢📢📢 Checkout LiquidCache, an open-source caching solution that reduces latency by 10x for cloud-native DataFusion.

Note: my research is funded by InfluxData, which made this blog post possible.

In the last post, we discussed how DataFusion prunes Parquet files to read only the necessary data. This post explores DataFusion’s caching mechanisms, which help avoid repeating reads to object storage.

Current state

DataFusion provides a flexible and layered caching architecture that enables developers to optimize data access at multiple levels. The caching system is designed to be extensible, allowing custom implementations while providing sensible defaults. Let’s examine the built-in caching mechanisms that make DataFusion efficient.

1. List files cache

The list files cache optimizes directory scanning operations in DataFusion. When a ListingTable needs to access files in a directory, it first checks this cache before performing expensive filesystem listing operations.

This cache is implemented as a simple but effective HashMap that stores directory paths as keys and lists of file metadata as values. The metadata includes important file information like size, last modified time, and other attributes that would otherwise require filesystem/network calls to retrieve:

struct ListFilesCache {
    cached: HashMap<Path, Vec<ObjectMeta>>,
}

2. File statistics cache

The file statistics cache stores important statistics about each file, such as row count and column statistics (min/max values). These statistics are used by DataFusion’s query optimizer to make better execution decisions, like pruning files that cannot contain matching data.

The cache avoids having to recompute/re-decode these statistics for each query by storing them in a HashMap that maps file paths to both file metadata and statistics:

struct FileStatisticsCache {
    cached: HashMap<Path, (ObjectMeta, Statistics)>,
}

3. Parquet metadata cache

Getting Parquet metadata can be costly for two main reasons:

Network overhead: It requires up to 2 separate network requests to retrieve the raw metadata bytes - one for the footer offset and another for the actual metadata
Processing overhead: Decoding the metadata can be computationally expensive, especially for tables with many columns

To address these challenges, DataFusion provides the ParquetFileReaderFactory trait. This trait allows developers to implement custom metadata handling strategies. The factory returns an AsyncFileReader that separates metadata access from data page access, enabling different caching and optimization approaches for each:

pub trait AsyncFileReader: Send {
    fn get_metadata(&mut self) -> BoxFuture<'_, Result<Arc<ParquetMetaData>>>;

    fn get_bytes(&mut self, range: Range<usize>) -> BoxFuture<'_, Result<Bytes>>;

    ... // other methods
}

4. Parquet file range cache

Lastly and most importantly, DataFusion caches the Parquet file ranges that have been read.

Under the hood, DataFusion uses object_store to interact with the S3, GCS, or local filesystem. While object_store provides a unified interface for different storage backends, but it does not implement caching by default.

However, DataFusion’s flexible architecture allows developers to implement custom caching strategies by implementing the AsyncFileReader trait. This enables optimizations like:

Caching frequently accessed data ranges in memory
Implementing tiered caching (e.g., memory + local disk)
Adding compression to reduce memory usage
Implementing cache eviction policies based on access patterns

Notably, the AsyncFileReader trait has a get_bytes_ranges method:

pub trait AsyncFileReader: Send {
    ... // previously mentioned methods 

    fn get_byte_ranges(&mut self, ranges: Vec<Range<usize>>) -> BoxFuture<'_, Result<Vec<Bytes>>>;
}

The default implementation of get_byte_ranges simply calls get_bytes sequentially for each range, which can be inefficient when reading many small ranges. This is because each range requires a separate network request, leading to higher latency and costs.

However, developers can implement their own IO coalescing logic to optimize performance. For example, they could:

Merge adjacent or overlapping ranges to reduce the number of requests
Batch multiple small ranges into a single larger request
Implement prefetching for ranges likely to be needed soon

These optimizations can significantly improve read performance and reduce storage costs, especially when working with remote storage like S3 or GCS.

So far we have discussed the core caching mechanisms built into DataFusion. These caches work together to optimize different aspects of query execution, from file discovery to data access.

The following sections will explore more advanced topics and future directions for DataFusion’s caching capabilities.

Caching Arrow

Arrow is the in-memory columnar format that DataFusion uses to process data efficiently. Before DataFusion can execute any query logic, Parquet data must be decoded into Arrow format. This decoding process involves decompressing the data, converting between data types, and validating constraints - operations that recent research¹ has shown can be a performance bottleneck for many analytical workloads.

One promising optimization approach is to cache the decoded Arrow data rather than the raw Parquet bytes. This allows us to skip the expensive decoding step on subsequent queries, potentially improving query latency significantly.

The figure below compares query latencies between two caching strategies using the ClickBench benchmark suite. The x-axis shows the query ID (0-42) and the y-axis shows query latency in milliseconds (lower is better). For each query, we measure the latency when caching the raw Parquet bytes versus caching the decoded Arrow arrays.

Takeaways

Caching Arrow consistently outperforms or matches caching Parquet across all queries.
The performance gains vary significantly:
- Scan-intensive queries (Q20-Q23) show the largest improvements, with up to 3x speedup, since they benefit directly from avoiding Parquet decoding
- Aggregation-heavy queries (Q8-Q18) see more modest gains, as their execution time is dominated by computation rather than data access
Memory usage can be a concern - Q23 triggered an out-of-memory error when caching Arrow data, highlighting its excessive memory usage.

But at what cost?

While caching Arrow data can significantly improve query performance, it comes with substantial memory overhead, as demonstrated by Q23’s out-of-memory error. The figure below compares memory usage between caching Parquet versus Arrow data across the benchmark queries. Since each query was run independently, the measurements reflect the memory requirements for executing a single query in isolation.

Takeaways

Parquet achieves roughly 4x-5x compression ratio compared to Arrow’s in-memory format
The benefit of caching Arrow data varies, but its cost is consistently 4x-5x higher memory usage.

Zoom-in a bit

Now we take a closer look at Q21 – one of the queries that benefit a lot from caching Arrow:

SELECT "SearchPhrase", MIN("URL"), COUNT(*) AS c 
FROM hits 
WHERE "URL" LIKE '%google%' 
      AND "SearchPhrase" <> '' 
GROUP BY "SearchPhrase" 
ORDER BY c 
DESC LIMIT 10;

The query scans two string columns (“URL” and “SearchPhrase”) and applies a filter on them.

Let’s now compare their query time and memory usage – we got roughly 3x speedup by using 4x more memory, not bad!

How to cache Arrow?

The figure below builds on the architecture from our previous post, adding the new Arrow cache component. The Arrow cache is positioned between the ParquetAccessPlan and the decoded Arrow RecordBatch. When a query requests data:

First, we check if the requested RecordBatch exists in the cache
If found, we can skip both fetching and decoding the Parquet data by pruning the AccessPlan
If not found, we fetch and decode the Parquet data as normal, then insert the resulting Arrow RecordBatch into the cache for future use

Although the architecture is simple, implementing it faces several challenges:

How to map Parquet byte ranges to the corresponding Arrow RecordBatches
Granularity/shape of caching – column-level vs batch-level caching.
How to efficiently test if the cached ranges contains the requested range? E.g., request range (1024, 2048), but cached ranges has [(0, 2049), (4096, 3072)].
Memory management – implementing efficient spill-to-disk strategies

We are actively working on solutions to these challenges as part of our research into high-performance Arrow caching systems. Our goal is to develop practical implementations that can be integrated into production environments. We plan to publish our findings and release the code as open source in the near future. Consider funding my research to support this work.

Standalone caching service

So far we have discussed caching within individual DataFusion instances, as shown in the Figure below (upper). While this approach works well for single-instance deployments, it is wasteful when multiple DataFusion instances access the same data.

A more scalable approach is to implement a standalone shared caching service (lower) that can be accessed by multiple DataFusion instances, with the following advantages:

Reduced resource usage - By eliminating redundant caching and decoding across instances, it optimizes both memory and CPU utilization
Improved manageability - Decoupling the caching logic from compute nodes simplifies scaling and operational management
Consistent performance - No latency spike after compute node restarts. With stateless compute nodes and a persistent centralized cache, the system avoids cache warmup delays after node restarts

Caching interface

The simplest caching interface treats the caching service as a transparent proxy for object storage. The service implements DataFusion’s AsyncFileReader trait to intercept Parquet file reads, caching byte ranges as they are accessed.

The bytes are transferred between DataFusion and the cache service using HTTP and can be implemented in any programming languages. This simple interface requires minimal changes to DataFusion.

A more sophisticated approach is to implement caching through the Arrow Flight protocol, with the caching service running as a full DataFusion instance capable of executing query plans. Rather than just serving cached bytes, this design allows the caching service to process queries directly.

When a querier needs data, it sends the complete ParquetExec physical plan to the caching service. The service can then:

Execute the plan against its cached data
Apply filters and projections directly on the cached data (filter pushdown)
Return only the necessary Arrow RecordBatches to the querier

This architecture provides several advantages: - Reduced network transfer by filtering data at the cache layer - Lower client-side CPU usage since filtering happens at the cache

The tradeoff is increased complexity in both the client and cache service implementations compared to the simple byte-range caching approach.

Caching medium

The standalone caching service allows a diverse set of storage mediums to optimize the performance and cost.

The most obvious choice is to cache data in memory, which is fast but expensive. We can do better by spilling the data to SSD when memory is full, and to HDD when SSD is full.

Conclusion

In this post, we discussed DataFusion’s caching mechanisms and explored the potential of caching Arrow data in standalone caching services. We believe that caching is a key component for every cloud-native analytics systems, and my research project is actively building systems that bridges the gap between what academia known as the state-of-the-art and what industry can actually use.

Footnotes

BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)↩︎

Parquet pruning in DataFusion

Xiangpeng Hao — Thu, 24 Oct 2024 00:00:00 GMT

Note: special thanks to InfluxData for funding this blog post.

Apache Parquet has become the industry standard for storing columnar data, and reading Parquet efficiently – especially from remote storage – is crucial for query performance.

Apache DataFusion implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query.

Achieving high performance adds complexity. This post provides an overview of the techniques used in DataFusion to selectively read Parquet files.

The pipeline

The diagram below illustrates the Parquet reading pipeline in DataFusion, highlighting how data flows through various pruning stages before being converted to Arrow format:

Background: Parquet file structure

As shown in the figure above, each Parquet file has multiple row groups. Each row group contains a set of columns, and each column contains a set of pages.

Pages are the smallest units of data in Parquet files and typically contain compressed and encoded values for a specific column. This hierarchical structure enables efficient columnar access and forms the foundation for the pruning techniques we’ll discuss.

Check out Querying Parquet with Millisecond Latency for more details on the Parquet file structure.

1. Read metadata

DataFusion first reads the Parquet metadata to understand the data in the file. Metadata often includes data schema, the exact location of each row group and column chunk, and their corresponding statistics (e.g., min/max values). It also optionally includes page-level stats and Bloom filters. This information is used to prune the file before reading the actual data.

Fetching metadata requires up to two network requests: one to read the footer size from the end of the file, and another to read the footer itself.

Decoding metadata is generally fast since it only requires parsing a small amount of data. However, for tables with hundreds or thousands of columns, the metadata can become quite large and decoding it can become a bottleneck. This is particularly noticeable when scanning many small files.

Reading metadata is latency-critical, so DataFusion allows users to cache metadata through the ParquetFileReaderFactory trait.

2. Prune by projection

The simplest yet perhaps most effective pruning is to read only the columns that are needed. This is because queries usually don’t select all columns, e.g., SELECT a FROM table only reads column a. As a columnar format, Parquet allows DataFusion to only read the columns that are needed.

This projection pruning happens at the column level and can dramatically reduce I/O when working with wide tables where queries typically access only a small subset of columns.

3. Prune by row group stats and Bloom filters

Each row group has basic stats like min/max values for each column. DataFusion applies the query predicates to these stats to prune row groups, e.g., SELECT * FROM table WHERE a > 10 will only read row groups where a has a max value greater than 10.

Sometimes min/max stats are too simple to prune effectively, so Parquet also supports Bloom filters. DataFusion uses Bloom filters when available.

Bloom filters are particularly effective for equality predicates (WHERE a = 10) and can significantly reduce the number of row groups that need to be read for point queries or queries with highly selective predicates.

4. Prune by page stats

Parquet optionally supports page-level stats – similar to row group stats but more fine-grained. DataFusion implements page pruning when the stats are present.

Page-level pruning provides an additional layer of filtering after row group pruning. It allows DataFusion to skip individual pages within a row group, further reducing the amount of data that needs to be read and decoded.

5. Read from storage

Now we (hopefully) have pruned the Parquet file into small ranges of bytes, i.e., the Access Plan. The last step is to make requests to fetch those bytes and decode them into Arrow RecordBatch.

Bonus: filter pushdown

So far we have discussed techniques that prune the Parquet file using only the metadata, i.e., before reading the actual data.

Filter pushdown, also known as predicate pushdown, is a technique that prunes data during scanning, with filters being generated and applied in the Parquet reader.

Unlike metadata-based pruning which works at the row group or page level, filter pushdown operates at the row level, allowing DataFusion to filter out individual rows that don’t match the query predicates during the decoding process.

DataFusion implements filter pushdown but has not enabled it by default due to some performance regressions.

We are working to remove the remaining performance issues and enable it by default, which we will discuss in the next blog post.

Conclusion

DataFusion employs a multi-step approach to Parquet pruning, from column projection to row group stats, page stats, and potentially row-level filtering. Each step may reduce the amount of data to be read and processed, significantly improving query performance.

Use StringView to make DataFusion faster

Xiangpeng Hao, Andrew Lamb — Fri, 13 Sep 2024 00:00:00 GMT

Editor note: This blog post is adapted from the InfluxData blog post series, make sure to also check out the original posts 1 2.

Note: Thanks to InfluxData for sponsoring this work as a summer intern project

This blog describes our experience implementing StringView in the Rust implementation of Apache Arrow, and integrating it into Apache DataFusion, significantly accelerating string-intensive queries in the ClickBench benchmark by 20%- 200% (Figure 1¹).

Getting significant end-to-end performance improvements was non-trivial. Implementing StringView itself was only a fraction of the effort required. Among other things, we had to optimize UTF-8 validation, implement unintuitive compiler optimizations, tune block sizes, and time GC to realize the FDAP ecosystem’s benefit. With other members of the open source community, we were able to overcome performance bottlenecks that could have killed the project. We would like to contribute by explaining the challenges and solutions in more detail so that more of the community can learn from our experience.

StringView is based on a simple idea: avoid some string copies and accelerate comparisons with inlined prefixes. Like most great ideas, it is “obvious” only after someone describes it clearly. Although simple, straightforward implementation actually slows down performance for almost every query. We must, therefore, apply astute observations and diligent engineering to realize the actual benefits from StringView.

Although this journey was successful, not all research ideas are as lucky. To accelerate the adoption of research into industry, it is valuable to integrate research prototypes with practical systems. Understanding the nuances of real-world systems makes it more likely that research designs will lead to practical system improvements.

StringView support was released as part of arrow-rs v52.2.0 and DataFusion v41.0.0. You can try it by setting the schema_force_string_view DataFusion configuration option, and we are hard at work with the community to make it the default. We invite everyone to try it out, take advantage of the effort invested so far, and contribute to making it better.

Figure 1: StringView improves string-intensive ClickBench query performance by 20% - 200%

Section 1: What is StringView?

The concept of inlined strings with prefixes (called “German Strings” by Andy Pavlo, in homage to TUM, where the Umbra paper that describes them originated) has been used in many recent database systems (Velox, Polars, DuckDB, CedarDB, etc.) and was introduced to Arrow as a new StringViewArray² type. Arrow’s original StringArray is very memory efficient but less effective for certain operations. StringViewArray accelerates string-intensive operations via prefix inlining and a more flexible and compact string representation.

Figure 2: Use StringArray and StringViewArray to represent the same string content.

A StringViewArray consists of three components:

The view array
The buffers
The buffer pointers (IDs) that map buffer offsets to their physical locations

Each view is 16 bytes long, and its contents differ based on the string’s length:

string length < 12 bytes: the first four bytes store the string length, and the remaining 12 bytes store the inlined string.
string length > 12 bytes: the string is stored in a separate buffer. The length is again stored in the first 4 bytes, followed by the buffer id (4 bytes), the buffer offset (4 bytes), and the prefix (first 4 bytes) of the string.

Figure 2 shows an example of the same logical content (left) using StringArray (middle) and StringViewArray (right):

The first string – “Apache DataFusion” – is 17 bytes long, and both StringArray and StringViewArray store the string’s bytes at the beginning of the buffer. The StringViewArray also inlines the first 4 bytes – “Apac” – in the view.
The second string, “InfluxDB” is only 8 bytes long, so StringViewArray completely inlines the string content in the view struct while StringArray stores the string in the buffer as well.
The third string “Arrow Rust Impl” is 15 bytes long and cannot be fully inlined. StringViewArray stores this in the same form as the first string.
The last string “Apache DataFusion” has the same content as the first string. It’s possible to use StringViewArray to avoid this duplication and reuse the bytes by pointing the view to the previous location.

StringViewArray provides three opportunities for outperforming StringArray:

Less copying via the offset + buffer format
Faster comparisons using the inlined string prefix
Reusing repeated string values with the flexible view layout

The rest of this blog post discusses how to apply these opportunities in real query scenarios to improve performance, what challenges we encountered along the way, and how we solved them.

Section 2: Faster Parquet Loading

Apache Parquet is the de facto format for storing large-scale analytical data commonly stored LakeHouse-style, such as Apache Iceberg and Delta Lake. Efficiently loading data from Parquet is thus critical to query performance in many important real-world workloads.

Parquet encodes strings (i.e., byte array) in a slightly different format than required for the original Arrow StringArray. The string length is encoded inline with the actual string data (as shown in Figure 4 left). As mentioned previously, StringArray requires the data buffer to be continuous and compact—the strings have to follow one after another. This requirement means that reading Parquet string data into an Arrow StringArray requires copying and consolidating the string bytes to a new buffer and tracking offsets in a separate array. Copying these strings is often wasteful. Typical queries filter out most data immediately after loading, so most of the copied data is quickly discarded.

On the other hand, reading Parquet data as a StringViewArray can re-use the same data buffer as storing the Parquet pages because StringViewArray does not require strings to be contiguous. For example, in Figure 4, the StringViewArray directly references the buffer with the decoded Parquet page. The string “Arrow Rust Impl” is represented by a view with offset 37 and length 15 into that buffer.

Figure 4: StringViewArray avoids copying by reusing decoded Parquet pages.

Mini benchmark
Reusing Parquet buffers is great in theory, but how much does saving a copy actually matter? We can run the following benchmark in arrow-rs to find out:

cargo bench --bench arrow_reader --features="arrow test_common experimental" "arrow_array_reader/Binary.*Array/plain encoded"

Our benchmarking machine shows that loading BinaryViewArray is almost 2x faster than loading BinaryArray (see next section about why this isn’t StringViewArray).

arrow_array_reader/BinaryArray/plain encoded                        time:   [315.86 µs 317.47 µs 319.00 µs]

arrow_array_reader/BinaryViewArray/plain encoded
time:   [162.08 µs 162.20 µs 162.32 µs]

You can read more on this arrow-rs issue: https://github.com/apache/arrow-rs/issues/5904

Section 2.1: From Binary to Strings

You may wonder why we reported performance for BinaryViewArray when this post is about StringViewArray. Surprisingly, initially, our implementation to read StringViewArray from Parquet was much slower than StringArray. Why? TLDR: Although reading StringViewArray copied less data, the initial implementation also spent much more time validating UTF-8 (as shown in Figure 5).

Strings are stored as byte sequences. When reading data from (potentially untrusted) Parquet files, a Parquet decoder must ensure those byte sequences are valid UTF-8 strings, and most programming languages, including Rust, include highly optimized routines for doing so.

Figure 5: Time to load strings from Parquet. The UTF-8 validation advantage initially eliminates the advantage of reduced copying for StringViewArray.

A StringArray can be validated in a single call to the UTF-8 validation function as it has a continuous string buffer. As long as the underlying buffer is UTF-8³, all strings in the array must be UTF-8. The Rust parquet reader makes a single function call to validate the entire buffer.

However, validating an arbitrary StringViewArray requires validating each string with a separate call to the validation function, as the underlying buffer may also contain non-string data (for example, the lengths in Parquet pages).

UTF-8 validation in Rust is highly optimized and favors longer strings (as shown in Figure 6), likely because it leverages SIMD instructions to perform parallel validation. The benefit of a single function call to validate UTF-8 over a function call for each string more than eliminates the advantage of avoiding the copy for StringViewArray.

Figure 6: UTF-8 validation throughput vs string length—StringArray’s contiguous buffer can be validated much faster than StringViewArray’s buffer.

Does this mean we should only use StringArray? No! Thankfully, there’s a clever way out. The key observation is that in many real-world datasets, 99% of strings are shorter than 128 bytes, meaning the encoded length values are smaller than 128, in which case the length itself is also valid UTF-8 (in fact, it is ASCII).

This observation means we can optimize validating UTF-8 strings in Parquet pages by treating the length bytes as part of a single large string as long as the length value is less than 128. Put another way, prior to this optimization, the length bytes act as string boundaries, which require a UTF-8 validation on each string. After this optimization, only those strings with lengths larger than 128 bytes (less than 1% of the strings in the ClickBench dataset) are string boundaries, significantly increasing the UTF-8 validation chunk size and thus improving performance.

The actual implementation is only nine lines of Rust (with 30 lines of comments). You can find more details in the related arrow-rs issue: https://github.com/apache/arrow-rs/issues/5995. As expected, with this optimization, loading StringViewArray is almost 2x faster than loading StringArray.

Section 2.2: Be Careful About Implicit Copies

After all the work to avoid copying strings when loading from Parquet, performance was still not as good as expected. We tracked the problem to a few implicit data copies that we weren’t aware of, as described in this issue.

The copies we eventually identified come from the following innocent-looking line of Rust code, where self.buf is a reference counted pointer that should transform without copying into a buffer for use in StringViewArray.

let block_id = output.append_block(self.buf.clone().into());

However, Rust-type coercion rules favored a blanket implementation that did copy data. This implementation is shown in the following code block where the impl> will accept any type that implements AsRef<[u8]> and copies the data to create a new buffer. To avoid copying, users need to explicitly call from_vec, which consumes the Vec and transforms it into a buffer.

impl<T: AsRef<[u8]>> From<T> for Buffer {
    fn from(p: T) -> Self {
        // copies data here
     ...
    }
}
impl Buffer { 
  pub fn from_vec<T>(data: Vec<T>) -> Self {
// zero-copy transformation
...
  }
}

Diagnosing this implicit copy was time-consuming as it relied on subtle Rust language semantics. We needed to track every step of the data flow to ensure every copy was necessary. To help other users and prevent future mistakes, we also removed the implicit API from arrow-rs in favor of an explicit API. Using this approach, we found and fixed several other unintentional copies in the code base—hopefully, the change will help other downstream users avoid unnecessary copies.

Section 2.3: Help the Compiler by Giving it More Information

The Rust compiler’s automatic optimizations mostly work very well for a wide variety of use cases, but sometimes, it needs additional hints to generate the most efficient code. When profiling the performance of view construction, we found, counterintuitively, that constructing long strings was 10x faster than constructing short strings, which made short strings slower on StringViewArray than on StringArray!

As described in Section 1, StringViewArray treats long and short strings differently. Short strings (<12 bytes) directly inline to the view struct, while long strings only inline the first 4 bytes. The code to construct a view looks something like this:

if len <= 12 {
   // Construct 16 byte view for short string
   let mut view_buffer = [0; 16];
   view_buffer[0..4].copy_from_slice(&len.to_le_bytes());
   view_buffer[4..4 + data.len()].copy_from_slice(data);
   ...
} else {      
   // Construct 16 byte view for long string
   ByteView {
       length: len,
       prefix: u32::from_le_bytes(data[0..4].try_into().unwrap()),
       buffer_index: block_id,
       offset,
   }
}

It appears that both branches of the code should be fast: they both involve copying at most 16 bytes of data and some memory shift/store operations. How could the branch for short strings be 10x slower?

Looking at the assembly code using godbolt, we (with help from Ao Li) found the compiler used CPU load instructions to copy the fixed-sized 4 bytes to the view for long strings, but it calls a function, ptr::copy_non_overlapping, to copy the inlined bytes to the view for short strings. The difference is that long strings have a prefix size (4 bytes) known at compile time, so the compiler directly uses efficient CPU instructions. But, since the size of the short string is unknown to the compiler, it has to call the general-purpose function ptr::copy_non_coverlapping. Making a function call is significant unnecessary overhead compared to a CPU copy instruction.

However, we know something the compiler doesn’t know: the short string size is not arbitrary—it must be between 0 and 12 bytes, and we can leverage this information to avoid the function call. Our solution generates 13 copies of the function using generics, one for each of the possible prefix lengths. The code looks as follows, and checking the assembly code, we confirmed there are no calls to ptr::copy_non_overlapping, and only native CPU instructions are used. For more details, see the ticket.

fn make_inlined_view<const LEN: usize>(data: &[u8]) -> u128 {
     let mut view_buffer = [0; 16];
     view_buffer[0..4].copy_from_slice(&(LEN as u32).to_le_bytes());
     view_buffer[4..4 + LEN].copy_from_slice(&data[..LEN]);
     u128::from_le_bytes(view_buffer)
}
pub fn make_view(data: &[u8], block_id: u32, offset: u32) -> u128 {
     let len = data.len();
     // generate special code for each of the 13 possible lengths
     match len {
         0 => make_inlined_view::<0>(data),
         1 => make_inlined_view::<1>(data),
         2 => make_inlined_view::<2>(data),
         3 => make_inlined_view::<3>(data),
         4 => make_inlined_view::<4>(data),
         5 => make_inlined_view::<5>(data),
         6 => make_inlined_view::<6>(data),
         7 => make_inlined_view::<7>(data),
         8 => make_inlined_view::<8>(data),
         9 => make_inlined_view::<9>(data),
         10 => make_inlined_view::<10>(data),
         11 => make_inlined_view::<11>(data),
         12 => make_inlined_view::<12>(data),
         _ => {
           // handle long string
}}}

Section 2.4: End-to-End Query Performance

In the previous sections, we went out of our way to make sure loading StringViewArray is faster than StringArray. Before going further, we wanted to verify if obsessing about reducing copies and function calls has actually improved end-to-end performance in real-life queries. To do this, we evaluated a ClickBench query (Q20) in DataFusion that counts how many URLs contain the word "google":

SELECT COUNT(*) FROM hits WHERE "URL" LIKE '%google%';

This is a relatively simple query; most of the time is spent on loading the “URL” column to find matching rows. The query plan looks like this:

Projection: COUNT(*) [COUNT(*):Int64;N]
  Aggregate: groupBy=[[]], aggr=[[COUNT(*)]] [COUNT(*):Int64;N]
    Filter: hits.URL LIKE Utf8("%google%")
      TableScan: hits

We ran the benchmark in the DataFusion repo like this:

cargo run --profile release-nonlto --bin dfbench -- clickbench --queries-path benchmarks/queries/clickbench/queries.sql --iterations 3 --query 20 --path benchmarks/data/hits.parquet --string-view

With StringViewArray we saw a 24% end-to-end performance improvement, as shown in Figure 7. With the --string-view argument, the end-to-end query time is 944.3 ms, 869.6 ms, 861.9 ms (three iterations). Without --string-view, the end-to-end query time is 1186.1 ms, 1126.1 ms, 1138.3 ms.

Figure 7: StringView reduces end-to-end query time by 24% on ClickBench Q20.

We also double-checked with detailed profiling and verified that the time reduction is indeed due to faster Parquet loading.

Section 3: Faster String Operations

We have discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies. In this second part of the post, we describe the rest of the journey: implementing additional efficient operations for real query processing.

Section 3.1: Faster comparison

String comparison is ubiquitous; it is the core of cmp, min/max, and like/ilike kernels. StringViewArray is designed to accelerate such comparisons using the inlined prefix—the key observation is that, in many cases, only the first few bytes of the string determine the string comparison results.

For example, to compare the strings InfluxDB with Apache DataFusion, we only need to look at the first byte to determine the string ordering or equality. In this case, since A is earlier in the alphabet than I, Apache DataFusion sorts first, and we know the strings are not equal. Despite only needing the first byte, comparing these strings when stored as a StringArray requires two memory accesses: 1) load the string offset and 2) use the offset to locate the string bytes. For low-level operations such as cmp that are invoked millions of times in the very hot paths of queries, avoiding this extra memory access can make a measurable difference in query performance.

For StringViewArray, typically, only one memory access is needed to load the view struct. Only if the result can not be determined from the prefix is the second memory access required. For the example above, there is no need for the second access. This technique is very effective in practice: the second access is never necessary for the more than 60% of real-world strings which are shorter than 12 bytes, as they are stored completely in the prefix.

However, functions that operate on strings must be specialized to take advantage of the inlined prefix. In addition to low-level comparison kernels, we implemented a wide range of other StringViewArray operations that cover the functions and operations seen in ClickBench queries. Supporting StringViewArray in all string operations takes quite a bit of effort, and thankfully the Arrow and DataFusion communities are already hard at work doing so (see https://github.com/apache/datafusion/issues/11752 if you want to help out).

Section 3.2: Faster `take` and `filter`

After a filter operation such as WHERE url <> ‘’ to avoid processing empty urls, DataFusion will often coalesce results to form a new array with only the passing elements. This coalescing ensures the batches are sufficiently sized to benefit from vectorized processing in subsequent steps.

The coalescing operation is implemented using the take and filter kernels in arrow-rs. For StringArray, these kernels require copying the string contents to a new buffer without “holes” in between. This copy can be expensive especially when the new array is large.

However, take and filter for StringViewArray can avoid the copy by reusing buffers from the old array. The kernels only need to create a new list of views that point at the same strings within the old buffers. Figure 8 illustrates the difference between the output of both string representations. StringArray creates two new strings at offsets 0-17 and 17-32, while StringViewArray simply points to the original buffer at offsets 0 and 25.

Figure 8: Zero-copy take/filter for StringViewArray

Section 3.3: When to GC?

Zero-copy take/filter is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any view. This leads to excessive memory usage, especially in a filter-then-coalesce scenario. For example, a StringViewArray with 10M strings may only refer to 1M strings after some filter operations; however, due to zero-copy take/filter, the (reused) 10M buffers can not be released/reused.

To release unused memory, we implemented a garbage collection (GC) routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC too late, we hold large buffers for too long, increasing memory use and decreasing cache efficiency. The Polars blog on StringView also refers to the challenge presented by garbage collection timing.

arrow-rs implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the CoalseceBatchesExec operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, which aligns perfectly with the scenario of GC in StringViewArray. We, therefore, implemented the GC procedure inside CoalseceBatchesExec[^5],with a heuristic that estimates when the buffers are too sparse.

Section 3.4: The art of function inlining: not too much, not too little

Like string inlining, function inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save). Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the #[inline(always)] directive. In performance-critical code, inlined code allows us to organize large functions into smaller ones without paying the runtime cost of function invocation.

However, function inlining is not always better, as it leads to larger function bodies that are harder for LLVM to optimize (for example, suboptimal register spilling) and risk overflowing the CPU’s instruction cache. We observed several performance regressions where function inlining caused slower performance when implementing the StringViewArray comparison kernels. Careful inspection and tuning of the code was required to aid the compiler in generating efficient code. More details can be found in this PR: https://github.com/apache/arrow-rs/pull/5900.

Section 3.5: Buffer size tuning

StringViewArray permits multiple buffers, which enables a flexible buffer layout and potentially reduces the need to copy data. However, a large number of buffers slows down the performance of other operations. For example, get_array_memory_size() needs to sum the memory size of each buffer, which takes a long time with thousands of small buffers. In certain cases, we found that multiple calls to concat_batches lead to arrays with millions of buffers, which was prohibitively expensive.

For example, consider a StringViewArray with the previous default buffer size of 8 KB. With this configuration, holding 4GB of string data requires almost half a million buffers! Larger buffer sizes are needed for larger arrays, but we cannot arbitrarily increase the default buffer size, as small arrays would consume too much memory (most arrays require at least one buffer). Buffer sizing is especially problematic in query processing, as we often need to construct small batches of string arrays, and the sizes are unknown at planning time.

To balance the buffer size trade-off, we again leverage the query processing (DataFusion) semantics to decide when to use larger buffers. While coalescing batches, we combine multiple small string arrays and set a smaller buffer size to keep the total memory consumption low. In string aggregation, we aggregate over an entire Datafusion partition, which can generate a large number of strings, so we set a larger buffer size (2MB).

To assist situations where the semantics are unknown, we also implemented a classic dynamic exponential buffer size growth strategy, which starts with a small buffer size (8KB) and doubles the size of each new buffer up to 2MB. We implemented this strategy in arrow-rs and enabled it by default so that other users of StringViewArray can also benefit from this optimization. See this issue for more details: https://github.com/apache/arrow-rs/issues/6094.

Section 3.6: End-to-end query performance

We have made significant progress in optimizing StringViewArray filtering operations. Now, let’s test it in the real world to see how it works!

Let’s consider ClickBench query 22, which selects multiple string fields (URL, Title, and SearchPhase) and applies several filters.

SELECT 
  "SearchPhrase", 
  MIN("URL"), MIN("Title"), COUNT(*) AS c, COUNT(DISTINCT "UserID") 
FROM hits 
WHERE 
  "Title" LIKE '%Google%' AND 
  "URL" NOT LIKE '%.google.%' AND 
  "SearchPhrase" <> '' 
GROUP BY "SearchPhrase" 
ORDER BY c DESC 
LIMIT 10;

We ran the benchmark using the following command in the DataFusion repo. Again, the --string-view option means we use StringViewArray instead of StringArray.

cargo run --profile release-nonlto --bin dfbench -- clickbench --queries-path benchmarks/queries/clickbench/queries.sql --iterations 3 --query 22 --path benchmarks/data/hits.parquet --string-view

To eliminate the impact of the faster Parquet reading using StringViewArray (see the first part of this blog), Figure 9 plots only the time spent in FilterExec. Without StringViewArray, the filter takes 7.17s; with StringViewArray, the filter only takes 4.86s, a 32% reduction in time. Moreover, we see a 17% improvement in end-to-end query performance.

Figure 9: StringViewArray reduces the filter time by 32% on ClickBench query 22.

Section 4: Faster String Aggregation

So far, we have discussed how to exploit two StringViewArray features: reduced copy and faster filtering. This section focuses on reusing string bytes to repeat string values.

As described in part one of this blog, if two strings have identical values, StringViewArray can use two different views pointing at the same buffer range, thus avoiding repeating the string bytes in the buffer. This makes StringViewArray similar to an Arrow DictionaryArray that stores Strings—both array types work well for strings with only a few distinct values.

Deduplicating string values can significantly reduce memory consumption in StringViewArray. However, this process is expensive and involves hashing every string and maintaining a hash table, and so it cannot be done by default when creating a StringViewArray. We introduced an opt-in string deduplication mode in arrow-rs for advanced users who know their data has a small number of distinct values, and where the benefits of reduced memory consumption outweigh the additional overhead of array construction.

Once again, we leverage DataFusion query semantics to identify StringViewArray with duplicate values, such as aggregation queries with multiple group keys. For example, some ClickBench queries group by two columns:

UserID (an integer with close to 1 M distinct values)
MobilePhoneModel (a string with less than a hundred distinct values)

In this case, the output row count is count(distinct UserID) * count(distinct MobilePhoneModel), which is 100M. Each string value of MobilePhoneModel is repeated 1M times. With StringViewArray, we can save space by pointing the repeating values to the same underlying buffer.

Faster string aggregation with StringView is part of a larger project to improve DataFusion aggregation performance. We have a proof of concept implementation with StringView that can improve the multi-column string aggregation by 20%. We would love your help to get it production ready!

Section 5: StringView Pitfalls

Most existing blog posts (including this one) focus on the benefits of using StringViewArray over other string representations such as StringArray. As we have discussed, even though it requires a significant engineering investment to realize, StringViewArray is a major improvement over StringArray in many cases.

However, there are several cases where StringViewArray is slower than StringArray. For completeness, we have listed those instances here:

Tiny strings (when strings are shorter than 8 bytes): every element of the StringViewArray consumes at least 16 bytes of memory—the size of the view struct. For an array of tiny strings, StringViewArray consumes more memory than StringArray and thus can cause slower performance due to additional memory pressure on the CPU cache.
Many repeated short strings: Similar to the first point, StringViewArray can be slower and require more memory than a DictionaryArray because 1) it can only reuse the bytes in the buffer when the strings are longer than 12 bytes and 2) 32-bit offsets are always used, even when a smaller size (8 bit or 16 bit) could represent all the distinct values.
Filtering: As we mentioned above, StringViewArrays often consume more memory than the corresponding StringArray, and memory bloat quickly dominates the performance without GC. However, invoking GC also reduces the benefits of less copying so must be carefully tuned.

Section 6: Conclusion and Takeaways

In these two blog posts, we discussed what it takes to implement StringViewArray in arrow-rs and then integrate it into DataFusion. Our evaluations on ClickBench queries show that StringView can improve the performance of string-intensive workloads by up to 2x.

Given that DataFusion already performs very well on ClickBench, the level of end-to-end performance improvement using StringViewArray shows the power of this technique and, of course, is a win for DataFusion and the systems that build upon it.

StringView is a big project that has received tremendous community support. Specifically, we would like to thank @tustvold, @ariesdevil, @RinChanNOWWW, @ClSlaid, @2010YOUY01, @chloro-pn, @a10y, @Kev1n8, @Weijun-H, @PsiACE, @tshauck, and @xinlifoobar for their valuable contributions!

As the introduction states, “German Style Strings” is a relatively straightforward research idea that avoid some string copies and accelerates comparisons. However, applying this (great) idea in practice requires a significant investment in careful software engineering. Again, we encourage the research community to continue to help apply research ideas to industrial systems, such as DataFusion, as doing so provides valuable perspectives when evaluating future research questions for the greatest potential impact.

Footnotes

Benchmarked with AMD Ryzen 7600x (12 core, 24 threads, 32 MiB L3), WD Black SN770 NVMe SSD (5150MB/4950MB seq RW bandwidth)↩︎
There is also a corresponding BinaryViewArray which is similar except that the data is not constrained to be UTF-8 encoded strings.↩︎
We also make sure that offsets do not break a UTF-8 code point, which is cheaply validated.↩︎

My research statement

Wed, 21 Aug 2024 00:00:00 GMT

North star

Research should connect to real people.

Impact definition

Research impact is defined by the number of connections between the paper and the users.

Note that the research paper itself does not produce an impact, as nobody’s life is changed because of reading a paper—we are not philosophers. This means that the research impact is always indirect; we have to apply the research to a system with real users.

Due to the indirect nature of the research’s impact, some effort must be made to connect the research to users.

My take: Researchers should take a holistic approach, seamlessly threading their work from the research paper to individual users.

No-goals

No goals help define goals. I’ve made all three types of mistakes, and this is my self-reflection to avoid repeating them.

We are tempted to work on projects proven to be publishable, e.g., improving an existing published method. However, it is not uncommon to see chains of research papers working on a topic irrelevant to the rest of the world. In other words, the impact of a research paper is not defined by how many papers cite it.

Researchers have the privilege to go wild and think big. But this privilege can be misused. Often, it is used to fool ourselves. Some research (e.g., physics and math) are/can be far from users, but not computer science – one of applied science. As applied scientists, we must ensure our “science” applies to the real world instead of just intellectual exercise.
Some research (e.g., theory) in applied science still enjoys greater freedom than others. However, my research topics are outside them.

This is the most common research mistake. We often believe that researchers should focus on making good ideas, and someone else will go off to make it happen. This is not true. There are way too many good ideas, yet few are impactful.

Most ideas are just combinations of existing techniques, and it is hard to believe anything fundamental will be discovered in system research.

Most importantly, a good idea evolves with the system it builds upon. It is a feedback loop that tries, fails, and repeats. And the great idea that comes out of the system always differs from its initial version. It is never the case that the research flow is a single-directional graph with no feedback from downstream users.

Today’s problem vs tomorrow’s problem

Both types of problems are important and meaningful. I don’t work on tomorrow’s problems (but I agree that someone must work on future problems).

Why not work on future problems:

We have enough of today’s problems to work on.
It’s easier to fool ourselves while working on future problems.

Don’t fool ourselves:

We like tomorrow’s problem because it’s easier to claim novelty – future problems naturally come with novelty.
We like tomorrow’s problem because we don’t have to establish user connections. Instead, we use it as an excuse to avoid working on the most important and challenging parts. We are free to propose anything we want, work around any challenges, and eventually disconnect from the rest of the world.
How do we avoid imaginary research? How can we ensure today’s solution is not funny when tomorrow’s problem arrives? Why work on it when future researchers will beat today’s solution with much more practical considerations?

Engineering problem vs research problem

My take on research vs engineering problems:

All engineering problems are meaningful as they come from users and practitioners—no one invented them out of thin air.
All research problems are publishable by definition.
Most engineering problems are also research problems, evidenced by the fact that they haven’t been solved — meaning they are challenging, resource-consuming, or require a deep understanding of practical nuances, which are the definitions of research problems.
Research problems, without being grounded by engineering problems, are unlikely to be meaningful.

Why this statement?

I probably only need to publish one more paper to graduate, as I have already published two papers in VLDB/SIGMOD. Moreover, my chance to stay in academia is quite low, meaning that this one-more-paper is likely the last paper I drive to publish in my career.

I have lots of cool ideas, but I have to choose only one. This statement helps me to crystalize my thoughts when deciding what to work on. It would be great if I knew all of these early in my Ph.D., but it is still not too late to write them down.

I can get lost when struggling to publish a paper or being determined to secure a faculty job. Everyone starts with the intention of doing meaningful research, but practical considerations forced them to do otherwise. If I encounter those situations (again), this statement would remind me of my north star.

InfluxData > Google and Microsoft

Sun, 18 Aug 2024 00:00:00 GMT

I have interned at Google, Microsoft (Research), and InfluxData. The experience at InfluxData has been the most enjoyable, and this blog post explains why.

Smaller companies are more connected to the real business, more connected within the company, and more willing to connect you to the rest of the world.

Problem definition

Non-problems

I have worked on great projects: (1) auto-tuned BigTable, (2) designed a new B-Tree system, and (3) implemented StringView in DataFusion.

I have met great people: Yixin Luo, Badrish Chandramouli, and Andrew Lamb. They are professional and helpful, and I learned a lot from them.

The problem

The problem is connectivity: Is the project connected to the real product/customer? Am I connected to the company? Am I connected to the rest of the world?

Big tech has systematic cultural issues that prevent each individual from being connected, and there’s no easy way to fix it.

Connect to business

I have constantly been reminded to connect my intern project to the InfluxData business – how it improves InfluxDB3.0, what Influx queries it can accelerate, how it compares with the current approaches, etc.

Andrew expects my project to deploy in production rather than behind a feature gate that no one uses, and he worked very hard to make it happen. Andrew also reminded me to connect my presentation to InfluxDB3.0 and discuss how it benefits the product.

This is an important feeling of real – I’m not being paid to work on a toy exploration, not a project that requires five extra teams to reach customers. Instead, it is a project grounded by real systems and real needs, and it will be running as part of a real product.

Being real means my time and efforts are respected and valued. Most importantly, it means the investment is sustainable because some kind of return is expected.

Connect to peers

Despite being a remote-only company, InfluxData is highly connected: cross-team collaborations, company-wide acknowledgments for great work, and short turn-around for approval.

I only learned to appreciate the above after seeing teams fighting for the same project, achievements downplayed due to politics, and bureaucratic processes blocking individual growth.

Connecting to peers means working with real people, rather than a faceless bureaucratic system. You work towards a shared goal, not a surprise order from high above. When unexpected happens, you know who to talk to, and you can reason the consequences by common sense, not with a rule book or guessing what the boss wants.

Being treated as a respected individual, rather than some resource/tool/refillable, is sometimes more important than being paid well.

Connect to the world

Big techs have trust and ego issues. They develop/control their own programming languages, frameworks, platforms, etc. They do not sincerely share with the rest of the world, and they design procedures to prevent individuals from doing so. This means your work is only visible to a small group of people, even if it could have a much broader impact.

At InfluxData, I have seen the opposite. Not only is all of my work open source, but the company is actively helping me share it on the company blog, X, and LinkedIn. Andrew even took me to a DataFusion meetup and introduced me to the larger FDAP community. (Meanwhile, Microsoft asks students to pay the travel fee to present company work at conferences)

Open source at InfluxData means not simply moving the repo to the public domain but sincerely engaging with the community, cultivating an environment where everyone is welcome to contribute, and making sure the goals and intentions are clear and well-communicated. As an example, Andrew and I have spent an extensive amount of time writing the blog post and trying our best to share what we have learned with the community, while we could have just merged the PR and moved forward.

What prevents a connected culture?

Small companies don’t automatically gain a connected culture, but I can tell that InfluxData and Andrew have made deliberate efforts to make it happen.

Large companies, on the other hand, have systematic issues that prevent this connected culture. Even though every individual I met at Google and Microsoft was nice and kind, there’s little they can do to fix the problem from the faceless bureaucratic system – you can’t even name a person to blame; it is the procedure.

One of my dreams is to create an employee-first company that prioritizes respect for individuals, explores new ways to organize and connect people, and focuses on maximizing their happiness and creativity rather than just pursuing profit.

(I’d like to thank Xuanwo for encouraging me to finish this blog post!)

Three types of good academic advisors

Tue, 07 May 2024 00:00:00 GMT

Good advisors

Innovators, educators, and entrepreneurs.

Innovator: good at finding the right thing to do. They have a clear vision of what important problems are. They are often smart people who deeply understand the problems they work on. Students learn a lot about their methodologies, visions, and ways of thinking/reasoning. Example: Mike Stonebraker.

Educator: good at teaching and empowering the students. They empower their student to reach their potential and to find what they are good at. They motivate their students and ignite their passions. Example: Remzi H. Arpaci-Dusseau(likely also Andrea C. Arpaci-Dusseau, but I haven’t worked with her).

Entrepreneur: good at resource acquisition and allocation. Students have enough resources to reach their potential. They connect people and match the right students with the right resources. Example: Aditya Akella.

A good advisor doesn’t necessarily have to be in only one category (e.g., advisors above span multiple categories). Still, a good advisor often has to be in at least one of the categories.

I’m privileged enough to have worked with all three types of good advisors; however, finding a good advisor requires a lot of luck and effort. Reality check: many faculty members (especially in prestigious universities) are not in any of the categories, and among them, most don’t even try to be a good advisor.

Signs of bad advisors

All happy families are alike; each unhappy family is unhappy in its own way. – Leo Tolstoy

Like unhappy families, bad advisors are pretty novel in their ways of being bad; we don’t summarize them here. Instead, we list some key symptoms of working with a bad advisor.

Long time-to-recovery, i.e., too long to recover from a meeting with your advisor.

Regular meetings with your advisor can be stressful; often the time you need to prepare many slides, teach your advisor about your work, and try to get some feedback. Often, especially in the early stages of your Ph.D., you don’t have good presentation/communication skills to explain your work to your advisor. Your advisor can get impatient and think you are not progressing or working hard enough.

This can be frustrating and demotivating, and this pattern repeats every week. The question is, how long does it take to recover from this meeting? I have experienced that it can take a few hours, or even an entire day, to recover from the frustration and continue to work on my research. However, it does not have to be this way; a good advisor can motivate students and give concrete feedback to help them improve.

Not being treated like a human, but a tool/machine to generate papers.

Graduate students are first humans, then students, and then researchers, but never tools/machines to generate papers.

Most bad advisors treat their students as leverage to fulfill their own goals. They often have very concrete expectations from their students: publish X papers in Y top-tier conferences/journals and work on Z topics with W skill sets. They evaluate their student by whether they can meet those expectations; even worse, those expectations are often inexplicit and change over time.

If students lag behind expectations – which happens all the time – they are often blamed for not working hard enough or not being smart enough. Worse, they will be threatened to delay graduation, withdraw funding, or even be kicked out of the program. Even if students are on track, they are often not appreciated for their hard work and dedication. They are given more work to do, and the cycle repeats.

Graduate study is a long journey; without being treated like a real human, you feel suffering, lonely, and lost. But you deserve a supportive, fruitful, and enjoyable PhD study.

Not aware of power dynamics, but you are my boss.

Many people dreamed an advisor-advisee relationship would be like a marriage partnership: they respect, trust, and work together to achieve common goals.

But that is never the case in reality; advisors pay their students’ salaries, and they can decide when they can graduate. The conversation between an advisor and a student is never equal, and the power dynamics are always there.

The real challenge is whether the advisor is aware of these power dynamics and how they use them. When in disagreement, does the student have the real freedom to say no? Does the advisor sincerely feel comfortable to be challenged? Navigating these power dynamics is art for both students and advisors, but it’s on the advisors to practice and ensure students feel safe and respected.

Self-reflections

It’s easy to blame bad advisors, but it is hard not to become one.

I suffered a lot from bad advisors, but I often wonder if I can be better than them. Toxic advisors result from systematic academia problems, and changing the system is hard. That’s why good advisors are rare and worth being recognized and appreciated.

When pressured with tenure, funding, and publication, will I be as nice/patient/motivating/caring as I imagined myself to be?

What happens when you type a SQL in the database

Fri, 26 Apr 2024 00:00:00 GMT

Preface

A database can be complex; it involves almost all aspects (research communities) of computer science: PL (programming language), SE (software engineering), OS (operating system), networking, storage, theory; more recently, NLP (natural language processing), and ML (machine learning). The database community is centered around the people interested in making the database (the product) better instead of pure intellectual/research interests; it is, therefore, a practical and multi-disciplinary field. This makes databases awesome but also hard to learn.

As complex as it is, the boundaries of the building blocks within a database are clear after decades of research and real-world operations. The recent (and state-of-the-art) Apache DataFusion project is a good example of building a database using well-defined industry standards like Apache Arrow, and Apache Parquet. Without home-grown solutions for storage and in-memory representation, DataFusion can be comparable or even better than alternatives like DuckDB.

This document aims to explain these well-defined boundaries, namely, how query engines (i.e., OLAP) transform a plain SQL query into the results we want, how every step works, and how they are connected.

flowchart LR
   id1[SQL text] --> |SQL parser| id2[SQL statement] 
   id2 --> |Query planner| id3[Logical plan] --> |Query optimizer| id4[Optimized logical plan] --> |Physical planner| id5
   id5[Physical plan] --> |Execution| id7[Output]

Note

This is a blog post I hoped I knew when I was younger.

I aim to make multi-year efforts to edit and improve it as I learn more about databases. I sometimes dreamed that this post could evolve to be the database equivalent of the OSTEP book (it might be too ambitious, though).

Section 1: End-To-End View

Input

Table definition

We have the following two tables (adapted from TPC-H spec): lineitem and orders. The lineitem defines the the shipment dates, while the order defines order details.

erDiagram
  lineitem {
      int l_orderkey
      int l_linenumber
      date l_shipdate
      date l_commitdate
      date l_receiptdate
      string l_shipmode
      string l_comment
  }
  orders {
      int o_orderkey
      date o_orderdate
      string o_orderpriority
      string o_clerk
      string o_comment
  }

SQL query

Let’s say we have this simple query (adapted from TPC-H query 5), which finds the l_orderkey, l_shipdate, and o_orderdate of orders that were placed in 1994.

SELECT
    l_orderkey, l_shipdate, o_orderdate
FROM
    orders
JOIN
    lineitem ON l_orderkey = o_orderkey
WHERE
    o_orderdate >= DATE '1994-01-01'
    AND o_orderdate < DATE '1995-01-01';

Output

The query is pretty simple; it joins two tables on the order key and then filters the results based on the order date. If everything goes well, we should get results similar to this:

+------------+------------+-------------+
| l_orderkey | l_shipdate | o_orderdate |
+------------+------------+-------------+
| 1          | 1994-06-01 | 1994-05-01  |
+------------+------------+-------------+

Section 2: Parsing

I skipped it for now as it is mostly orthogonal to the data system pipelines.

Input

The SQL query text.

Output

Structured statement from the SQL (significantly simplified for brevity):

from: [
  TableWithJoins {
    relation: Table {
      name: ObjectName([
        Ident {
          value: "orders",
          quote_style: None,
        },
      ]),
    },
    joins: [
      Join {
        relation: Table {
          name: ObjectName([
            Ident {
              value: "lineitem",
              quote_style: None,
            },
          ]),
        },
        join_operator: Inner(
          On(
            BinaryOp {
              left: Identifier(
                Ident {
                  value: "l_orderkey",
                  quote_style: None,
                },
              ),
              op: Eq,
              right: Identifier(
                Ident {
                  value: "o_orderkey",
                  quote_style: None,
                },
              ),
            },
          ),
        ),
      },
    ],
  },
],
selection: Some(
  BinaryOp {
    left: BinaryOp {
      left: Identifier(
        Ident {
          value: "o_orderdate",
          quote_style: None,
        },
      ),
      op: GtEq,
      right: TypedString {
        data_type: Date,
        value: "1994-01-01",
      },
    },
    op: And,
    right: BinaryOp {
      left: Identifier(
        Ident {
          value: "o_orderdate",
          quote_style: None,
        },
      ),
      op: Lt,
      right: TypedString {
        data_type: Date,
        value: "1995-01-01",
      },
    },
  },
),

Section 3: Query Planning

Input

The query statement from the last step.

Output

The logical query plan is something like this:

Projection: lineitem.l_orderkey, lineitem.l_shipdate, orders.o_orderdate
  Filter: orders.o_orderdate >= CAST(Utf8("1994-01-01") AS Date32) AND orders.o_orderdate < CAST(Utf8("1995-01-01") AS Date32)
    Inner Join:  Filter: lineitem.l_orderkey = orders.o_orderkey
      TableScan: orders
      TableScan: lineitem

Plot it as a tree.

Logical vs physical.

Todo: describe why we must distinguish between physical and logical plans.

Section 4: Query Optimizing

Input

The (unoptimized) logical plan from the last step.

Output

An optimized logical plan.

Projection: lineitem.l_orderkey, lineitem.l_shipdate, orders.o_orderdate
  Inner Join: orders.o_orderkey = lineitem.l_orderkey
    Filter: orders.o_orderdate >= Date32("8766") AND orders.o_orderdate < Date32("9131")
      TableScan: orders projection=[o_orderkey, o_orderdate], partial_filters=[orders.o_orderdate >= Date32("8766"), orders.o_orderdate < Date32("9131")]
    TableScan: lineitem projection=[l_orderkey, l_shipdate]

Note the difference between an unoptimized and an optimized plan! The Filter has been pushed down to lower-level nodes. Part of the projection has been embedded in the TableScan.

Section 5: Physical Planning

Input

A logical plan.

Output

A physical plan. Unlike logical plans, physical plans are more concrete about what to do; here’s an example:

Physical plan:
ProjectionExec: expr=[l_orderkey@1 as l_orderkey, l_shipdate@2 as l_shipdate, o_orderdate@0 as o_orderdate]
  CoalesceBatchesExec: target_batch_size=8192
    HashJoinExec: mode=Partitioned, join_type=Inner, on=[(o_orderkey@0, l_orderkey@0)], projection=[o_orderdate@1, l_orderkey@2, l_shipdate@3]
      CoalesceBatchesExec: target_batch_size=8192
        RepartitionExec: partitioning=Hash([o_orderkey@0], 8), input_partitions=8
          CoalesceBatchesExec: target_batch_size=8192
            FilterExec: o_orderdate@1 >= 8766 AND o_orderdate@1 < 9131
              RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1
                CsvExec: file_groups={1 group: [[Users/xiangpeng/work/coding/db-ml/bin/example-data/orders.csv]]}, projection=[o_orderkey, o_orderdate], has_header=true
      CoalesceBatchesExec: target_batch_size=8192
        RepartitionExec: partitioning=Hash([l_orderkey@0], 8), input_partitions=8
          RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1
            CsvExec: file_groups={1 group: [[Users/xiangpeng/work/coding/db-ml/bin/example-data/lineitem.csv]]}, projection=[l_orderkey, l_shipdate], has_header=true

We can also plot a physical plan to a tree graph:

Note

Note that a physical plan has much more details than a logical plan; it contains everything needed to execute the query!

(Optional: we often have physical optimizers that optimize on a physical plan. Omitted here for simplicity)

Section 6: Query Execution

Input

A physical plan

Output

The final output is like this:

+------------+------------+-------------+
| l_orderkey | l_shipdate | o_orderdate |
+------------+------------+-------------+
| 1          | 1994-06-01 | 1994-05-01  |
+------------+------------+-------------+

Execution order

The simplest execution model is pull-based execution, which implements a post-order traversal of the physical plan. For a tree (like blow), we get a traversal order of D -> E -> B -> F -> G -> C -> A:

Applying our physical graph above, we get an execution order of:

The RepartitionExec and CoalesceBatchesExec are executors that partition the data for multi-thread processing (based on the Volcano execution style).

A simplified, single-threaded, no-partitioned execution order would be:

graph LR;
    e1["CsvExec (orders.csv)"] --> FilterExec
    FilterExec --> e2 
    e2["CsvExec (lineitem.csv)"] --> HashJoinExec
    HashJoinExec --> ProjectionExec

Reading from disk

CSV files are row-based, and we read them row by row, it is efficient when we frequently need to read the whole row. However, modern data analytic workloads do not always need to read the whole row; they often only need to read a subset of columns. In our example above, we only need to read l_orderkey, l_shipdate, o_orderdate, o_orderkey from the tables. If using a row-based file format (like CSV), we need to load all columns into memory, which is inefficient. Column-based file formats (like Apache Parquet) can be more efficient in this case.

See the Parquet pruning in DataFusion for more details.

Xiangpeng's blog

A system programmer’s guide to LLM inference

How to read the name? Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

What’s inside the file?

The model architecture

Token <–> String

From token to string

From string to token

Four mistakes during my PhD

Mistake 1: Giving up too late

Mistake 2: Getting real too late

Mistake 3: Care too much about paper/research community

Mistake 4: Lost in the Church of Reason

Conclusion

parquet-linter: A better Parquet is Parquet itself

Parquet is a spec, not an implementation

We need different kinds of Parquet

Use case 1: long term archival

Use case 2: fast query performance

Use case 3: ecosystem compatibility

Vision: Parquet Linter

Level 1: pure gain

Level 2: trade-offs guided

Level 3: towards intelligence

Does it work?

File Size Leaderboard (MB, lower is better)

Decode Time Leaderboard (ms, lower is better)

Benchmark parquet-linter

Vector search using only Parquet and DataFusion

Wait, Isn’t Parquet Terrible for Random Access?

Zero-copy Vector Indexing

In-Place Index Creation

DataFusion Integration

How It Looks in Code

1. Building the Index

2. Searching with SQL (DataFusion)

Does It Actually Work?

Need your help!

Stop building systems for agents

The Wrong Direction

The Unaccountable Machine

The “Happen-to-Work” Foundation

LLM Agents Shake Our Fragile Foundation

The Real Agent-Native Systems

What is LiquidCache?

Why LiquidCache?

We like S3

But S3 is slow and expensive

LiquidCache: foundation of diskless architectures

How LiquidCache Works

We like Parquet

But sometimes we want more aggressive performance

LiquidCache: cache-only, pushdown-optimized data representation

Conclusions

Who are we?

On the Nature of Strings

TL;DR

Conclusion

Build your own S3-Select in 400 lines of Rust

TL;DR

Architecture

Compute

Storage

Communication

Life of a query

Implementing the Storage Server

1. Schema Resolution

2. Query Planning

3. Data Streaming

Implementing the Compute Node

Implementing FlightTable

Implementing FlightExec

Putting It All Together

Server Binary

Client Binary

Running the System

What’s Next: LiquidCache

Conclusion

Efficient Filter Pushdown in Parquet

Why filter pushdown in Parquet?

Benchmark `parquet-linter`

Section 3.2: Faster `take` and `filter`