JSONPath Is In! The AI Assistant Will See You Now

JSONL is a neat and kind of weird data format. It is well-known to be useful for logs and API calls, among other things. And your favorite AI assistant API is one place you’ll probably find it.

CsvPath Framework supports validating JSONL. (In fact, it supports JSONL for the whole data preboarding lifecycle, but that’s a longer story).

And now CsvPath Validation Language supports JSONPath expressions. Since AI prompts are only kinda sorta JSONL, having JSONPath to dig into them is helpful.

What I mean by kinda-sorta is that your basic prompt sequence is a series of JSON lines, but the lines are all 1-column wide and arbitrarily deep. That sounds more like a series of JSON “documents” than it does like single JSONL stream. Or, anyway, that’s my take.

Let’s look at how to use JSONPath to inspect a JSONL file using CsvPath Validation Language in FlightPath Data. For those of you who don’t already know, FlightPath Data is the dev and ops frontend to CsvPath Framework. It is a free and open source download from the Apple MacOS Store or the Microsoft Store.

The file is a common example prompt. We’ll start by looking at one line.

Here’s the last line:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a happy assistant that puts a positive spin on everything."
    },
    {
      "role": "user",
      "content": "I'm hungry."
    },
    {
      "role": "assistant",
      "content": "Eat a banana!"
    }
  ]
}

From CsvPath Framework’s perspective this is a one-header document. The one header is messages. If you open this in the grid view you see only the one header. (i.e. one column; but with delimited files we try to stick to the word “header” because with “column” your RDBMS-soaked brain starts to make incorrect assumptions).

Here’s what it looks like:

That’s not super fun. The reason is that:

JSONL doesn’t present its headers in the grid view (for good reasons)
The messages header is arbitrarily deeply nested, unlike the typical JSONL log line

Nevertheless, that’s what we have. Will it blend? I mean validate? Yes. JSONPath to the rescue. That said I’ll pause to admit that I’m not a JSONPath expert.

Right-click in the project files tree on the left of FlightPath and create a new .csvpath file, e.g. messages.csvpath. Drop this simple example in it.

$[*][ 
    push("roles", jsonpath(#messages, "$[?(@.role == 'assistant')].content") )

    last() -> print("See the variables tab for results")
]

You can see the jsonpath() function. It is acting on the messages header, as we’d want. We’re pushing the data pulled by the JSONPath expression into a stack variable named roles.

A stack variable is like a Python list or tuple. You create a variable by using it. roles is part of the set of zero or more variables that are available throughout the csvpath statement run. They are captured to the Variables tab for a one-off FlightPath Data test run. For a production run they end up in the vars.json file in the run results.

Put your cursor in the csvpath statement and click cmd-r (or ctrl-r on Windows). The output tabs should open at the bottom-middle of the screen, below your csvpath file. Click to the Variables tab and have a look.

Our JSONPath:

$[?(@.role == 'assistant')].content

picked out the objects in the messages array where role equaled assistant. And from those objects it extracted and returned the value of the content key.

Pretty simple stuff. Tho, I have to admit it took me a few minutes to wrap my JSONPath-inexperienced head around the context for the JSONPath expression. I was thinking of the whole document or the whole line, but that wasn’t right.

It is obviously the JSON value assigned to the messages key, which is an array, in this case. Once I was operating from that correct context, the JSONPath became pretty straightforward. (Those of us with XPath scars need not be as afraid as we might be!)

The point here is two-part. First, we can deal with AI prompts or any other JSONL that is deeply nested. Hooray! The data may look odd, if you are comparing it to regular tabular data, but that’s no reason to not validate.

Second, this example makes the point that we’re doing JSONPath rules-based validation within our CsvPath context. How very Schematron-like, since Schematron does XPath validation within XSLT.

Maybe this sounds complicated, but really it’s not. CsvPath Validation Language is great for all things line-oriented. In this case, there isn’t much for it to do, except hand off to JSONPath, which is great at documents (a.k.a. objects). Simple enough.

If we wanted to create a bunch of JSONPath rules to validate our AI prompt JSONL, we could do that. To just do a quick throw-away second rule as an example try this:

$[*][ 
    push("roles", jsonpath(#messages, "$[?(@.role == 'assistant')].content") )

    @stmts = jsonpath(#0, "$.`len`")
    print("Line $.csvpath.line_number: $.variables.stmts")
    @stmts == 3 

    last.nocontrib() -> print("See the variables tab for results")
]

That new rule will net you 2 lines, which are either valid or failed, depending on how you want to use your csvpath statement. You will see them in the Matches tab.

At the same time the expanded csvpath statement will continue to pull in the same data to the variables tab that we got with the first version of the csvpath.

To clean it up just a little, you can do:

$[*][ 
    push("roles", jsonpath(#messages, "$[?(@.role == 'assistant')].content") )

    jsonpath(#0, "$.`len`") == 3
]

There you go, a valid 2-rule validation statement using JSONPath on nested JSON in a JSONL document. Useful? Totally! Give it a try.

Leave a Reply Cancel reply