Per your request @ogencoglu (here), I'll leave some comments on why I tried, and we I'm abandoning LMQL, and this will include comparisons with outlines and guidance.

EDIT 2 weeks later: The lmql team has been responsive, and gracious, and I've found that this project is the best working option available right now, so I'm continuing my efforts using lmql instead of alternatives. The following is unedited, some of which still stands.

1.) guidance is abandonware, so I thought I'd try the next top recommendation, LMQL

2.) My first impression is that the LMQL team put far more effort into a shoddy .lmql language, instead of properly supporting python. Python "works", but documentation barely mentions it and you instead focus on this .lmql script which, for one, only has code highlighting in vscode, and for two, why do I need .lmql? It separates my concerns back out of python. My problem + solution all live within python!

3.) So python "is supported" right? The syntax parser stinks. It breaks for no apparent reason, and gives inscrutable error messages. Now, a comparison to guidance: guidance nails the dev UI. I want to write 1 prompt, and summon LLM generations inline. I don't want a query language that has me jumping all over the place to see, for instance, what the constraints are. And then it's not obvious if LMQL supports multiple different samplers, like guidance does, inline.

4.) I've logged 3 bugs. And 1 discord question** that's gone unanswered, which is, if you use nested prompting (which is so much less ergonomic than guidance) how can you thread known static variables in to be interpolated within the prompt? I couldn't figure it out, so had to flatten everything.

5.) I could not get GPTQ models working with LMQL

6.) There's a memory leak in the server.

7.) The server doesn't allow me to set up my own model, which would have helped me get AutoGPTQ's params set up properly.

8.) STOPS_AT is doesn't actually "stop at" if you set a long token length. It just keeps generating.

9.) Falcon was a nightmare to get running, and GPTQ was impossible.

10.) Guideline's "token healing" is a necessary feature, IMO.

And there was more to that battle I'm not mentioning, but those were the big ones.

So how about outlines?

Outlines is less mature, fewer features, but they've nailed down their core thing which is "guided generation", (and none of this "let's invent a superset of python!" junk. apologies, but it's truly frustrating.)

I would guess outlines is far faster than LMQL (and guidance) since they do this whole FSM-precompliation step which allows for O(c) lookups during inference. I wouldn't know though, since I never got LMQL properly running with the models I intended (gptq'd falcon).

My major gripe with outlines is it doesn't allow any form of inline generation, as far as I know; only prompt continuation. LMQL and guidance both do, and that's a killer feature. Although like mentioned above, LMQL also introduces miles of indirection between your problem and your solution. I think I can try making a regex that accomplishes the same, eg Hello [a-z]*, tell me about [a-z]* please!, but I don't know if I'll get lookaheads/STOPS_AT working with outlines, not sure if they offer it. So I will likely have to break the prompts out into many pieces.

As a litmus test, guidance allows you to read a prompt like a passage from a book. LMQL has you jumping all over the place and trying to remember all the pieces to understand the final prompt.

I'll be happy to revisit LMQL some day for that killer inlining feature if any of the above are addressed!

And sorry for the rant. I just burned a day each on trying to get guidance and lmql to work, and am finally settling on outlines since it "just works", albeit with my much-missed inline gen feature.

We have a comparison with Guidance here: https://docs.lmql.ai/en/stable/python/comparison.html. Outlines is another option in the constrained decoding space, however, as far as I know the project, they are much more focused on e.g. Regex constraining and less on an overall programmability aspect of LLMs.

We are closely tracking their work on FSM-based constraining, although it is less obvious to bring this directly to LMQL, as our goal is to support a broader set of backends, including high-level APIs like OpenAI, which do not support this kind of optimisation directly. However, I think it is also actually possible to combine LMQL and Outlines, e.g. using their form constraining as LMQL constraints, if one writes the necessary glue code for that.

All in all, I would advise to test drive both, and to decide based on personal preference and requirements, what better suits your workload.