Flide is extremely fast when one input carries many questions, as the shared input is computed only once. Even with a single question, it can still be up to 3x faster, depending on the model size and input length.

You really don't need an entire new inference engine for this. I modified vllm to work just fine for my decision model experiments.

It properly uses the single shared context and only recomputes the questions, or if the questions are longer, I let the questions go first and then the context that changes often can go after, and then the answer menu tokens... I got roughly the same speed increase with the right workloads that you're saying though.

I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!

Hi HN! I'm one of the Flide team.

We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.

Here is the source code: https://github.com/OmniJev/Flide

Look forward to hearing your feedback and ideas!

do i need to host the model to do this ? if yes what model can i host in 3050 card

[flagged]

[flagged]