Rendered at 12:03:59 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tmzt 1 days ago [-]
This looks useful for somebody with a 16-32GB Mac Mini interested in running larger MoE models.
I've been working on a mesh environment that relies on an explicit prefix hash in the request and enables constructing a new session with a cached pre-filled system prompt specific KV-cache beyond what OpenAI-compatible APIs offer. Can you see a feature like that being supported?
tmzt 8 hours ago [-]
I went ahead and forked this and added a new remote INFER protocol. You can see it at https://github.com/tmzt/ds4 if interested.
I've been working on a mesh environment that relies on an explicit prefix hash in the request and enables constructing a new session with a cached pre-filled system prompt specific KV-cache beyond what OpenAI-compatible APIs offer. Can you see a feature like that being supported?