I d be fine with writing an events buffer for osquery go wit

Question

I d be fine with writing an events buffer for osquery go with a backing store if we could figure out how to resolve the events optimize issue

seph · Answer

`events optimize` is in many ways analogous to kafka or redis consumer groups Now I m giggling about creating a redis stream backed event store

zwass · Answer

Maybe `events optimize` means each query only sees each event once the first time the query is run after the event is generated

seph · Answer

Yes If you imagined events on a log pipeline each query is a consumer group The underlying pipeline keeps track of that dispatch

seph · Answer

It s the core innovation in kafka And something all streaming systems do now

seph · Answer

Without having written any code I can imagine storing an event stream seems pretty straight forward Borrow redis trick for naming them with timestamps Track a pointer to the last seen item in the stream

seph · Answer

But I kinda wish there was a pure go library for this

zwass · Answer

Yeah 100 we can do that but we need a way to get the name of the executing query or some equivalent to the extension so that it can do the optimization

zwass · Answer

I ve been looking at <https github com dgraph io badger> as a local store for Go lately

seph · Answer

Do we need the name or can you just hash the query Slightly different semantics

seph · Answer

I bet the extension doesn t get `LIMIT` which might break things

seph · Answer

I ve really enjoyed <http github com dgraph io ristretto|github com dgraph io ristretto> for a project But it s a bit simpler

seph · Answer

Oh we probably don t get the query eithger

zwass · Answer

Yeah I think we don t get the query