Channels
- j
jom
2 years agoHi everyone, I’m the guy behind this feature request: https://github.com/osquery/osquery/issues/5678. We will implement this function as soon as possible. For this i have decided to register into this chat 😉. Many greetings Jom - s
seph
2 years agohi! welcome! have you considered writing it as an extension? - a
alessandrogario
2 years agoHey jom! We are currently changing how dependencies are obtained; not only for technical reasons (better and reproducible builds) but also because we can't easily update the libraries where they are stored now. If you take seph's approach, then you can more easily bring your own Windows-compatible kakfa library and work around the problem. You can look at the branch I'm working on and copy how I imported it: https://github.com/osquery/osquery/commit/083df997519cf930c8b74f86498b62fb742b07e2 If you decide to work on this, we can help out! 🙂 - j
jom
2 years agoThanks for the quick answers! Yes, I see that you are rebuilding quite a bit right now. Just when we finished our poc for the kafka export. 😄 We’ll take a look at the link, thank you very much! Is there already a defined process/pipeline to provide extensions to the the public? - a
alessandrogario
2 years agoExtensions had good support until the huge refactor happened; right now things are still a little messy 😞 If you use osquery 3.3.2 you can more easily build extensions that are easy to deploy
and those extensions are then also compatible with the latest version
We are trying to fix everything and hopefully things will be a lot better by the end of this month
We would like to land new features that add dependencies inside extensions, but for this specific case we would only be enabling what we already have on macOS and Linux (+ some code changes to enable SSL) so I think we can merge this without problem
seph's suggestion is good because you don't have to wait until everything on our side is sorted before you can have the functionality up and running
If you were wondering more about the actual process of copying the extensions (rather than developing them), I think most people are using things like puppet/chef or taking advantage of Kolide products - j
jom
2 years agommh. we have other requirements. For us, it’s only a matter of passing on the relevant questions to a kafka. We do nothing on the producing-machine itself. Then a flink job comes to the kafka and processes the data into our PostgreSQL/druid. I am wondering a little bit: The export of data to kafka is integrated into the linux clients. Does it make sense to omit this function from the windows clients?
Only that I just got it right:- With the extensions plugin in v4.0 there is a lot under construction. - The same applies to the integrated libraries for 4.0. it’s not easy to start developing a feature right now, is it? - a
alessandrogario
2 years agoIf it's a new feature (and not enabling what we already have on Linux/macOS) then it's better if it remains inside an extension. I really don't know much about Kafka 😞
The extensions framework is not in 4.0 yet, but it is in 3.3.2. Nothing has changed there. The extension protocol is still compatible, so it's not a problem to build them with 3.3.2 and have them run within 4.0.0 (we will port the SDK soon!)
New features, provided they do not add dependencies, can be implemented right now
I realize the current situation is not ideal, but a lot has happened in the past month (a huge refactor from Facebook + the project has moved under the Linux Foundation)
Both moves have been really good for the health of the project! Things will return to normal once the first stable is out, at the end of this month 🙂 - j
jom
2 years agoNo stress. I have just allocated money and resources here and am thinking about how we can get our feature “kafka export for the windows client” most cleverly. Best of all, that it is available for the whole community and we don’t want to start from scratch as soon as 4.0 is out 🙂 - a
alessandrogario
2 years agoI think we all agree that we should create a new github.com/osquery/extensions repository
So we can merge it under the osquery organization regardless of whether it's an extension or not - j
jom
2 years agoSorry to ask you again stupidly: In Linux and Mac the kafka exporter works yes. It’s not an extension there, is it?
https://osquery.readthedocs.io/en/stable/deployment/logging/#logging-as-a-kafka-producer - a
alessandrogario
2 years agohttps://github.com/osquery/osquery/blob/master/plugins/logger/CMakeLists.txt#L144
it is currently present inside macOS and Linux, and disabled for Windows
most likely because it was not building - j
jom
2 years agoYes, we wanted to sit on librdkafka. With the intention that it will then build. But that won’t be enough with 4.0, will it?
But in general it makes sense to make kakfa export possible on windows as well, doesn’t it?
View count: 1