Channels
  • i

    Ivanlei

    2 years ago
    • we use benchmarks to create artificial constant rate of process events per second • we monitor CPU, RAM, & diskIO • we monitor the osquery logs to see when event buffers overflow one finding i wanted to share - we set
    snapshot=true
    on our eventing queries and measured a 5X increase in the events per second throughput of
    process_events
    before we saw events buffers overflow
  • theopolis

    theopolis

    2 years ago
    Do your queries on
    process_events
    look similar to
    select * from process_events
    or are they more complex?
  • i

    Ivanlei

    2 years ago
    they are more complex. we join on stuff like process table & process_containers table
    those joins seemingly have no real perf impact
  • theopolis

    theopolis

    2 years ago
    ah, cool, I asked because I think there's some under the hood logic to detect simple queries against
    _events
    tables and turn on
    snapshot=True
    for you (I am going from memory though)
  • i

    Ivanlei

    2 years ago
    we have about the same performance whether we join to the tables or not
  • theopolis

    theopolis

    2 years ago
    this is worth investigating, thanks for sharing the analysis!
  • zwass

    zwass

    2 years ago
    One thing that this brings up for me... If someone has
    events_optimize
    on, then we can probably skip the diffing step on a differential query to an event-based table... the results should all be "added".
  • i

    Ivanlei

    2 years ago
    @zwass - kinda naively assumed that was how things were working. we had
    removed=true
    on our event queries and had
    events_optimize=true
    . it surprised me how much of a difference
    snapshot=true
    made for the audit data.
  • zwass

    zwass

    2 years ago
    The major difference in perf I would anticipate for snapshot vs. differential would be the diffing step. Both generate the results in approximately the same way. Someone would have to have special-cased the diffing for event-based tables -- I'm not sure that has been done.
  • i

    Ivanlei

    2 years ago
    my read says it hasn’t been
  • zwass

    zwass

    2 years ago
    Seems ripe for doing!
  • i

    Ivanlei

    2 years ago
    there is code related to
    events_optimize
    that introspects the query on a specific event table. potentially a cheap spot to simply flip the
    snapshot
    option on for the
    ScheduledQuery

    sounds cleaner in some respects than special-casing the diffing logic
  • zwass

    zwass

    2 years ago
    It's an interesting idea. snapshot vs. diff queries have different log schema so that could be an issue.
  • i

    Ivanlei

    2 years ago
    a doc update recommending
    snapshot
    would even go a long way