Channels
  • e

    ehrhardt

    1 year ago
    Looks like after I pushed a query pack change, osquery stopped using
    rocksdb
    and started using
    ephemeral
    for pack results which means the results are not stored anywhere. Does anyone know where this setting configured?
  • zwass

    zwass

    1 year ago
    How did you push the query packs?
    database_plugin
    would be set in the flags or config file.
    Although... I'm not even sure that config still exists?
  • e

    ehrhardt

    1 year ago
    We use puppet to push these changes. The query pack is one JSON file. Changes to the pack were mostly minimal. I am seeing this in the logs
    I0925 13:47:46.259778 187797504 database.cpp:140] Resetting the database plugin: rocksdb
    I0925 13:53:25.853595 187797504 database.cpp:140] Resetting the database plugin: rocksdb
    I0925 13:59:04.234082 187797504 database.cpp:140] Resetting the database plugin: rocksdb
    W0925 14:04:38.143973 289832384 database.cpp:77] Failed to activate database plugin "rocksdb": IO error: While lock file: /var/osquery/osquery.db/LOCK: Resource temporarily unavailable
    I0925 14:05:03.339555 63148032 database.cpp:140] Resetting the database plugin: ephemeral
    I0925 14:10:43.635411 63148032 database.cpp:140] Resetting the database plugin: ephemeral
    I0925 14:16:06.220286 63148032 database.cpp:140] Resetting the database plugin: ephemeral
  • theopolis

    theopolis

    1 year ago
    What version of osquery are you using? I think I might know what is happening and that is indeed unintended behavior that I was indirectly trying to fix recently.
  • e

    ehrhardt

    1 year ago
    4.4.0
  • theopolis

    theopolis

    1 year ago
    Essentially osquery had under-the-hood fallback code that if another process was using the database, or for some reason the error on
    database.cpp:77
    occurred, it would fallback to using in-memory storage.
  • e

    ehrhardt

    1 year ago
    How do I stop it from using in memory storage?
  • theopolis

    theopolis

    1 year ago
    Another process should not be trying to use the database in normal scenarios. And it is not possible to have multiple processes open a handle to the database.
    However osquery should not silently fallback to in-memory. It should either work or not work.
    (in theory)
    Unfortunately the only fix I can think of is upgrading to 4.5.0 or 4.5.1
  • e

    ehrhardt

    1 year ago
    Hrm, so our experience is changing a query pack json seems to have caused osquery to permanently use in memory storage for about half our osx systems
    Just a statement so you know it can happen 🙂
    Were changes made in 4.5 that would prevent this from recurring?
  • theopolis

    theopolis

    1 year ago
  • e

    ehrhardt

    1 year ago
    Perfect, thank you
  • theopolis

    theopolis

    1 year ago
    I am sorry this is happening 😞