• o

    Ojas

    5 months ago
    Hey I updated my fleet and osquery recently but now: I am not able to find the osquery logs in /var/osquery/* not in C:\Program Files\osquery*
  • there seems to be a new folder orbit. Inside this i did find osquery.db but the logs are not the sme way it used to be. its very badly formatted and just in one single file. Most of them just say what fleet tried to run on the machine and no results are logged
  • mac logs
  • ubuntu
  • Windows
  • All i need now is someone to show me how to configure the logs to be stored on the host itself. This is my config rn and it dosent seem to do the trick
  • Luke Heath

    Luke Heath

    5 months ago
    Hi Ojas, yes you can do this using the filesystem as the logger plugin: https://fleetdm.com/docs/using-fleet/osquery-logs#filesystem
  • Lucas Rodriguez

    Lucas Rodriguez

    5 months ago
    Hi Ojas!
    logger_plugin
    is for osquery
    result
    (scheduled queries results) and osquery
    status
    logs. If you are interested on osquery logs in general, you can check the following paths:https://github.com/fleetdm/fleet/tree/main/orbit#logs (Both Orbit and osquery logs should be in those locations.)
  • o

    Ojas

    5 months ago
    @Luke Heath Thanks for you response. So i have configured correctly as given in above screenshot? Coz even after configuring this i dont see any proper logs on my hosts. I do see a file which contains data of what agent is connecting to and what query ran on the machine and everything. But not he proper logs like which contains the results of the query which ran on the machine.
  • @Lucas Rodriguez thanks for you response. I have configured the logger_plugin but still no logs on the endpoint. The orbit logs are not as proper as the older osquery logs which were is /var/log/osquery/* In the orbit logs file we have everything logged like what agent is doing what query ran and not in a standard format. any way i can configure it to goto old format
  • I have this err on mac os: E0304 05:47:53.505184 129021440 shutdown.cpp:75] Cannot activate filesystem logger plugin: Could not create file: /var/log/osquery/osqueryd.results.log
  • If i create the folder osquery there it starts working fine but when the folder is not there it throws err that cannot create it. Any fix for this? as i have to install it on alot of systems, manually creating those folders would be painfull
  • Same err on windows: Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\
  • Lucas Rodriguez

    Lucas Rodriguez

    5 months ago
    Hi @Ojas! I believe you hit this bug: https://github.com/fleetdm/fleet/issues/4146 (Which will be fixed on our next release coming today/early-next-week, TBD)
  • You will have to remove
    filesystem
    from
    logger_plugin
    (and just leave
    tls
    or empty until the new release is out).
  • Though, once we release the new Orbit version, your instances should auto-update.
  • o

    Ojas

    5 months ago
    alright. Thanks Lucas. I’ll wait for the update.
  • So after the update do i need to again generate the installer agent or do i need to install osquery on machine again to get updated config? or will that be auto-updated too
  • Lucas Rodriguez

    Lucas Rodriguez

    5 months ago
    Hi Ojas!, once we release fleet-osquery (aka orbit) - sometime early this week - it should auto-update automatically. (If things work as expected you won't need to re-generate the installers.)
  • o

    Ojas

    5 months ago
    awesome thanks
  • Hey @Lucas Rodriguez I still dont see any logs created. Any update on patch? Do i need to generate a new installer?
  • Also now i see another older issue, my fleet_osquery service in windows keeps stopping.
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    Hi @Ojas!
    Do i need to generate a new installer? No, should auto-update.
    Also now i see another older issue, my fleet_osquery service in windows keeps stopping. Could you check
    C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log
    ? (You may be hitting a known issue we are trying to fix for next release.)
  • o

    Ojas

    4 months ago
    it says ” Cannot activate filesystem logger plugin” :?
  • @User i still see the old error: 2022-03-30T11:19:42Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” E0330 11:22:40.311066 3380 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 11:22:42.511026 5160 shutdown.cpp:79] Worker returned exit status
  • Also on manually creating folders it works fine. It’s still the issue of not able to create the folders
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    OK, could you run
    "C:\Program Files\Orbit\bin\orbit\orbit.exe" --version
    ?
    Could not create file: \Program Files\osquery\log\osqueryd.results.log On the latest version we changed the path, that looks like the old default path.
  • o

    Ojas

    4 months ago
    orbit 0.0.6
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    OK, latest is 0.0.7 (and soon 0.0.8). For some reason it's not auto-updating.
  • Does the host have access to https://tuf.fleetctl.com?
  • o

    Ojas

    4 months ago
    how do i check that?
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    Any other network error logs related to updating (in
    C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log
    )?
  • how do i check that? Try visiting the URL from a browser in the host, or using the
    curl
    or
    wget
    commands (if available).
  • o

    Ojas

    4 months ago
    i can ping it from the host
  • 2022-03-30T11:19:42Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” E0330 11:19:44.483722 4888 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 11:19:45.709700 5776 shutdown.cpp:79] Worker returned exit status 2022-03-30T11:19:45Z ERR unexpected exit error=“osqueryd exited with error: exit status 78" 2022-03-30T11:19:47Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” E0330 11:19:48.367669 5808 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 11:19:51.128486 3348 shutdown.cpp:79] Worker returned exit status 2022-03-30T11:21:33Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” E0330 11:21:34.335160 3884 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 11:21:37.178841 3876 shutdown.cpp:79] Worker returned exit status 2022-03-30T11:22:38Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” E0330 11:22:40.311066 3380 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 11:22:42.511026 5160 shutdown.cpp:79] Worker returned exit status 2022-03-30T11:25:42Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” I0330 11:25:43.293851 3316 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration I0330 11:25:46.206353 1712 interfaces.cpp:102] Failed to retrieve network statistics for interface 4 I0330 11:25:46.266278 1712 interfaces.cpp:102] Failed to retrieve network statistics for interface 1 I0330 11:25:46.269050 1712 interfaces.cpp:130] Failed to retrieve physical state for interface 1 I0330 11:25:46.287117 1712 interfaces.cpp:157] Failed to retrieve DHCP and DNS information for interface 1 I0330 11:25:46.377097 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.389892 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.390563 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.392493 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.395962 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.396636 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.398648 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.400411 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 11:25:46.401648 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle W0330 11:25:46.763278 1712 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable E0330 11:25:48.131688 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_1: no such table: gatekeeper W0330 11:25:48.153442 1712 bitlocker_info.cpp:52] Error retreiving information from WMI. E0330 11:25:48.159821 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_3: no such table: disk_encryption E0330 11:25:48.164297 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_4: no such table: disk_encryption E0330 11:25:48.168017 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_5: no such table: sip_config E0330 11:25:48.170768 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_6: no such table: managed_policies E0330 11:25:48.173740 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_7: no such table: managed_policies E0330 11:25:48.176471 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_8: no such table: managed_policies E0330 11:25:48.179143 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_9: no such table: plist I0330 11:26:12.553747 2764 registry.cpp:555] Failed to expand globs: Failed to open registry handle W0330 11:26:12.555688 2764 virtual_table.cpp:961] The chrome_extensions table returns data based on the current user by default, consider JOINing against the users table W0330 11:26:12.558739 2764 virtual_table.cpp:961] The firefox_addons table returns data based on the current user by default, consider JOINing against the users table W0330 11:26:12.572578 2764 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable W0330 11:26:12.583083 2764 virtual_table.cpp:961] The atom_packages table returns data based on the current user by default, consider JOINing against the users table I0330 11:26:12.585950 2764 query.cpp:102] Storing initial results for new scheduled query: pack_test_Get installed Windows software 2022-03-30T11:29:09Z INF start osqueryd cmd=“C:\Program Files\Orbit\bin\osqueryd\windows\stable\osqueryd.exe --pidfile=C:\Program Files\Orbit\osquery.pid --database_path=C:\Program Files\Orbit\osquery.db --extensions_socket=\\.\pipe\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\Program Files\Orbit\certs.pem --force --flagfile C:\Program Files\Orbit\osquery.flags” I0330 11:29:09.557931 3980 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration I0330 11:32:44.828908 3940 registry.cpp:555] Failed to expand globs: Failed to open registry handle W0330 11:32:44.830849 3940 virtual_table.cpp:961] The chrome_extensions table returns data based on the current user by default, consider JOINing against the users table W0330 11:32:44.834173 3940 virtual_table.cpp:961] The firefox_addons table returns data based on the current user by default, consider JOINing against the users table W0330 11:32:44.848613 3940 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable W0330 11:32:44.859524 3940 virtual_table.cpp:961] The atom_packages table returns data based on the current user by default, consider JOINing against the users table
  • Able to access the site from browser as well.
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    OK, a guess: so due to the issue (which causes orbit to crash), orbit cannot auto-update. Could you disable the
    filesystem
    logger configuration from Fleet temporarily? (to give the process a chance to auto-update)
  • o

    Ojas

    4 months ago
    Do i change it to tls? or just remove it from there all together?
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    Yes, just
    tls
    .
  • o

    Ojas

    4 months ago
    alright. Done config: options: logger_plugin: tls
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    What I fear is that osquery already stored
    filesystem
    in its internal rocksdb local storage..., but let's see if this helps.
  • o

    Ojas

    4 months ago
    should i restart the machine? or just wait few minutes?
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    Let's wait a few minutes first.
  • Worst case scenario: You will need to regenerate a
    msi
    installer and re-install Orbit.
  • o

    Ojas

    4 months ago
    for that would orbit need to be cleaned from the machine first? or as i run new installer it will auto install new one
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    Yes. Running a new installer should override current installation. Let me know if it doesn't.
  • o

    Ojas

    4 months ago
    yea with new installer it got updated to orbit 0.0.7
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    OK, let me know if the
    filesystem
    change is still not working.
  • o

    Ojas

    4 months ago
    Tried it on a new machine from scratch. Works nicely. thanks 😃 So everytime there is a patch/update do i need to create a new installer?
  • Lucas Rodriguez

    Lucas Rodriguez

    4 months ago
    No, orbit should automatically update. The issue here is that the filesystem configuration was causing issues at startup, so it didn't have the chance to run the auto-updater routine...