• t

Ted Dorosheff

8 months ago
Hey all, so i'm still unable to get any ntfs_journal_events data from my windows hosts, after recently adding file paths to my agent settings. I do however see data from other queries such as hardware events and firmware snapshots. My file events query:
SELECT action, category, old_path, path, file_attributes, time FROM ntfs_journal_events;
my agent config
config:
options:
events_expiry: 60
config_refresh: 600
host_identifier: instance
distributed_interval: 60
decorators:
- >-
hostname) as hostname FROM system_info;
file_paths:
etc:
- /etc/group
- /etc/passwd
- /etc/services
- /etc/sudoers
- /etc/ld.so.conf
- /etc/ld.so.conf.d/%%
- /etc/pam.d/%%
- /etc/resolv.conf
- /etc/modules
- /etc/hosts
- /etc/hostname
- /etc/fstab
- /etc/rsyslog.conf
ssh:
- /root/.ssh/%%
- /home/%/.ssh/%%
- /etc/ssh/%%
- /var/lib/sia/keys/
- /var/lib/sia/certs/
logs:
- /var/log/secure
docker:
- /etc/docker/%%
- /etc/default/docker
- /etc/docker/daemon.json
- /usr/bin/containerd
- /usr/sbin/runc
- /etc/sysconfig/docker
- /usr/lib/systemd/system/docker.service
- /usr/lib/systemd/system/docker.socket
osquery:
- /etc/osquery/%%
- /usr/share/osquery/packs/%%
firewalls:
- /etc/sysconfig/iptables
- /home/y/conf/yakl/%%
- /etc/yakl/conf/%%
overrides:
platforms:
windows:
options:
events_expiry: 60
config_refresh: 600
host_identifier: instance
distributed_interval: 60
decorators:
- >-
hostname) as hostname FROM system_info;
file_paths:
users:
- 'C:\users\AppData\Roaming'
- 'C:\users\AppData\Local'
- 'C:\users\AppData\Local\temp'
- >-
C:\users\AppData\Roaming\Microsoft\Windows\Start
- 'C:\Users\Default'
windows:
- 'C:\Windows'
- 'C:\Windows\temp'
- 'C:\Windows\system32\Drivers'
- 'C:\Windows\SysWOW64\Drivers'
- 'C:\Windows\system32\GroupPolicy\Machine\Scripts'
- 'C:\Windows\system32\GroupPolicy\User\Scripts'
- 'C:\Windows\system32\Wbem'
- 'C:\Windows\SysWOW64\Wbem'
- 'C:\Windows\system32\WindowsPowerShell'
- 'C:\Windows\SysWOW64\WindowsPowerShell'
- 'C:\Windows\AppPatch\Custom%'
ProgramData:
- 'C:\$WINDOWS.~BT\Sources\%' - 'C:\Windows\Installer\%' - 'C:\Windows\System32\Tasks\Adobe Acrobat Update Task%' - 'C:\Windows\System32\Tasks\Adobe Flash Player Updater%' - >- C:\Windows\System32\Tasks\OfficeSoftwareProtectionPlatform\SvcRestartTask% @Mystery Incorporated helped with this before, if you could just give me another look that would be much appreciated. I'm really lost. • Mystery Incorporated 8 months ago @Ted Dorosheff show flags file please, what you need to set is probably in flags file • zwass 8 months ago @Ted Dorosheff Try live querying select * from osquery_events to see whether the publishers are activated and receiving any events. • t Ted Dorosheff 7 months ago hey @zwass happy new year. Back at the keyboard today, and i think you may be right about the event publishers not being activated. Log file created at: 2021/12/27 11:01:32 Running on machine: COOLEDRULED Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I1227 11:01:32.819942 5964 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration • zwass 7 months ago Seems like @Mystery Incorporated has it right then -- you'll need to configure the appropriate flags. • t Ted Dorosheff 7 months ago okay, do these look like the right flags? --enable_windows_events_publisher=true --enable_windows_events_subscriber=true --enable_ntfs_event_publisher=true They were currently missing from my flags file • although going off the log, looks like i may only need --enable_ntfs_event_publisher=true • zwass 7 months ago That sounds right. The windows_events ones would be for the windows_events table. • t Ted Dorosheff 7 months ago alright so i just tried live querying the host, and now at least i'm not getting the error about the event publisher not enabled, but i'm still not seeing any journal events • I ran osqueryi from cmd, with all those flags set. osqueryi.exe -S --config_path="C:\Program Files\osquery\osquery.conf" --disable_events=false --enable_windows_events_publisher=true --enable_windows_events_subscriber=true --enable_ntfs_event_publisher=true --verbose But i'm guessing that file_paths still needs to be defined? We have file_paths set in our yaml config in fleetDM, but i would assume that isn't being loaded if i'm running osqueryi like this? • osquery> select * from osquery_events; +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | name | publisher | type | subscriptions | events | refreshes | active | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher | 1 | 17 | 0 | 1 | | ntfs_event_publisher | ntfs_event_publisher | publisher | 0 | 0 | 934 | 1 | | ntfs_journal_events | ntfs_event_publisher | subscriber | 0 | 0 | 0 | 1 | | powershell_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | | windows_events | WindowsEventLogPublisher | subscriber | 1 | 437 | 0 | 1 | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ osquery> • zwass 7 months ago Yeah that's right. • t Ted Dorosheff 7 months ago before i ran the query, i dropped a .txt file into one of the paths which we've got listed in our yaml config • zwass 7 months ago Can you run osqueryi with the normal flagfile so it pulls the config from Fleet? • t Ted Dorosheff 7 months ago yes • C:\Program Files\osquery>osqueryi.exe Using a [1mvirtual database[0m. Need help, type '.help' osquery> select * from osquery_events; +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | name | publisher | type | subscriptions | events | refreshes | active | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher | 0 | 0 | 0 | 0 | | ntfs_event_publisher | ntfs_event_publisher | publisher | 0 | 0 | 0 | 0 | | ntfs_journal_events | ntfs_event_publisher | subscriber | 0 | 0 | 0 | 0 | | powershell_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | | windows_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ osquery> • zwass 7 months ago Is that top line your command? If so, missing --flagfile • t Ted Dorosheff 7 months ago ahh okay • whats the command to stop, and just return to the shell? • or do i have to kill the pid entirely and open a new shell? • doing this all on a VM, and anytime i need to escalate to an admin shell, i need to swap windows and grab a password • zwass 7 months ago Does control-d do it? • t Ted Dorosheff 7 months ago ahh yes, thanks • whew that took longer than expected.... C:\Program Files>osquery\osqueryi.exe --flagfile="C:\Program Files\osquery\osquery.flags" Using a [1mvirtual database[0m. Need help, type '.help' osquery> osquery> select * from osquery_events; +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | name | publisher | type | subscriptions | events | refreshes | active | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher | 0 | 0 | 0 | 1 | | ntfs_event_publisher | ntfs_event_publisher | publisher | 3 | 0 | 0 | 0 | | ntfs_journal_events | ntfs_event_publisher | subscriber | 3 | 0 | 0 | 1 | | powershell_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | | windows_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ osquery> • osquery.db was locked by another process • zwass 7 months ago Looks like the subscriptions are going now... Does it pick up the changes? • select * from ntfs_journal_events; • t Ted Dorosheff 7 months ago osquery> select * from osquery_events; +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | name | publisher | type | subscriptions | events | refreshes | active | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher | 0 | 0 | 0 | 1 | | ntfs_event_publisher | ntfs_event_publisher | publisher | 3 | 0 | 0 | 0 | | ntfs_journal_events | ntfs_event_publisher | subscriber | 3 | 0 | 0 | 1 | | powershell_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | | windows_events | WindowsEventLogPublisher | subscriber | 0 | 0 | 0 | 0 | +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+ osquery> select * from ntfs_journal_events; osquery> • nothing • perhaps my yaml config isn't done correctly? file_paths incorrectly defined, or some syntax error in the config. side note: if we want to monitor user directories, could we wildcard where the user name would be? like: C:\Users\*\foo C:\Users\*\bar Running this in an enterprise environment where the username is going to be different on every machine. • zwass 7 months ago Yes, you ought to be able to wildcard like that for users. I just took another look at your file_paths and I think you probably intend to have wildcards at the end of those paths? • Because I think most of those are directories you've got, so you probably want a trailing \% if you're looking to cover files in those directories. • t Ted Dorosheff 7 months ago gotcha, okay so would it be like C:\Users\%\foo\% to wildcard the unique username string, and then everything in some subdirectory within the users home directory? C:\Users\foo\% or like this? This one seems like it would only work on some user named "foo" • Also, i'm realizing that we are using specific file_paths entries, as well as specific exclude_paths entries. This seems like it would be unnecessary, because if you are doing file_paths - A - B - C exclude_paths - D - E - F wouldn't "D" "E" "F" be implicitly excluded since they are not under file_paths ? • for reference, here is our current yaml config config: options: events_expiry: 60 config_refresh: 600 host_identifier: instance distributed_interval: 60 decorators: load: - >- SELECT COALESCE((select instance_id FROM ec2_instance_metadata), hostname) as hostname FROM system_info; file_paths: etc: - /etc/group - /etc/passwd - /etc/shadow - /etc/services - /etc/sudoers - /etc/ld.so.preload - /etc/ld.so.conf - /etc/ld.so.conf.d/%% - /etc/pam.d/%% - /etc/resolv.conf - /etc/modules - /etc/hosts - /etc/hostname - /etc/fstab - /etc/rsyslog.conf ssh: - /root/.ssh/%% - /home/%/.ssh/%% - /etc/ssh/%% - /var/lib/sia/keys/ - /var/lib/sia/certs/ logs: - /var/log/secure docker: - /etc/docker/%% - /etc/default/docker - /etc/docker/daemon.json - /usr/bin/containerd - /usr/sbin/runc - /etc/sysconfig/docker - /usr/lib/systemd/system/docker.service - /usr/lib/systemd/system/docker.socket osquery: - /etc/osquery/%% - /usr/share/osquery/packs/%% firewalls: - /etc/sysconfig/iptables - /home/y/conf/yakl/%% - /etc/yakl/conf/%% overrides: platforms: windows: options: events_expiry: 60 config_refresh: 600 host_identifier: instance distributed_interval: 60 decorators: load: - >- SELECT COALESCE((select instance_id FROM ec2_instance_metadata), hostname) as hostname FROM system_info; file_paths: users: - 'C:\users\AppData\Roaming' - 'C:\users\AppData\Local' - 'C:\users\AppData\Local\temp' - >- C:\users\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup - 'C:\users\AppData\Roaming\Microsoft\Windows\Start Menu\Programs' - 'C:\Users\Default' windows: - 'C:\Windows' - 'C:\Windows\temp' - 'C:\Windows\system32\Drivers' - 'C:\Windows\SysWOW64\Drivers' - 'C:\Windows\system32\GroupPolicy\Machine\Scripts' - 'C:\Windows\system32\GroupPolicy\User\Scripts' - 'C:\Windows\system32\Wbem' - 'C:\Windows\SysWOW64\Wbem' - 'C:\Windows\system32\WindowsPowerShell' - 'C:\Windows\SysWOW64\WindowsPowerShell' - 'C:\Windows\Tasks' - 'C:\Windows\system32\Tasks' - 'C:\Windows\AppPatch\Custom%' ProgramData: - 'C:\ProgramData\Microsoft\Windows\Start Menu' - 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs' exclude_paths: windows: - 'C:\Windows\system32\DriverStore\Temp\%' - 'C:\Windows\system32\wbem\Performance%' - 'C:\$WINDOWS.~BT\Sources\%'
C:\Windows\System32\Tasks\OfficeSoftwareProtectionPlatform\SvcRestartTask%