• t

    Ted Dorosheff

    8 months ago
    Hey all, so i'm still unable to get any ntfs_journal_events data from my windows hosts, after recently adding file paths to my agent settings. I do however see data from other queries such as hardware events and firmware snapshots. My file events query:
    SELECT action, category, old_path, path, file_attributes, time FROM ntfs_journal_events;
    my agent config
    config:
      options:
        events_expiry: 60
        config_refresh: 600
        host_identifier: instance
        distributed_interval: 60
      decorators:
        load:
          - >-
            SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
            hostname) as hostname FROM system_info;
      file_paths:
        etc:
          - /etc/group
          - /etc/passwd
          - /etc/shadow
          - /etc/services
          - /etc/sudoers
          - /etc/ld.so.preload
          - /etc/ld.so.conf
          - /etc/ld.so.conf.d/%%
          - /etc/pam.d/%%
          - /etc/resolv.conf
          - /etc/modules
          - /etc/hosts
          - /etc/hostname
          - /etc/fstab
          - /etc/rsyslog.conf
        ssh:
          - /root/.ssh/%%
          - /home/%/.ssh/%%
          - /etc/ssh/%%
          - /var/lib/sia/keys/
          - /var/lib/sia/certs/
        logs:
          - /var/log/secure
        docker:
          - /etc/docker/%%
          - /etc/default/docker
          - /etc/docker/daemon.json
          - /usr/bin/containerd
          - /usr/sbin/runc
          - /etc/sysconfig/docker
          - /usr/lib/systemd/system/docker.service
          - /usr/lib/systemd/system/docker.socket
        osquery:
          - /etc/osquery/%%
          - /usr/share/osquery/packs/%%
        firewalls:
          - /etc/sysconfig/iptables
          - /home/y/conf/yakl/%%
          - /etc/yakl/conf/%%
    overrides:
      platforms:
        windows:
          options:
            events_expiry: 60
            config_refresh: 600
            host_identifier: instance
            distributed_interval: 60
          decorators:
            load:
              - >-
                SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
                hostname) as hostname FROM system_info;
          file_paths:
            users:
              - 'C:\users\AppData\Roaming'
              - 'C:\users\AppData\Local'
              - 'C:\users\AppData\Local\temp'
              - >-
                C:\users\AppData\Roaming\Microsoft\Windows\Start
                Menu\Programs\Startup
              - 'C:\users\AppData\Roaming\Microsoft\Windows\Start Menu\Programs'
              - 'C:\Users\Default'
            windows:
              - 'C:\Windows'
              - 'C:\Windows\temp'
              - 'C:\Windows\system32\Drivers'
              - 'C:\Windows\SysWOW64\Drivers'
              - 'C:\Windows\system32\GroupPolicy\Machine\Scripts'
              - 'C:\Windows\system32\GroupPolicy\User\Scripts'
              - 'C:\Windows\system32\Wbem'
              - 'C:\Windows\SysWOW64\Wbem'
              - 'C:\Windows\system32\WindowsPowerShell'
              - 'C:\Windows\SysWOW64\WindowsPowerShell'
              - 'C:\Windows\Tasks'
              - 'C:\Windows\system32\Tasks'
              - 'C:\Windows\AppPatch\Custom%'
            ProgramData:
              - 'C:\ProgramData\Microsoft\Windows\Start Menu'
              - 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs'
          exclude_paths:
            windows:
              - 'C:\Windows\system32\DriverStore\Temp\%'
              - 'C:\Windows\system32\wbem\Performance%'
              - 'C:\$WINDOWS.~BT\Sources\%'
              - 'C:\Windows\Installer\%'
              - 'C:\Windows\System32\Tasks\Adobe Acrobat Update Task%'
              - 'C:\Windows\System32\Tasks\Adobe Flash Player Updater%'
              - >-
                C:\Windows\System32\Tasks\OfficeSoftwareProtectionPlatform\SvcRestartTask%
    @Mystery Incorporated helped with this before, if you could just give me another look that would be much appreciated. I'm really lost.
  • Mystery Incorporated

    Mystery Incorporated

    8 months ago
    @Ted Dorosheff show flags file please, what you need to set is probably in flags file
  • zwass

    zwass

    8 months ago
    @Ted Dorosheff Try live querying
    select * from osquery_events
    to see whether the publishers are activated and receiving any events.
  • t

    Ted Dorosheff

    7 months ago
    hey @zwass happy new year. Back at the keyboard today, and i think you may be right about the event publishers not being activated.
    Log file created at: 2021/12/27 11:01:32
    Running on machine: COOLEDRULED
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    I1227 11:01:32.819942  5964 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration
  • zwass

    zwass

    7 months ago
    Seems like @Mystery Incorporated has it right then -- you'll need to configure the appropriate flags.
  • t

    Ted Dorosheff

    7 months ago
    okay, do these look like the right flags?
    --enable_windows_events_publisher=true 
    --enable_windows_events_subscriber=true 
    --enable_ntfs_event_publisher=true
    They were currently missing from my flags file
  • although going off the log, looks like i may only need
    --enable_ntfs_event_publisher=true
  • zwass

    zwass

    7 months ago
    That sounds right. The
    windows_events
    ones would be for the
    windows_events
    table.
  • t

    Ted Dorosheff

    7 months ago
    alright so i just tried live querying the host, and now at least i'm not getting the error about the event publisher not enabled, but i'm still not seeing any journal events
  • I ran osqueryi from cmd, with all those flags set.
    osqueryi.exe -S --config_path="C:\Program Files\osquery\osquery.conf" --disable_events=false --enable_windows_events_publisher=true --enable_windows_events_subscriber=true --enable_ntfs_event_publisher=true --verbose
    But i'm guessing that file_paths still needs to be defined? We have file_paths set in our yaml config in fleetDM, but i would assume that isn't being loaded if i'm running osqueryi like this?
  • osquery> select * from osquery_events;
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | name                     | publisher                | type       | subscriptions | events | refreshes | active |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 1             | 17     | 0         | 1      |
    | ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 0             | 0      | 934       | 1      |
    | ntfs_journal_events      | ntfs_event_publisher     | subscriber | 0             | 0      | 0         | 1      |
    | powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    | windows_events           | WindowsEventLogPublisher | subscriber | 1             | 437    | 0         | 1      |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    osquery>
  • zwass

    zwass

    7 months ago
    Yeah that's right.
  • t

    Ted Dorosheff

    7 months ago
    before i ran the query, i dropped a .txt file into one of the paths which we've got listed in our yaml config
  • zwass

    zwass

    7 months ago
    Can you run
    osqueryi
    with the normal flagfile so it pulls the config from Fleet?
  • t

    Ted Dorosheff

    7 months ago
    yes
  • C:\Program Files\osquery>osqueryi.exe
    Using a [1mvirtual database[0m. Need help, type '.help'
    osquery> select * from osquery_events;
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | name                     | publisher                | type       | subscriptions | events | refreshes | active |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 0             | 0      | 0         | 0      |
    | ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 0             | 0      | 0         | 0      |
    | ntfs_journal_events      | ntfs_event_publisher     | subscriber | 0             | 0      | 0         | 0      |
    | powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    | windows_events           | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    osquery>
  • zwass

    zwass

    7 months ago
    Is that top line your command? If so, missing
    --flagfile
  • t

    Ted Dorosheff

    7 months ago
    ahh okay
  • whats the command to stop, and just return to the shell?
  • or do i have to kill the pid entirely and open a new shell?
  • doing this all on a VM, and anytime i need to escalate to an admin shell, i need to swap windows and grab a password
  • zwass

    zwass

    7 months ago
    Does control-d do it?
  • t

    Ted Dorosheff

    7 months ago
    ahh yes, thanks
  • whew that took longer than expected....
    C:\Program Files>osquery\osqueryi.exe --flagfile="C:\Program Files\osquery\osquery.flags"
    Using a [1mvirtual database[0m. Need help, type '.help'
    osquery>
    osquery> select * from osquery_events;
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | name                     | publisher                | type       | subscriptions | events | refreshes | active |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 0             | 0      | 0         | 1      |
    | ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 3             | 0      | 0         | 0      |
    | ntfs_journal_events      | ntfs_event_publisher     | subscriber | 3             | 0      | 0         | 1      |
    | powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    | windows_events           | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    osquery>
  • osquery.db was locked by another process
  • zwass

    zwass

    7 months ago
    Looks like the subscriptions are going now... Does it pick up the changes?
  • select * from ntfs_journal_events;
  • t

    Ted Dorosheff

    7 months ago
    osquery> select * from osquery_events;
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | name                     | publisher                | type       | subscriptions | events | refreshes | active |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    | WindowsEventLogPublisher | WindowsEventLogPublisher | publisher  | 0             | 0      | 0         | 1      |
    | ntfs_event_publisher     | ntfs_event_publisher     | publisher  | 3             | 0      | 0         | 0      |
    | ntfs_journal_events      | ntfs_event_publisher     | subscriber | 3             | 0      | 0         | 1      |
    | powershell_events        | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    | windows_events           | WindowsEventLogPublisher | subscriber | 0             | 0      | 0         | 0      |
    +--------------------------+--------------------------+------------+---------------+--------+-----------+--------+
    osquery> select * from ntfs_journal_events;
    osquery>
  • nothing
  • perhaps my yaml config isn't done correctly? file_paths incorrectly defined, or some syntax error in the config. side note: if we want to monitor user directories, could we wildcard where the user name would be? like:
    C:\Users\*\foo
    C:\Users\*\bar
    Running this in an enterprise environment where the username is going to be different on every machine.
  • zwass

    zwass

    7 months ago
    Yes, you ought to be able to wildcard like that for users. I just took another look at your file_paths and I think you probably intend to have wildcards at the end of those paths?
  • Because I think most of those are directories you've got, so you probably want a trailing
    \%
    if you're looking to cover files in those directories.
  • t

    Ted Dorosheff

    7 months ago
    gotcha, okay so would it be like
    C:\Users\%\foo\%
    to wildcard the unique username string, and then everything in some subdirectory within the users home directory?
    C:\Users\foo\%
    or like this? This one seems like it would only work on some user named "foo"
  • Also, i'm realizing that we are using specific
    file_paths
    entries, as well as specific
    exclude_paths
    entries. This seems like it would be unnecessary, because if you are doing
    file_paths
       - A
       - B
       - C
    exclude_paths
       - D
       - E
       - F
    wouldn't "D" "E" "F" be implicitly excluded since they are not under
    file_paths
    ?
  • for reference, here is our current yaml config
    config:
      options:
        events_expiry: 60
        config_refresh: 600
        host_identifier: instance
        distributed_interval: 60
      decorators:
        load:
          - >-
            SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
            hostname) as hostname FROM system_info;
      file_paths:
        etc:
          - /etc/group
          - /etc/passwd
          - /etc/shadow
          - /etc/services
          - /etc/sudoers
          - /etc/ld.so.preload
          - /etc/ld.so.conf
          - /etc/ld.so.conf.d/%%
          - /etc/pam.d/%%
          - /etc/resolv.conf
          - /etc/modules
          - /etc/hosts
          - /etc/hostname
          - /etc/fstab
          - /etc/rsyslog.conf
        ssh:
          - /root/.ssh/%%
          - /home/%/.ssh/%%
          - /etc/ssh/%%
          - /var/lib/sia/keys/
          - /var/lib/sia/certs/
        logs:
          - /var/log/secure
        docker:
          - /etc/docker/%%
          - /etc/default/docker
          - /etc/docker/daemon.json
          - /usr/bin/containerd
          - /usr/sbin/runc
          - /etc/sysconfig/docker
          - /usr/lib/systemd/system/docker.service
          - /usr/lib/systemd/system/docker.socket
        osquery:
          - /etc/osquery/%%
          - /usr/share/osquery/packs/%%
        firewalls:
          - /etc/sysconfig/iptables
          - /home/y/conf/yakl/%%
          - /etc/yakl/conf/%%
    overrides:
      platforms:
        windows:
          options:
            events_expiry: 60
            config_refresh: 600
            host_identifier: instance
            distributed_interval: 60
          decorators:
            load:
              - >-
                SELECT COALESCE((select instance_id FROM ec2_instance_metadata),
                hostname) as hostname FROM system_info;
          file_paths:
            users:
              - 'C:\users\AppData\Roaming'
              - 'C:\users\AppData\Local'
              - 'C:\users\AppData\Local\temp'
              - >-
                C:\users\AppData\Roaming\Microsoft\Windows\Start
                Menu\Programs\Startup
              - 'C:\users\AppData\Roaming\Microsoft\Windows\Start Menu\Programs'
              - 'C:\Users\Default'
            windows:
              - 'C:\Windows'
              - 'C:\Windows\temp'
              - 'C:\Windows\system32\Drivers'
              - 'C:\Windows\SysWOW64\Drivers'
              - 'C:\Windows\system32\GroupPolicy\Machine\Scripts'
              - 'C:\Windows\system32\GroupPolicy\User\Scripts'
              - 'C:\Windows\system32\Wbem'
              - 'C:\Windows\SysWOW64\Wbem'
              - 'C:\Windows\system32\WindowsPowerShell'
              - 'C:\Windows\SysWOW64\WindowsPowerShell'
              - 'C:\Windows\Tasks'
              - 'C:\Windows\system32\Tasks'
              - 'C:\Windows\AppPatch\Custom%'
            ProgramData:
              - 'C:\ProgramData\Microsoft\Windows\Start Menu'
              - 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs'
          exclude_paths:
            windows:
              - 'C:\Windows\system32\DriverStore\Temp\%'
              - 'C:\Windows\system32\wbem\Performance%'
              - 'C:\$WINDOWS.~BT\Sources\%'
              - 'C:\Windows\Installer\%'
              - 'C:\Windows\System32\Tasks\Adobe Acrobat Update Task%'
              - 'C:\Windows\System32\Tasks\Adobe Flash Player Updater%'
              - >-
                C:\Windows\System32\Tasks\OfficeSoftwareProtectionPlatform\SvcRestartTask%
  • thanks so much for your help with this BTW