Channels
  • Martin Pöhlmann

    Martin Pöhlmann

    7 months ago
    Now to the problem: I have one windows machine (virtual box) that refuses to "talk" to fleet. Honestly I do not know where the error is. Fleet shows the machine (with some basic info but as "never fetched"). But I cannot run queries. This is also the same if I reset the osquery db on the machine. And even more curious I cannot delete the machine from fleet. The error of the HTTP request in chrome dev tools shows as follows:
    {
      "message": "unsupported Scan, storing driver.Value type <nil> into type *time.Time",
      "errors": [
        {
          "name": "base",
          "reason": "unsupported Scan, storing driver.Value type <nil> into type *time.Time"
        }
      ]
    }
    Any ideas?
  • Lucas Rodriguez

    Lucas Rodriguez

    7 months ago
    As for the error log above, it looks like a bug, we'll try to reproduce on our end and get back to you.
    Were you able to find osquery logs in the VM?
    (to troubleshoot)
    Also, did you upgrade from a previous version of fleet? If so, any warnings about upgrades when starting fleet?
  • Martin Pöhlmann

    Martin Pöhlmann

    7 months ago
    Yes, I've updated from 4.3.1. I think the upgrade logs are gone, but I did not remember anything special. When I start fleet the following shows up:
    fleet_1          | {"component":"crons","cron":"cleanups","details":"looping through ids: running visitFunc for queries: getting user_time p50 for query 1: timestamp: 2021-11-22T13:45:38+01:00: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'row_number, mm.* FROM (\n\t\tSELECT d.scheduled_query_id, d.user_time, d.executions' at line 4","err":"aggregating query stats","level":"error","ts":"2021-11-22T12:45:38.017855521Z"}
    fleet_1          | {"component":"crons","cron":"cleanups","details":"looping through ids: running visitFunc for scheduled_queries: getting user_time p50 for scheduled_query 4: timestamp: 2021-11-22T13:45:38+01:00: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'row_number, mm.* FROM (\n\t\tSELECT d.scheduled_query_id, d.user_time, d.executions' at line 4","err":"aggregating scheduled query stats","level":"error","ts":"2021-11-22T12:45:38.019472394Z"}

    And the strange thing is that I do not have any logs/files under C:\ProgramFiles\osquery\logs on the machine
  • Lucas Rodriguez

    Lucas Rodriguez

    7 months ago
    OK, did you run
    fleet prepare db
    before running the new version of fleet?
  • Martin Pöhlmann

    Martin Pöhlmann

    7 months ago
    sure, that's run everytime the container starts
    command: sh -c "/usr/bin/fleet prepare db --no-prompt && /usr/bin/fleet serve"

    also the machine was added AFTER the upgrade
  • Lucas Rodriguez

    Lucas Rodriguez

    7 months ago
    OK, let me check with the team
  • Martin Pöhlmann

    Martin Pöhlmann

    7 months ago
    cool, thx
  • Lucas Rodriguez

    Lucas Rodriguez

    7 months ago
    QQ: MySQL server and version?
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    @Lucas Rodriguez 8.0.26 - I'm seeing the same error now for another newly onboarded machine
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    Hi Martin! Another user hit the same error, we have an issue and are working on a fix: https://github.com/fleetdm/fleet/issues/3095
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    Thanks. I subscribed to the ticket. Will the clients resume to work normally after the fix? If there is need for manual intervention (which I have no problem with) it's okay to mention that in the ticket.
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    I think there are two issues,1. One is the fleet side bug (unsupported Scan). 2. The other one is some issue on the osquery side, are you running osquery manually on the VM? (could you get logs?)
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    no, I run it as service (osquery MSI installer). The logs folder is empty and I did not find anything in win eventlog. Are there any other places to look for?
    @User let us continue discussion on GitHub (https://github.com/fleetdm/fleet/issues/3095) if you do not mind. I've added our osquery config there. If there is anything else I could provide, please let me know.
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    Hi Martin I just saw your comment, great timing, I've started working on the issue today.
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    Good to hear 🙂
    It's already late in 🇩🇪 , but if you need anything, I'll provide tomorrow
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    There are a few related issues we are working on:1. Proper Orbit logs on Windows 2. Allow setting a "platform" to policies.
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    we're not using orbit
    (maybe that'S also part of the issue)
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    Not sure, we do support vanilla osquery.
    I'll continue troubleshooting and ask questions in the issue.
    Again, thanks a lot for the detailed comment.
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    Cool, thanks.
    I'll have another look in the fleet server logs while I am on it
    so far the only suspicious status logs (delivered to server) are regarding the policies. I'm wondering if we could auto-detect the machines to send queries to? This is already the case for the query editor as it suggests where the query may run on.
    but as 1st step, manual selection would be more than okay
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    Correct, we will allow configuring platform for policies and run them as live queries to test them out first (coming soon).
  • f

    Flngen Flugen

    6 months ago
    We're having this same issue with fleetdm/fleet:main image on k8s and osquery 5.0.1
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    Hi Flngen! We are working on a fix for this that we'll try to include in fleet 4.6.2. (ETA: some day this week, hopefully Thursday).
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    @Lucas Rodriguez I've updated to 6.4.2 just now. The hosts that were throwing this error still show the same error 🙂 will it take some time to get these updated? I also cannot delete these hosts, nor force-refresh these via the API (same error)
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    Hi Martin!, can you double check the version in top right ->
    My Account
    -> fleet version should show up in the bottom right.
    Also, do you have access to the MySQL database? To help us troubleshoot better.
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago

    I think I can somehow hop on the docker mysql container
  • Lucas Rodriguez

    Lucas Rodriguez

    6 months ago
    OK, if possible, please run the following query
    SELECT * FROM fleet.hosts h LEFT JOIN fleet.host_seen_times hst ON h.id = hst.host_id
    (feel free to not include any sensitive data like hostnames)
  • Martin Pöhlmann

    Martin Pöhlmann

    6 months ago
    See PM