Channels
Martin Pöhlmann
7 months agoNow to the problem: I have one windows machine (virtual box) that refuses to "talk" to fleet. Honestly I do not know where the error is. Fleet shows the machine (with some basic info but as "never fetched"). But I cannot run queries. This is also the same if I reset the osquery db on the machine. And even more curious I cannot delete the machine from fleet. The error of the HTTP request in chrome dev tools shows as follows:{ "message": "unsupported Scan, storing driver.Value type <nil> into type *time.Time", "errors": [ { "name": "base", "reason": "unsupported Scan, storing driver.Value type <nil> into type *time.Time" } ] }
Lucas Rodriguez
7 months agoAs for the error log above, it looks like a bug, we'll try to reproduce on our end and get back to you.
Were you able to find osquery logs in the VM?
(to troubleshoot)
Also, did you upgrade from a previous version of fleet? If so, any warnings about upgrades when starting fleet?Martin Pöhlmann
7 months agoYes, I've updated from 4.3.1. I think the upgrade logs are gone, but I did not remember anything special. When I start fleet the following shows up:fleet_1 | {"component":"crons","cron":"cleanups","details":"looping through ids: running visitFunc for queries: getting user_time p50 for query 1: timestamp: 2021-11-22T13:45:38+01:00: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'row_number, mm.* FROM (\n\t\tSELECT d.scheduled_query_id, d.user_time, d.executions' at line 4","err":"aggregating query stats","level":"error","ts":"2021-11-22T12:45:38.017855521Z"} fleet_1 | {"component":"crons","cron":"cleanups","details":"looping through ids: running visitFunc for scheduled_queries: getting user_time p50 for scheduled_query 4: timestamp: 2021-11-22T13:45:38+01:00: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'row_number, mm.* FROM (\n\t\tSELECT d.scheduled_query_id, d.user_time, d.executions' at line 4","err":"aggregating scheduled query stats","level":"error","ts":"2021-11-22T12:45:38.019472394Z"}
And the strange thing is that I do not have any logs/files under C:\ProgramFiles\osquery\logs on the machineLucas Rodriguez
7 months agoOK, did you run
before running the new version of fleet?fleet prepare dbMartin Pöhlmann
7 months agosure, that's run everytime the container startscommand: sh -c "/usr/bin/fleet prepare db --no-prompt && /usr/bin/fleet serve"
also the machine was added AFTER the upgradeLucas Rodriguez
7 months agoOK, let me check with the teamMartin Pöhlmann
7 months agocool, thxLucas Rodriguez
7 months agoQQ: MySQL server and version?Martin Pöhlmann
6 months ago@Lucas Rodriguez 8.0.26 - I'm seeing the same error now for another newly onboarded machineLucas Rodriguez
6 months agoHi Martin! Another user hit the same error, we have an issue and are working on a fix: https://github.com/fleetdm/fleet/issues/3095Martin Pöhlmann
6 months agoThanks. I subscribed to the ticket. Will the clients resume to work normally after the fix? If there is need for manual intervention (which I have no problem with) it's okay to mention that in the ticket.Lucas Rodriguez
6 months agoI think there are two issues,1. One is the fleet side bug (unsupported Scan). 2. The other one is some issue on the osquery side, are you running osquery manually on the VM? (could you get logs?)Martin Pöhlmann
6 months agono, I run it as service (osquery MSI installer). The logs folder is empty and I did not find anything in win eventlog. Are there any other places to look for?
@User let us continue discussion on GitHub (https://github.com/fleetdm/fleet/issues/3095) if you do not mind. I've added our osquery config there. If there is anything else I could provide, please let me know.Lucas Rodriguez
6 months agoHi Martin I just saw your comment, great timing, I've started working on the issue today.Martin Pöhlmann
6 months agoGood to hear 🙂
It's already late in 🇩🇪 , but if you need anything, I'll provide tomorrowLucas Rodriguez
6 months agoThere are a few related issues we are working on:1. Proper Orbit logs on Windows 2. Allow setting a "platform" to policies.Martin Pöhlmann
6 months agowe're not using orbit
(maybe that'S also part of the issue)Lucas Rodriguez
6 months agoNot sure, we do support vanilla osquery.
I'll continue troubleshooting and ask questions in the issue.
Again, thanks a lot for the detailed comment.Martin Pöhlmann
6 months agoCool, thanks.
I'll have another look in the fleet server logs while I am on it
so far the only suspicious status logs (delivered to server) are regarding the policies. I'm wondering if we could auto-detect the machines to send queries to? This is already the case for the query editor as it suggests where the query may run on.
but as 1st step, manual selection would be more than okayLucas Rodriguez
6 months agoCorrect, we will allow configuring platform for policies and run them as live queries to test them out first (coming soon).- f
Flngen Flugen
6 months agoWe're having this same issue with fleetdm/fleet:main image on k8s and osquery 5.0.1 Lucas Rodriguez
6 months agoHi Flngen! We are working on a fix for this that we'll try to include in fleet 4.6.2. (ETA: some day this week, hopefully Thursday).Martin Pöhlmann
6 months ago@Lucas Rodriguez I've updated to 6.4.2 just now. The hosts that were throwing this error still show the same error 🙂 will it take some time to get these updated? I also cannot delete these hosts, nor force-refresh these via the API (same error)Lucas Rodriguez
6 months agoHi Martin!, can you double check the version in top right ->
-> fleet version should show up in the bottom right.My Account
Also, do you have access to the MySQL database? To help us troubleshoot better.Martin Pöhlmann
6 months ago
I think I can somehow hop on the docker mysql containerLucas Rodriguez
6 months agoOK, if possible, please run the following query
(feel free to not include any sensitive data like hostnames)SELECT * FROM fleet.hosts h LEFT JOIN fleet.host_seen_times hst ON h.id = hst.host_idMartin Pöhlmann
6 months agoSee PM
View count: 3