Hey there I m having a bit of a hard time tracking down wher

Question

Hey there I m having a bit of a hard time tracking down where some `socket events` may be disappearing to On a low load development machine I ve been finding that I don t appear to be entries in the `socket events` table for some connections whereas others are just fine I was wondering if someone could point me in the right direction of how to debug this a bit further thread

Peter · Answer

For a bit of background the test system is Ubuntu 18 04 5 running Kernel 5 4 0 1051 aws with osquery 4 9 0 I m using the Audit source for `socket events` and not eBPF and There s no auditd installed on the machine

Peter · Answer

The configuration flags in use has `audit allow sockets` and `audit allow config` set to true `disable events` and `disable audit` are set to false I ve used `auditctl l` to get the installed rules when osquery is running and I m able to see the osquery installed rule s for `connect bind execve execveat` at syscall exit

Peter · Answer

I ve also increased `events max` temporarily on this machine to `500000` as a just in case

Peter · Answer

As an example running osqueryi using the flags above and running `curl <https www google com>` I correctly see an entry in the `socket events` a few moments later ```osquery gt select path remote address from socket events where remote port = 443 + + + | path | remote address | + + + | usr bin curl | 142 250 69 206 | + + +```

Peter · Answer

However if I compile a pretty generic reverse shell which just opens a TCP socket to a remote server spawns a shell and attaches stdin stderr and stdout to the socket I never see an entry in the `socket events` table

Peter · Answer

What s especially strange is if I `strace` osquery and look for all `recvfrom` events I can see an audit event for the associated `connect` syscall being sent over the audit socket to osquery Apologies for the screen grab but it s a bit easier to visualise

Peter · Answer

Another screen grab apologies that shows the whole scenario The Victim is establishing an outbound TCP connection on TCP 1234 to the C2 via a compiled reverse shell A `curl` to Google shows up in the `socket events` table shortly before but the outbound connection for TCP 1234 doesn t appear to ever appear in the `socket events` table

Peter · Answer

Does anyone have any pointers around how to best track down what s happening to these audit events Once the events reach osquery Is there some additional levels of debug past ` verbose` which might assist here or am I in attach a debugger territory slightly smiling face

Stefano Bonicatti · Answer

< Peter> you can try enabling ` audit debug=true`

Stefano Bonicatti · Answer

Peter · Answer

Ah nice thank you for that It appears to confirm that I m seeing the events in the log but not in the `socket events` table ```I0805 15 45 11 034433 9994 auditdnetlink cpp 778 1300 audit 1628178308 957 370181 arch=c000003e syscall=42 success=no exit= 115 a0=3 a1=c00008402c a2=10 a3=0 items=0 ppid=10035 pid=10064 auid=4294967295 uid=998 gid=1002 euid=998 suid=998 fsuid=998 egid=1002 sgid=10 02 fsgid=1002 tty=pts6 ses=4294967295 comm= shell exe= home ssm user shell shell key= null ```

Stefano Bonicatti · Answer

oh I see the syscall has failed though We discard failed syscalls

Stefano Bonicatti · Answer

Peter · Answer

Ah gotcha It doesn t look like the socket has actually failed as connection completes without another call to `connect` Looking at the fields in the audit message it appears that it may be non blocking as result seems to indicate the exit code as ` 115` Looking at `errno h` for Kernel 5 4 it appears this is `EINPROGRESS`

Stefano Bonicatti · Answer

You re right sorry I gave it a quick look It was actually reported here <https github com osquery osquery issues 6609> We are indeed missing tracing for non blocking calls I don t recall specifically if there was some complication in handling them or they are simply a missing feature though

Peter · Answer

Not a bother at all thank you for having a look all the same What concerns me a little is it appears that Golang uses non blocking sockets by default since at least Go1 13 and likely before <https github com golang go blob go1 13 src net sock cloexec go L21> I guess that d mean these `connect` calls are invisible to osquery due to the issue mentioned above at least for the time being slightly smiling face

Stefano Bonicatti · Answer

I was discussing this with Alessandro which has originally done the work he will add some details about that on the issue I ve linked

Peter · Answer

Ah that s fantastic thank you for your help

Peter · Answer

I dropped a note as to my findings on that issue as it s been a couple of days of digging so hopefully it helps someone else joy

Stefano Bonicatti · Answer

Thank you That s much appreciated

alessandrogario · Answer

hey < Peter> this issue is being addressed here

Peter · Answer

That s great news thanks for the ping and the help slightly smiling face