Channels
  • r

    Robin Powell

    8 months ago
    I'm trying very hard to understand https://github.com/osquery/osquery/pull/6982 and what, exactly, it breaks, and I'm having some trouble.
    I *think* that in <5.0,
    select * from augeas where path LIKE '/etc/hosts%';
    returns a bunch of stuff but in >=5.0 it's going to return nothing.
    I can't see anything else that looks breaking-ish. Anyone have any clues to dole out? 🙂
    I am, in particular, deeply confused by this part:
    ASSERT_EQ(
           SQL("select * from augeas where path LIKE '/etc/hosts/%'").rows().size(),
           0U);

    AFAICT, that *should* get converted to
    match /files/etc/hosts/*
    , which totally works and does not return zero results.
  • s

    seph

    8 months ago
    It's always possible that I introduced a bug. I'll try to look harder sometime in the next couple days.
  • r

    Robin Powell

    8 months ago
    Oh hello PR author. 😄
    I don't actually even know if our (internal) customers use LIKEs with augeas; I was just trying to understand the change for producing a report on 5.0 in general.
  • s

    seph

    8 months ago
    Hi!
    Okay, I’m trying to remember all the things.
    The old version of this table basically ran a
    match /files/*
    and let sqlite filter the output. This presented two issues. The bigger issue was that there was no way to access things _outside_ the files tree. (eg,
    /augeas
    ). Second, and smaller, was that the “return everything” approach just feels wrong to me. I think practically speaking the performance was okay, but it feels like there are dragons about. I’d usually rather call underlying APIs narrowly.
    Wildcard handling between the two was problematic. And TBH, I’m not sure I got it right.
    In augeas, wildcards are tree based.
    *
    is a single level, and
    //*
    is recursive. But in sql, wildcards are simple strings
    And there’s some magic in how
    /
    is treated.
    The end results is, I think: •
    /etc/hosts/%
    is converted to
    /files/etc/hosts/%’. So augeas returns data. But sql filters it. (because the augeas return is is missing that trailing slash)
    •
    /etc/host%
    is converted to
    /files/etc/hosts*
    which augeas has no matches for, because it’s a weird postfix search.
    •
    /etc/host%%
    is converted to
    /files/etc/hosts/*
    which is a full recursion return, and it will get passed back through the sql filter<-hr>For
    path
    I don’t think it’s very meaningful to wildcard a file. Wildcarding a directory is more meaningful.
    
    Compare
    select * from augeas where path LIKE '/etc/%';
    and
    select * from augeas where path LIKE '/etc/%%';
    <-hr>Does that help any?
  • r

    Robin Powell

    8 months ago
    It does, thank you!
    Some questions still though. 😄
    /etc/hosts/%
     is converted to
    /files/etc/hosts/%’.
    ^^ Why doesn't that get converted to
    /filles/etc/hosts/*
    ?  Like, not "why did you make that decision?" but "where in the code does that happen?".<-hr>> (because the augeas return is is missing that trailing slash)
    ^^ I didn't follow that part at alll.<-hr>I still don't feel like I have a handle on what's *breaking*.  I understand the changes and why you made them and I think they make sense, but I can't come up with any examples of queries that work in 4.9.0 that'll break in 5.0<-hr>Oh, actually, in the PR thread there's:
    > This seems reasonable but we should mark it as an API change due to change with queries like
    select * from augeas where path LIKE '/etc/hosts%';
    , where before this would full-scan and have SQL apply the
    LIKE
    filtering.
    ; do I correctly understand that that query currrently returns stuff (which I just checked) but it won't in 5.0 because it gets converted to
    match /files/etc/hosts*
    ?<-hr>If that's the only breaking example we have, that seems like it's no going to come up very much.  :slightly_smiling_face:  Which is yay.
  • s

    seph

    8 months ago
    > /etc/hosts/% is converted to
    /files/etc/hosts/%’.
    > ^^ Why doesn’t that get converted to /filles/etc/hosts/*?  Like, not “why did you make that decision?” butwhere in the code does that happen?”
    Typo, I mean to
    /files/etc/hosts/*
    And all the conversion is in
    patternsFromOsquery
    <https://github.com/osquery/osquery/blob/master/osquery/tables/system/posix/augeas.cpp#L156><-hr>This is a specific case of the the only breaking example I know. Namely, I introduced
    %
    and
    %%
    akin to the existing file pattern as single wild card, vs recursive. Thus breaking a couple of places