Channels
  • a

    alessandrogario

    2 years ago
  • s

    seph

    2 years ago
    So you think it is git version dependent? If it’s consistently just augeas, does making it not-shallow help? Does using a release that’s within the last 50 commits help?
  • a

    alessandrogario

    2 years ago
    Yeah, it is; if you were to try it on the same git version that we use on the CI, it will work
    The augeas repository is not huge, so I would just clone it fully; there are some others however that are causing the issue and they are huge
    I think we have this issue with around ~4 submodules
  • s

    seph

    2 years ago
    Interesting. Do you know how far back the commits for the other broken ones are?
    Alternately, we could just slap something in that complains if the git version is below XXX
  • a

    alessandrogario

    2 years ago
    it is very erratic, I don't think we can just filter with a simple
    version > xxx

    I think the shallow modules were squashed inside the main commit that contains the third-party refactor
    Yeah, it's all in 6481b34e2382ffb5b4c469de3157c63e0ed23ea3
  • s

    seph

    2 years ago
    I was trying to read it last night — it looks like the submodules stuff is all cmake we wrote?
    Are we linking to specific shas or tags? What happens if you add something like
    cd ...moduledir && git fetch -t
  • a

    alessandrogario

    2 years ago
    I would:1. Remove the shallow everywhere; I'm not really a fan of that 2. Add ~/.git/modules to the cache 3. CMake will automatically get the delta from the remote repo OR 1. Remove the shallow everywhere 2. Add a pre-build script Script would do this:1. Fully clone all the remotes inside the cached CI folder 2. Go though all the submodules in osquery and update the remote to point inside the cached submodules 3. git submodule sync --recursive 4. Resume normal operations
    Yeah, all the submodules are pointing to an upstream release tag; exceptions are bad upstream repositories such as libudev ('cause it's now systemd and they kind of nuked the version history)
    (Some of them got upgraded to a new tagged version)
    The script would be easy to build with
    git submodule foreach
  • s

    seph

    2 years ago
    I dunno… I really like using shallow for this sort of thing.
    I sorta think you’re running into submodules being kinda a mess. ISTR they’re not super well supported
    Either way though, if there’s a simple enough reproduce case, I’d try to play with fetch, and keep shallow.
    Or think a lot harder and ditch the submodules
  • a

    alessandrogario

    2 years ago
    I can reproduce it 100% of the times on my machine
    it is really that bad on recent git versions (I have 2.23.0 here)
  • s

    seph

    2 years ago
    If you’re up for it, try adding that fetch in? I don’t think it unshallows
  • a

    alessandrogario

    2 years ago
    It doesn't, it needs
    git fetch --unshallow
    (there's a submodule command for that, i don't remember it)
    sometimes you just restart the cmake configuration, git tries again and it works
    but some modules are stubborn and will never work (like google logging)
  • s

    seph

    2 years ago
    Weird. Is there enough of a reproduce case that I can test this with bare git commands in a test repo? I can’t easily interate inside the osquery dependency system.
  • a

    alessandrogario

    2 years ago
    since this happens at configure time, it should be usable anywhere without installing anything but git and cmake
    this should reproduce the issue
    git clone <https://github.com/osquery/osquery> && mkdir build && cd build && cmake ../osquery
  • s

    seph

    2 years ago
    That’s not something I can iterate from — I can’t quickly tinkering with the git commands in the depths of cmake
    Maybe I’ll try though. What git version is expected to fail
    version 2.21.0 on macOS works
    works fine on macOS git version 2.23.0
  • a

    alessandrogario

    2 years ago
    Interesting, then it's more random than I thought
    we can maybe ask more info to the other contributors that encountered this issue
  • s

    seph

    2 years ago
    macOS vs linux may matter
  • s

    sharvil

    2 years ago
    works fine on git 2.17.2 on macOS as well
  • theopolis

    theopolis

    2 years ago
    Let’s try MITMing our connection to GH for failing versions, and/or stracing and narrowing this down. I’m interested in the root cause.
  • s

    seph

    2 years ago
    Is there an OS this is likely to fail on?
  • theopolis

    theopolis

    2 years ago
    for more context, I had tried switching all the URIs to
    git://
    and adding
    .git
    and was still able to reproduce. I was testing inside of
    oss-fuzz
    's Dockerfile, which is:
    FROM <http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
    and I tried the
    git
    available from a basic
    apt-get install
    as well as the 2.23 version available here: https://askubuntu.com/questions/568591/how-do-i-install-the-latest-version-of-git-with-apt/568596
  • s

    seph

    2 years ago
    Trying to make a test case… The docker container
    <http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
    has a cmake that’s way too old. Ditto ubuntu16.04. Though now I’m trying with the cmake in stefano’s docker files
  • theopolis

    theopolis

    2 years ago
    you can just download an install a new cmake
  • s

    seph

    2 years ago
    Yes, that’s what’s in stefano’s dockerfile
    Anyhow, I don’t have a good reproduce case yet. Entirely because cmake does a bunch of toolchain sanity checks, and I don’t have a linux build environment for it.
  • a

    alessandrogario

    2 years ago
    you can in theory just launch the git command manually
    git submodule update --init --depth=1 libraries/cmake/source/augeas/src
    should do the trick
  • s

    seph

    2 years ago
    Woo! That’s a reproduce case.
    In
    ubuntu:16.04
    I can get that easily
    Using the above instruction, it fails on ubuntu16, git version 2.7.4 But it works with ubuntu18, git version 2.17.1
    <http://gcr.io/oss-fuzz-base/base-builder|gcr.io/oss-fuzz-base/base-builder>
    uses git 2.7
    https://github.com/boostorg/boost/issues/245 seems to match both what we’re seeing here, and my understanding of this error.
  • theopolis

    theopolis

    2 years ago
    sometimes you just restart the cmake configuration, git tries again and it works
    this is so odd, I am seeing this behavior with 2.23.0 too
    I think I figured it out this commit breaks the behavior we want (merged between 2.21.0 and 2.22.0-rc0) https://github.com/git/git/commit/34066f06614d6dbc1955c92ba0c3a17dbfeb0077#diff-7891df559f581cdcc9168363e0f932a1