Posts tagged binpkg

As I pointed out back in October, it’s rather easy to create a setup which syncs a built binary package to a remote node (which is serving them to the world – via http,rsync,ftp – pick your poison).

Now, ever since we had slight space problems on miranda (cough my binpkgs cough), I wanted to look into methods on how to get rid of storing them on the buildnode and the webnode. I think now (hehe, it’s only 7pm), I finally managed to get a “proper” bashrc which does a lot of that foo. Take a look at this:

...
syncpkg() {
  # Syncing the binary tbz2 to my webhost
  if is_feature "buildpkg" && \
  [[ -n $REPO_HOST && -n $REPO_BASE && -n $REPO_PATH ]] ; then
    REMOTE_TARGET="$REPO_HOST:$REPO_BASE/$REPO_PATH"
    REMOTE_ECACHE="$REPO_BASE/$REPO_PATH/settings/.ebuild.x"

    einfo "Publishing data to remote repository ($REPO_PATH) on ${REPO_HOST##*@}"

    if ! $( ssh $REPO_HOST "test -d $REPO_BASE/$REPO_PATH/settings" ) ; then
      ssh $REPO_HOST "mkdir -p $REPO_BASE/$REPO_PATH/settings" \
        >> /var/log/syncpkg.log 2>&1
    fi

    if [ -x /usr/bin/q ] ; then
      /usr/bin/qlist -IvCU > /etc/portage/package.list
    fi

    ebegin " [ SYNC: 1/3] /etc/make.conf, /etc/portage/package* to $REPO_PATH/settings"
    rssh /etc/{make.conf,portage/package*,portage/profile,portage/bin,portage/bashrc} \
      $REMOTE_TARGET/settings/ > /var/log/syncpkg.log 2>&1
    eend $?

    ebegin " [ SYNC: 2/3] $PKGDIR to $REPO_PATH"
    rssh $PKGDIR/ $REMOTE_TARGET/ >> /var/log/syncpkg.log 2>&1
    eend $?

    ebegin " [ SYNC: 3/3] Cleaning $PKGDIR"
    rm -rf $PKGDIR/* >> /var/log/syncpkg.log 2>&1
    eend $?

    # We sync our copy of .ebuild.x over to the webnode, as it might not be
    # synced as often / at the same time as the buildnode. Thus the
    # `qpkg --eclean' would remove packages, which *are* in the tree, but
    # the webnode hasn't synced up yet.

    rssh $PORTDIR/.ebuild.x $REMOTE_TARGET/settings/.ebuild.x \
      >> /var/log/syncpkg.log 2>&1

    if $( ssh $REPO_HOST "test -f $REMOTE_ECACHE" ) ; then
      ebegin " [MAINT: 1/3] Removing stale packages in $REPO_PATH"
      ssh $REPO_HOST "CACHE_EBUILD_FILE=$REMOTE_ECACHE qpkg -Eq -P \
        $REPO_BASE/$REPO_PATH/" >> /var/log/syncpkg.log 2>&1
      eend $?
    fi

    # And now remove the .ebuild.x copy again.
    ssh $REPO_HOST "rm -f $REMOTE_ECACHE" >> /var/log/syncpkg.log 2>&1

    # The user ssh'ing, needs to be in the portage group on the remote
    # end. Otherwise, things *will* go wrong (like genpkgindex being
    # unable to write to /var/cache/edb/xpak). Also /var/cache/edb/xpak
    # needs to be owned by portage:portage, as well as group-writeable.

    if $( ssh $REPO_HOST "test -w /var/cache/edb/xpak" ) ; then
      ebegin " [MAINT: 2/3] Regenerating $REPO_PATH/Packages"
      ssh $REPO_HOST "$REPO_BASE/$REPO_PATH/settings/bin/genpkgindex \
        $REPO_BASE/$REPO_PATH/All" >> /var/log/syncpkg.log 2>&1
      eend $?
    fi

    if ! $( ssh $REPO_HOST "test -L $REPO_BASE/$REPO_PATH/Packages" ) ; then
      ebegin " [MAINT: 3/3] Fixing $REPO_PATH/Packages symlink"
      ssh $REPO_HOST "cd $REPO_BASE/$REPO_PATH; rm -f Packages; ln -s \
        All/Packages" >> /var/log/syncpkg.log 2>&1
      eend $?
    fi
  fi
}
...

As you can see, it does a lot of things, which are all connected with binary package repositories (including cleaning up old packages no longer in the tree – trying not to waste too much space). Sadly, I currently have to use a custom patched qpkg version, as the one implementing the –eclean features isn’t in the tree yet. When I talked to Ned the other day, he complained about it being slow (well, yeah — it has to go through the whole tree) which I don’t really see when you look at what it’s doing.

Also, I had a weird phenomenon today happening: the buildnode built a binary package, sent it to the webnode, which ran `qpkg –eclean‘ afterwards. But after that the binary package was gone. “Why” you ask now ? Well, apparently the webnode isn’t synced the same time the buildnode syncs (the webnode is in Germany, the buildnode in the US). So I had to come up with a trick, in order to fool qpkg into not cleaning the freshly built binary packages. See the `rssh’ in front of the qpkg call ? Guess what, that’s the lil’ dirty trick …

Anyway, the full bashrc is available. The next thing I’m gonna have to look at (which Markus already did), is building packages via buildbot.

Update: as you see, I updated the bashrc a bit. That’s because after writing this, I started a new (fresh) binpkg repository (empty), and out of the sudden the thing ain’t syncing correctly (as in no Packages file, no portage settings). Turns out, rsync doesn’t create directories which ain’t there. So another extra `ssh‘ execution to create the settings/ directory inside the repo.