Docs GODI Archive
Projects Blog Knowledge

GODI Project:
Home 
Highlights 
Screenshots 
Get GODI 
Docs 
Mailing List 
Resources 
Packages 
Workgroups 
OASIS 
Why Ocaml? 
**********************************************************************

                            Skylift

**********************************************************************


This document gives a quick overview of Skylift, the next release of
the GODI package management tools (godi-tools, and bootstrap).

Skylift is currently in a testing phase. Skylift must use the
bootstraps named skylift-<date>.tar.gz (found in
download.camlcity.org). There is no way to upgrade existing GODI
deployments using Rocketboost to Skylift.

The packaging format for sources has not changed in Skylift. You can
build all existing packages. There are slight changes in binary
packages, though (explained below).

Skylift includes a lot of new features:

 - Profiles are a new concept which allows the user to configure
   non-standard package sources easily. This includes sources
   provided by third parties, but also filters on package lists,
   as well as patches (see below).

 - Binary packages are handled in a better way, so that it is
   now reasonable to distribute pre-built packages. Especially,
   Skylift defines a scheme to describe whether binary packages
   are compatible to each other. Remote repositories are also
   supported. There is a user interface for selecting binary
   packages.

 - Patching has become way easier than before. If an extracted
   source is edited, Skylift can often create the patch file with
   a single command. The patches can be saved permanently in three
   places: as a local patch, in the package definition (useful
   when you are package maintainer), or, and this is new, in the
   profile.

 - One can now export sources, binaries, and profiles. There are
   special scripts that copy these objects to export directories,
   where they are provided in the right format so that other
   GODI deployments can use them as remote repository. When a
   web server is pointed to these directories, this also works
   over HTTP.

 - There is now built-in support for HTTP and FTP in godi_console
   (i.e. wget is no longer needed). There is an option to activate
   HTTPS and WebDAV (post-boot). Furthermore, GODI can also access
   password-protected resources. Together with the export feature
   this means one can set up repositories for private use.

----------------------------------------------------------------------
1. Profiles
----------------------------------------------------------------------

A profile defines:

 - The GODI section (i.e. "3.12" when everything is for OCaml 3.12)

 - URLs from where to download build files (i.e. the special GODI
   description how to build and install sources, including metadata
   like version number, homepage, maintainer etc.).

 - URLs from where to download distfiles (i.e. the original tarballs
   provided by the authors of a software).

 - A list of package conditions. For example, one can define to
   only take certain packages from a URL but ignore all others provided
   there. Or, one can restrict the package version, especially one can
   request a fixed version.

 - Package options. If it is already known that a certain package build
   option is required, one can put these into the profile.

 - Patches that are specific to the profile.

When Skylift is freshly installed, the user can select two profiles:

 - head_3.12: This profile includes the latest versions of the official
   GODI packages for 3.12. This profile is the default.

 - archive_3.12: This profile includes also historic package versions.

These profiles should not be edited by the user because these profiles
are automatically kept up to date, and can thus be overwritten when a
new version is available. It is easy to copy profiles, however, so
one can do on the command-line:

$ godi_console profile -enable archive_3.12              # enable this
$ godi_console profile -copy-to my_3.12                  # copy this

Now edit my_3.12, and enable it:

$ godi_console profile -enable my_3.12

The profile is, so far, a single file in ini format, stored at
<prefix>/build/profiles/<name>/profile.ini

Profiles can be downloaded from the web:

$ godi_console rprofile -site http://foo/path/ -get <name>

The profile needs to exist in this directory in a special archive
form. You can also upload profiles to FTP and WebDAV servers:

$ godi_console rprofile -site http://foo/path/ -put <name>

The -site can also be a normal path (e.g. -site /tmp).


----------------------------------------------------------------------
2. Binary packages
----------------------------------------------------------------------

As before, when a build is completed, the resulting binary files are
not only installed, but also packaged up in <prefix>/build/packages/All.
The file format is roughly the same as before, but there are nevertheless
a few changes:

 - The name includes a checksum suffix which is derived from compatibility
   information, e.g. foo-3.14@BUILD:b4cdf5.tgz. This means that there can
   now be several packages foo-3.14 which have been built with different
   options, or which are dependent on different required packages.

 - The binary package includes extended information about binary
   compatibility. Especially, there is now a reasonable build version
   block describing all files in detail (including checksums) that
   may be relevant here. See the output of godi_info -long for getting
   an impression. Note that the dependencies have an attached MD5
   checksum (e.g. godi-ocaml-3.12.1@MD5:72f61f87b905332732216794fb77d2c3
   might appear here). The checksum is taken from the text in build_version.

The directory <prefix>/build/packages/All includes now a "table of contents",
the PKGMANIFEST file. It is automatically updated whenever a package is put
into this directory. This saves godi_console from having to look into every
package file just for checking what is available (and for calculating
dependencies).

As mentioned, there is now a user interface for selecting binary packages.
After the bootstrap, you see here only the installed packages, but you cannot
select new ones. This is because the variable

GODI_BINPKG

is empty. You can set this in godi.conf, and put a path to a package
directory here (or an URL for HTTP or FTP):

GODI_BINPKG = <prefix>/build/packages/All

would mean that the UI offers now all packages here for installation, i.e.
what you have built locally before. Normally, however, you would put here
the URL of an external package provider.

How to become such a provider? Very easy - just copy your packages to
a directory that is made public by a web server. Of course, you also need
the PKGMANIFEST file, and this can simply be generated by running

$ godi_console ptool -in-binary "$f" -out-manifest

for all package files $f, and by concatenating the output in PKGMANIFEST.

----------------------------------------------------------------------
3. Patching
----------------------------------------------------------------------

If you need to fix a source before you can use it, there has always been
the option to make a patch. In Skylift, this is now very easy:

(a) Go into the build directory of the package: 

$ cd <prefix>/build/<cat>/<package>

(where <cat> is one of apps, conf, godi).

(b) Extract the sources, and apply all patches that already exist:

$ godi_make patch

(c) The source are now unpacked in work/<name>. Fix your problem, and
try to build the package. You are (technically) done with the step when

$ godi_make build

works (from the package directory).

(d) Now, we can create the patch. The easy case is that you neither had
to create an extra file nor to delete a file to make the build working
again:

$ godi_make makepatch

This outputs the patch on the screen, but does not store it yet. There are
three places for saving the patch:

  (i) Save the patch as local site patch in a directory you configure.
 (ii) Save the patch in the profile.
(iii) Save the patch in the package definition.

Place (i) will be the right one for most users. Add to godi.conf the
directory where to save patches:

LOCALPATCHES = /a/path/to/a/directory

Then just run

$ godi_make makepatch SAVE=LOCAL

This command will output the paths of the created files. If there are already
patches for this package, the new patch is appended.

Place (ii) is interesting if you want to apply modifications to a
package source in a profile. Just use SAVE=PROFILE then, and the patch
is stored in the currently enabled profile.

Place (iii) applies for package maintainers. This puts the patch into the
patches/ directory, and adds the checksum to the distinfo file. In order
to select this option, use SAVE=PACKAGE.

If the patch requires new files to add, specify these as

$ godi_make makepatch ADD_FILES="<path> <path> ..."

The <path> muse be relative to the root of the source. Deleted files
are specified by

$ godi_make makepatch DEL_FILES="<path> <path> ..."

Happy patching!

----------------------------------------------------------------------
4. Exporting
----------------------------------------------------------------------

The export scripts expect that you configure a directory where you
want to store the exported objects: Add to godi.conf:

EXPORT_DESTDIR = /path/to/dir

The export scripts will here create subdirectories for buildfiles,
distfiles, profiles, packages, and seeds.



4.1. Exporting builds

Go into the package directory:

$ cd <prefix>/build/<cat>/<package>

and run there:

$ godi_make export-build

This will create a buildfile in ${EXPORT_DESTDIR}/buildfiles, and
copies the distfile to ${EXPORT_DESTDIR}/distfiles. Also, the index
files "packages.ini", "packages_all.ini", and "available.new" are
updated in ${EXPORT_DESTDIR}/buildfiles.

Other GODI deployments can use these builds if the directories (made
public via a web server) are entered into the source of the profile:

Edit profile.ini, and add a new source section:

[source_3]
name = mysource
build_sites = <URL of buildfiles directory>
master_sites = <URL of distfiles directory>
include = *-*

Don't forget to add source_3 to the sources variable.

If the archive flag is set for the source, the index file
"packages_all.ini" is taken, otherwise "packages.ini". The file
"available.new" is provided for backward compatibility with Rocketboost.



4.2. Exporting profiles

Enable the profile to export:

$ godi_console profile -enable myprofile

Then run from an arbitrary package directory:

$ godi_console export-profile

The profile is put in the right format into ${EXPORT_DESTDIR}/profiles.
Import it into another GODI deployment with:

$ godi_console rprofile -site http://foo/directory/ -get myprofile



4.3. Exporting binary packages

One option was already described above in section 2: Copy the package
files to the export directory manually, and regenerate there the
PKGMANIFEST file.

If you just want to add a single package to the export: Run

$ godi_make export-package

from the package directory. It is required that the package is currently
installed. The script writes the package tarball to
${EXPORT_DESTDIR}/packages, and updates the PKGMANIFEST there.

As already explained, you can use the packages from another GODI
deployment by setting GODI_BINPKG so it points to the export directory.



4.4. Exporting seeds

A seed file is a minimalistic GODI deployment as tarball. The idea is
that you create the seed file on machine A, and unpack it on machine B,
and get a working GODI that is capable of installing packages. The seed
file may even be preconfigured, so the right profile is enabled, and the
packages are taken from the right source or right binary repository.
Seeds are useful when you want to make the installation of GODI very
easy for users, and don't want that they bootstrap GODI on their own.

Of course, the machines A and B must be - in some way - "compatible":
same CPU, same OS, maybe even same OS version. You will have to
experiment to find this out. Considerations:

 - If machine B installs binary packages made on A: This will normally
   work if both machines have the same OS version, or if B has a newer
   OS version. If A has the newer version, you may run into problems.

 - As the seed always contains godi_console in binary form, you will
   have to ensure that at least this program is runnable on B.

 - If machine B will install from sources, be aware that even here OS
   version compatibility problems may exist. It is normally less
   problematic, though.

 - Avoid that B mixes sources and binaries.

In order to create a seed: Go into an arbitrary package directory, and
run there

$ godi_make export-seed

The new seed is placed into ${EXPORT_DESTDIR}/seeds/.

Install the seed file on another computer:

- Create the destination directory:

  $ mkdir <prefix>

  The prefix must be identical to the machine on which the seed was
  created. GODI is generally non-relocatable after the bootstrap.

- Untar the seed file:

  $ cd <prefix>
  $ tar xzf seed-...tar.gz


----------------------------------------------------------------------
5. "wtool"
----------------------------------------------------------------------

The built-in support for accessing remote resources is made available
by a component called wtool. After the bootstrap, only a simple version
of wtool is available:

 - Restricted to HTTP and FTP
 - Passwords supported
 - Proxies supported

You can rebuild godi-tools and enable the "power wtool":

 - HTTP, HTTPS, and FTP
 - HTTP and HTTPS also understand the extended command set of WebDAV

You will need this if you want to use remote repositories that are
only reachable via these protocols.

Special note about users and passwords: You can use the special string
"GODI_USER" in URLs to denote that the variable ${GODI_USER} is to be
substituted. For example, http://GODI_USER@server/directory would
actually use the contents of ${GODI_USER} as user name. GODI_USER is
normally set in godi.conf. This way, you can inject the user name into
URLs defined in profiles.

Passwords inside URLs are ignored. The user is normally asked for a
password, and the password is stored into ~/.godi_pw. Note that the
password is not securely encrypted there.

This web site is published by Informatikbüro Gerd Stolpmann
Powered by Caml