Subsections of Advanced

GoboLinux Scripts

Notes on command-line switches

Many scripts accept command-line options. All options feature a short, one-letter form, and a long form. Following the GNU conventions, short form options are preceded by a single hyphen (as in -a, -b) and long form options are preceded by two hyphend (as in --foo, --bar).

All command-line options have to be passed first, before other types of arguments (file names, package names, etc).

In other words:

FooScript -m Foo

works, but

FooScript Foo -m      # WRONG! Switches must come first.

does not.

Some command-line options accept arguments. These arguments must be passed as the word following the argument, both in short and long forms. For example, say that -s and –long are options that take a parameter. This is the correct way to use them:

FooScript -s value --long another

These are not recognized:

FooScript -s=value --long another      # WRONG! Use distinct tokens.

Each option should be passed in a separate token, even when in short mode. If -a, -b and -c are options for an immaginary FooScript, then

FooScript -a -b -c blah.txt

is correct, but

FooScript -abc blah.txt      # WRONG! Options must be separated.

is not.

All scripts have a --help option (or, in short form, -h). When a program that needs arguments is run without arguments, the help text will be displayed. (Note: Actually, not all scripts conform to this yet, but this is being worked on).

Info

Many of the restrictions above are actually implementation limitations. “Fixing” them is not a high-priority in the project, but patches to lift this restrictions are welcome. See /Programs/Scripts/Current/Functions/OptionParser if you’re curious.

Subsections of GoboLinux Scripts

Notes on Command-Line switches

Many scripts accept command-line options. All options feature a short, one-letter form, and a long form. Following the GNU conventions, short form options are preceded by a single hyphen (as in -a, -b) and long form options are preceded by two hyphens (as in --foo, --bar).

Order of Command-Line Arguments

All command-line options have to be passed first, before other types of arguments (file names, package names, etc). In other words,

FooScript -m Foo

works, but

FooScript Foo -m      # WRONG! Switches must come first.

does not.

Switches taking arguments

Some command-line options accept arguments. These arguments must be passed as the word following the argument, both in short and long forms. For example, say that -s and --long are options that take a parameter. This is the correct way to use them:

FooScript -s value --long another

These are not recognized:

FooScript -s=value --long another      # WRONG! Use distinct tokens.

Do not conglomerate

Each option should be passed in a separate token, even when in short mode. If -a, -b and -c are options for an imaginary FooScript, then

FooScript -a -b -c blah.txt

is correct, but

FooScript -abc blah.txt      # WRONG! Options must be separated.

is not.

Help

All scripts have a --help option (or, in short form, -h). When a program that needs arguments is run without arguments, the help text will be displayed. (Note: Actually, not all scripts conform to this yet, but this is being worked on).

GoboPath

This script is used everywhere inside many scripts. It exports shell variables such as $goboExecutables, $goboUsers, $goboKernel, and so on, which contains a string specifying where in the file system these entries are (/System/Index/bin, /Users, /System/Kernel, and so on).

For example, in the Scripts package, within the bin/ subdirectory are files such as ScriptFunctions or SuggestUpdates, among others. These source GoboPath via:

. GoboPath

This is normally within the Scripts package, at bin/GoboPath. The whole GoboLinux hierarchy is kept as referential prefix in that file. The variable $goboPrefix keeps track as to where GoboLinux is mounted at.

Let’s avoid hardcoding things, sourcing this file and using these variables makes the world a better place :-)

Guidlines for script authors

This section documents the coding style used in GoboLinux shell scripts.

It’s important to note that not all scripts follow these guidelines, because of historical baggage (some of the scripts are older than GoboLinux itself). Patches to correct the non-conformities are welcome.

Indentation and block organization

A few rules of thumb:

  • Three spaces for indentation. Avoid joining do and then in the same line with ;, instead put it on a line by itself, aligned with for, while or if.
  • Prefer using if rather than idioms like && { }, but apply your common sense.
  • Be generous in you use of quotes whenever referring or defining variables, and the ${x} syntax when merging variables inside strings.
  • Bear in mind that esac is ridiculous.
  • When doing weird stuff such as functional-like programming with eval, hide it in a pretty function to pretend it is a bit more readable. Eventually we might make a Functional module. By now, Map() is defined in the Array module.

It’s hard to believe, but the only shell module containing GoboLinux-specific stuff is the one aptly called GoboLinux. Keep that in mind when submitting functions for inclusion in one of the modules.

Names

The idea in the naming convention is to orthogonally describe scope and purpose of each name. We define “local scope” as names that are specific to a given script, and “library scope” as names defined in Scripts modules such as GoboPath, ScriptFunctions or one of the imported function modules.

These are the guidelines:

  • Function names have underscores between words

Example: local_function, Library_Function

  • Variable names do not, they’re just connected

Example: localvariable, LibraryVariable

  • Library names (for functions and variables) have capital letters

Example: Library_Function, LibraryVariable

  • Local names (for functions and variables) are in all-lowercase

Example: local_function, localvariable

  • All-uppercase variables are reserved for standard Unix usage

Example: PATH, LD_LIBRARY_PATH

  • Configuration variables used in .conf files start with the script name in lowercase, resulting in a case style similar to that used in Java variables

Example: compileRecipeDirs, editKeymapLayout

Steps of the compile process

Normally, you shouldn’t need to compile “manually”, i.e., without using the Compile tool. However, it is possible to manually run the same commands that Compile uses.

See also: How To’s → Manual Compile

Overview

Compiling programs in Linux is typically a three-part process: prepare the sources with ./configure, compile them with make, and install the compiled program with make install.

To this process, Gobo adds a source configuration step to target the GoboLinux directory structure, and after the install phase, a step that symlinks the files from the application directory under /Programs to /System/Index. It is this last step that makes programs “visible” to the GoboLinux system.

These are the relevant commands:

PrepareProgram is a wrapper to the first step described above. SandboxInstall is a wrapper to the third step, ensuring safe execution of make install. SymlinkProgram performs the final operations that integrate the new program into the system.

Setting up the sources: PrepareProgram

The PrepareProgram script does two things. It creates a directory hierarchy for the program under /Programs, and it attempts to prepare the sources for compilation.

The syntax for the PrepareProgram is:.

PrepareProgram <program-name> <version-number> [ -- <additional-options> ]

Passing a program name and version number is mandatory. These names are the ones used in the directories under programs. For example,

PrepareProgram --tree Foo 1.0

(the --tree switch) creates the directories /Programs/Foo/Settings, /Programs/Foo/1.0, /Programs/Foo/1.0/bin and so on.

The second task performed by PrepareProgram is to prepare the sources. Since there isn’t a standardized format for distribution of source-code tarballs in the free software world, there is no way to implement completely automated preparation. Fortunately, the popularization of the GNU AutoTools brings us much closer.

PrepareProgram, in this second step, will detect availability of preparation tools and perform one of the following:

  1. If the program includes a configure script generated by GNU autoconf, PrepareProgram will run it, passing the necessary options (mainly --prefix, --sysconfdir) as well as any additional options requested by the user in the command line (as <additional-options>).
  2. Some authors develop their own configure scripts, but due to the popularity of GNU autoconf, design a command line interface similar to that used by autoconf. PrepareProgram tries to detect if a non-autoconf configure script accepts at least the --prefix option, and uses it.

In short, PrepareProgram can be considered a wrapper to configure. Instead of running, for example,

cd foo-1.0
./configure --with-shared=yes

you’ll run

cd foo-1.0
PrepareProgram Foo 1.0 -- --with-shared=yes

SandboxInstall

Strictly speaking, if everything were configured correctly you could simply run make and make install, and all files would be installed under the appropriate subdirectory under Programs/.

To ensure this happens is to guarantee the modularity of the software installation.

To prevent any possibility of files being copied elsewhere in the directory tree than the target under /Programs, GoboLinux conducts the install step in a “sandbox” that is isolated from the rest of the filesystem.

SandboxInstall constructs a container for make install. It uses a sandbox provided by the FiboSandbox script, which configures custom permissions for an allowed set of paths to a special user, fibo. On systems with a UnionFS backend (such as FunionFS, UnionFS-FUSE or OverlayFS), the sandbox is provided by the alternative script UnionSandbox. Installation then runs confined to a sandbox, with no permission to write anywhere but the few places SandboxInstall allows it to, such as /Programs/Foo/Current and /Programs/Foo/Settings. Therefore, no files can be installed in “random places”. A typical call to SandboxInstall looks like this:

SandboxInstall Foo 1.0

If necessary, additional options can be passed. Refer to the SandboxInstall reference page for details.

Linking the sources: SymlinkProgram

The final step in the compilation of a program is performed by the SymlinkProgram. This is the script responsible for creating the symbolic links under /System/Index.

The syntax for SymlinkProgram is:

SymlinkProgram program-name [ version-number ]

The second argument is optional. If no version number is specified, the one linked as Current will be used. A commonly used command-line parameter is -c overwrite, which tells it to overwrite any existing symlinks in case of conflicts (the default action is to preserve the existing links in order to not affect previously existing applications). To integrate a program Foo into the /System structure, overwriting any links, you would type:

SymlinkProgram -c overwrite Foo

SymlinkProgram features many command-line switches, to cover more advanced needs. You can refer to the SymlinkProgram reference page for details if necessary.

GoboHide

To simplify the users’ view of filesystem, GoboHide conceals legacy unix directories such as /usr, /lib, /sbin, and /etc, which contain links to files placed elsewhere under The GoboLinux Filesystem Hierarchy.

GoboHide works by hooking directory read operations directly in the root of the problem: since every readdir() call is translated and performed by the kernel, we have added a list which is kept in kernel, checking every readdir() operation. If the current inode being read is stored in this list, then it simply doesn’t get copied onto the destination buffer, which should be returned to the user.

The user’s interface to the GoboHide ioctl’s is through a userspace tool, called gobohide, and has the following options:

gobohide --help

gobohide: Hide/Unhide a directory

-h, --hide     Hide the directory
-u, --unhide   Unhide the directory
-l, --list     List the hidden directories
--version  Show the program version
--help     Show this message

In order to hide a directory, one would need to run gobohide with -h, passing the target entry. A subsequent ls will not show the hidden entry, then:

ls /

Depot  Mount     System bin  etc  proc  sys  usr
Files  Programs  Users  dev  lib  sbin  tmp  var

gobohide -h /usr
gobohide -h /etc

ls /

Depot  Mount     System  bin  lib   sbin  tmp
Files  Programs  Users   dev  proc  sys   var

This allows entries to be really hidden from the filesystem. But don’t worry, this can be only performed by the superuser, and he/she has power to ask the kernel for the entries being hidden. This ensures that nothing gets hidden without the superuser’s conscience:

gobohide -l

Hidden directories:
/etc
/usr

And the best of it all: you can still access your files inside these hidden entries, and even bash would tell you that files exist in these directories:

if [ -f /etc/fstab ]; then echo "okay"; fi
okay

ls /etc/zshrc

rwxrwxrwx  28 /etc/zshrc -> /Programs/ZSH/Settings/zshrc
========================================================
28 in 1 file - 7614808 kB used (96%), 388760 kB free

GoboHide currently supports hiding entries on any mounted filesystem. Because it is implemented at the level of the Linux virtual filesystem, it is independent of specific filesystems.

Linux Kernel Topics

Linux kernel under GoboLinux

The Linux kernel differs from standard packages. It has nothing to be linked against the legacy tree: no libraries, no binaries, no headers, no manuals or info pages. Moreover, there are many things on a regular system which are very tied to the kernel itself, such as the proc and sys filesystems, the device nodes and the boot loader files.

These characteristics led to the creation of a special directory for the kernel, called /System/Kernel. This tree is organized in the following way:

  • /System/Kernel/Boot - Bootloader files, including the kernel image
  • /System/Kernel/Devices - Device nodes, populated by Udev + Hotplug
  • /System/Kernel/Modules - Kernel modules
  • /System/Kernel/Objects - Sysfs, providing information gathered from Linux 2.6
  • /System/Kernel/Status - The mounted proc filesystem

Installing a kernel

Thanks to Compile, installing a new kernel is pretty straightforward on GoboLinux. The Linux recipe takes into account the existence of a file called config.gz inside /System/Kernel/Status. This file contains the current configuration for the running kernel, and is used thereby to feed the new kernel options.

In short, running Compile Linux will fetch the latest available recipe (which already contains GoboLinux optional patches). After doing that, the kernel itself is automatically downloaded, patched and filled with the current configuration, taken from config.gz.

The menuconfig entry then appears, and allows for the user to modify their kernel options. Just selecting Exit and telling the script to save the changes will finish the user’s interaction with the Linux kernel compilation. After completed, a new entry will appear under /System/Kernel/Modules/$KERNEL_RELEASE, and the new bzImage and System.map files will get installed under /System/Kernel/Boot.

The old bzImage and System.map files aren’t overwritten, though. They’re just symlinks to the current kernel image, and this guarantees that if something goes wrong, a rollback can be done by simply modifying the kernel image at the GRUB’s bootloader prompt, and later by reverting the symlink’s target to the previous release.

To install a kernel which is newer than the available recipe, or one other than vanilla, you may use NewVersion.

If you already have a kernel downloaded, or have a special source package, you may place it in /Data/Compile/Archives.

  • Use NewVersion Linux <Version> https://kernel.org/pub/linux/kernel/v6.12/<your-archive-name> to create a recipe. Using a fake URL is all right if you don’t intend to distribute the recipe.
  • Place any custom patches you need to apply into the /Data/Compile/LocalRecipes/Linux/<Version> directory.
  • Then Compile as usual.

Kernel patches

The Linux recipe comes with a few patches in order to improve the user’s experience with the system. The patchset includes, but is not restricted to, the following modifications:

  • GoboHide: allows the legacy tree to be hidden from userspace applications
  • SquashFS: A compressed filesystem which gets uncompressed on demand. This filesystem is currently used on the GoboLinux ISO, and so it’s interesting to have it in order to get the CD contents easily accessible through the mount command

Subsections of GoboLinux Design Quirks

Files that cannot be symlinks

Symbolic links play a major role in a GoboLinux system, but some programs don’t behave as expected when they are used. This section lists the files that cannot be symlinks.

“That is the reason why /System/Settings is not /System/Index/Settings: it does not only contain links, by definition (or, better put, by necessity)” – Hisham Muhammad

  • /System/Settings/sudoers

The configuration file for sudo must be a regular file. If it is not, sudo will complain and do nothing.

  • /System/Settings/passwd and friends

The settings files used by the Shadow package are quite interesting. Theoretically, they can be symlinks, but there is a caveat: utilities like useradd don’t just modify these files; they remove and recreate them as regular files. Hence, in practice, they cannot be symbolic links.

Sandboxing under GoboLinux

The build system in GoboLinux uses sandboxing to ensure that all the filesystem writes during the software install phase are limited to an appropriate part of the filesystem. GoboLinux 016 ships with two different sandbox implementations.

UnionSandbox is a modern implementation which uses file system unions to achieve isolation. It is the default sandbox installer.

FiboSandbox is a fallback method used when a union-filesystem implementation is not available in the running kernel. It sets up an isolated environment and commands are run by a special user (named fibo) without root privileges.

During the installation phase of the software build process, most build systems call the install program to copy files to their destination directories with the proper ownership and attributes. install belongs to the CoreUtils package.

Since user fibo lacks authority to change file ownership, the link at /System/Index/bin/install points to a wrapper script in the Scripts package.

Under UnionSandbox, this wrapper script translates the superuser name if necessary and calls real_install, a symlink to the CoreUtils install utility, passing along the modified arguments.

Under FiboSandbox, the wrapper discards change-of-owner directives before calling real_install.

Replacements for standard commands

GoboLinux has its own replacements for a few standard commands like su and make. The reasons for making a replacement ranges from adding necessary functionality (su) to making prettier output (make).

While having these “replaced programs” is not a problem in itself (in fact, most provide enhanced functionality which adds to the GoboLinux experience), it’s important to point out these differences here so that you can know what’s going on with these perhaps unexpected behaviors.

The replacements are provided by several different packages. All except install are handled by making an alias in the shell initialization scripts. This way, you can easily disable the alias and restore the original version of a program, if you so wish. They can be often found in their own subdirectory Resources/Wrappers inside the program directory.

  • su - aliased to Su in the Shadow package to handle names other than root as super user
  • sudo - aliased to Sudo in the Sudo package to handle names other than root as super user
  • top - aliased to htop in the Htop package
  • make - aliased to ColorMake in the Scripts package to produce colorful output
  • info - aliased to Info in the Pinfo package
  • man - aliased to Man in the Pinfo package
  • install - install in CoreUtils has been renamed to real_install with install in the Scripts package as a wrapper (see section Sandboxing under GoboLinux for a detailed explanation)
  • which - the Scripts package provides a “which” script which resolves symlinks in the path of the returned program, so that it indicates the /Programs path the binary refers to.

Aliases enabling “enhanced replacements” such as Htop and Pinfo are set in the Environment section of these packages. This way, if you want to stick to plain top and/or info, all you need to do is to remove (or just not install) these replacements.