Go to file
Franco Fichtner 837c52fa54 build/test: repair os-* globbing
We can't install both types of plugins, so when SUFFIX= we need to
exclude "-devel".  Not the prettiest approach, but should do it in
99% of cases.
2017-08-26 08:27:08 +02:00
build build/test: repair os-* globbing 2017-08-26 08:27:08 +02:00
config config: rebase i386 for 11.1 as well 2017-08-21 20:39:38 +02:00
device device: beware of the mighty "sake", cheers! 2016-06-22 18:50:44 +02:00
doc some mental notes for kernel debugging (wiki stuff) 2015-04-15 08:23:05 +02:00
scripts scripts: add latest link to nightly logs 2017-08-26 07:39:13 +02:00
.gitignore build: allow signing keys to exist in CONFIGDIR 2016-07-05 08:57:14 +02:00
LICENSE README/LICENSE: bumpy copyright and make separate file 2017-01-01 14:43:57 +01:00
Makefile build: preinstall os-dyndns, as discussed in the forums 2017-08-09 09:56:22 +02:00
README.md build: at least allow to clean the log directory 2017-08-05 13:18:52 +02:00

README.md

About the OPNsense tools

In conjunction with src.git, ports.git, core.git and plugins.git they create sets, packages and images for the OPNsense project.

Setting up a build system

Install FreeBSD 11.0-RELEASE (i386 or amd64 depending on your target) on a machine with at least 25GB of hard disk (UFS works better than ZFS) and at least 4GB of RAM to successfully build all standard images. All tasks require a root user. Do the following to grab the repositories (overwriting standard ports and src):

# pkg install git
# cd /usr
# git clone https://github.com/opnsense/tools
# cd tools
# make update

TL;DR

# make dvd

If successful, a dvd image can be found here: /tmp/images

Detailed build steps and options

The build is broken down into individual stages: base, kernel and ports can be built separately and repeatedly without affecting the others. All stages can be reinvoked and continue building without cleaning the previous progress. A final stage assembles all three stages into a target image.

All build steps are invoked via make(1):

# make step OPTION="value"

Available early build options are:

  • CONFIG: reads the below from the specified file
  • SETTINGS: the name of the requested config directory

Available build options are:

  • ADDITIONS: a list of packages/plugins to add to images
  • ARCH: the target architecture if not native
  • DEVICE: loads device-specific modifications, e.g. "a10" (default)
  • FLAVOUR: "OpenSSL" (default), "LibreSSL", "Base"
  • KERNEL: the kernel config to use, e.g. SMP (default)
  • MIRRORS: a list of mirrors to prefetch sets from
  • NAME: "OPNsense" (default)
  • PRIVKEY: the private key for signing sets
  • PUBKEY: the public key for signing sets
  • SPEED: serial speed, e.g. "115200" (default)
  • TYPE: the base name of the top package to be installed
  • SUFFIX: the suffix of top package name (empty, "-devel")
  • UEFI: "yes" for amd64 hybrid images with optional UEFI boot
  • VERSION: a version tag (if applicable)

The default CONFIG file is under "config/SUBDIR/build.conf". It can also be overrided by "/dev/null".

Build the userland binaries, bootloader and administrative files:

# make base

Build the kernel and loadable kernel modules:

# make kernel

Build all the third-party ports:

# make ports

Build additional plugins if needed:

# make plugins

Wrap up our core as a package:

# make core

A dvd live image is created using:

# make dvd

A serial memstick live image is created using:

# make serial

A vga memstick live image is created using:

# make vga

A flash card full disk image is created using:

# make nano

A virtual machine full disk image is created using:

# make vm

Cross-building for other architecures

This feature is currently experimental. It requires to install a qemu package for user mode emulation:

# pkg install qemu-user-static

A viable cross-build target is the Raspberry Pi 2 using the following options:

# make base kernel ARCH=arm:armv6 KERNEL=RPI2

In order to speed up building of using an emulated build, the xtools set can be built:

# make xtools ARCH=arm:armv6

The xtools set is used during the packages build similar to the distfiles set.

# make packages ARCH=arm:armv6

We do not provide ARM images at this point. The sets can be used to bootstrap a running FreeBSD on such devices.

About other scripts and tweaks

Device-specific settings can be found and added in the device/ directory. Of special interest are hooks into the build process for required non-default settings for image builds. The .conf files are shell scrips that can define hooks in the form of e.g.:

serial_hook()
{
    # ${1} is the target file system root
    touch ${1}/my_custom_file
}

These hooks are available for all image types, namely dvd, nano, serial, vga and vm. Device-specific hooks are loaded after config-specific hooks and both of them can coexist in a given build.

Before building images, you can run the regression tests to check the integrity of your core.git modifications plus generate output for the style checker:

# make test

For very fast ports rebuilding of already installed packages the following works:

# make ports-<packagename>[,...]

For even faster ports building it may be of use to cache all distribution files before running the actual build:

# make distfiles

Updating all or individual repositories can be done as follows:

# make update[-<repo1>[,...]]

Available update options are: core, plugins, ports, src, tools

Compiled sets can be prefetched from a mirror, while removing any previously available set:

# make prefetch-<option>[,...] VERSION=version.to.prefetch

Available prefetch options are:

  • base: prefetch the base set
  • kernel: prefetch the kernel set
  • packages: prefetch the packages set

Core packages (pristine copies) can be batch-built using:

# make core-<repo_branch_or_tag>[,...]

Package sets ready for web server deployment are automatically generated and modified by ports.sh, plugins.sh and core.sh. If signing keys are available, the packages set will be signed twice, first embedded into repository metadata (inside) and then again as a flat file (outside) to ensure integrity.

Signing for all sets can be redone or applied to a previous run that did not sign by invoking:

# make sign

A verification of all available set signatures is done via:

# make verify

Nano images can be adjusted in size using an argument as follows:

# make nano-<size>

Virtual machine images come in varying disk formats and sizes. The default format is vmdk with 20G and 1G swap. If you want to change that you can manually alter the invoke using:

# make vm-<format>[,<size>[,<swap>]]

Available virtual machine disk formats are:

  • qcow: Qemu, KVM (legacy format)
  • qcow2: Qemu, KVM (not backwards-compatible)
  • raw: Unformatted (sector by sector)
  • vhd: VirtualPC, Hyper-V, Xen (dynamic size)
  • vhdf: Azure, VirtualPC, Hyper-V, Xen (fixed size)
  • vmdk: VMWare, VirtualBox (dynamic size)

The swap argument is either its size or set to "off" to disable.

Release sets can be built using:

# make release VERSION=product.version.number_revision

Kernel, base, packages and release sets are stored under /tmp/sets

All final images are stored under /tmp/images

A couple of build machine cleanup helpers are available via the clean script:

# make clean-<option>[,...]

Available clean options are:

  • base: remove base set
  • distfiles: remove distfiles set
  • dvd: remove dvd image
  • core: remove core from packages set
  • images: remove all images
  • kernel: remove kernel set
  • logs: remove all logs
  • nano: remove nano image
  • obj: remove all object directories
  • packages: remove packages set
  • plugins: remove plugins from packages set
  • ports: alias for "packages" option
  • release: remove release set
  • serial: remove serial image
  • sets: remove all sets
  • src: reset kernel/base build directory
  • stage: reset main staging area
  • vga: remove vga image
  • vm: remove vm image
  • xtools: remove xtools set

The ports tree has a few of our modifications and is sometimes a bit ahead of FreeBSD. In order to keep the local changes, a skimming script is used to review and copy upstream changes:

# make skim[-<option>]

Available options are:

  • used: review and copy upstream changes
  • unused: copy unused upstream changes
  • (none): all of the above

In case a release was wrapped up, the base package list and obsoleted files need to be regenerated. This is done using:

# make rebase

Shall any debugging be needed inside the build jail, the following command will use chroot(8) to enter the active build jail:

# make chroot[-<subdir>]

There's also the posh way to boot a final image using bhyve(8):

# make boot-<image>

Normally the build scripts will pick up version numbers based on commit tags or given version tags or a date-type string. Should it not fit your needs, you can change the name using:

# make rename-<set>[,<another_set>] VERSION=<new_name>

The available targets are: base, kernel and package.

The current state or a tagged state of required build repositories on the system can be printed using:

# make info[-<version>]

Last but not least, in case build variables needs to be inspected, they can be printed selectively using:

# make print-<variable1>[,<variable2>]