Posted
almost 3 years
ago
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid
... [More]
packages maintained in
Git. This post is part of a series about the progress of the Janitor.
The FOSS world uses a wide variety of different build tools; given a git
repository or tarball, it can be hard to figure out how to build and install a
piece of software.
Humans will generally know what build tool a project is using when they check
out a project from git, or they can read the README. And even then, the answer
may not always be straightforward to everybody. For automation, there
is no obvious place to figure out how to build or install a project.
Debian
For Debian packages, Debian maintainers generally will have determined
that the appropriate tools to invoke are, and added appropriate invocations to
debian/rules. This is really nice when rebuilding all of Debian - one can just
invoke debian/rules - a consistent interface - and it will in turn invoke the
right tools to build the package, meeting a long list of requirements.
With newer versions of debhelper
and most common build systems, debhelper can figure a lot of this out
automatically - the maintainer just has to add the
appropriate build and run time dependencies.
However, debhelper needs to be consistent in its behaviour per compat level -
otherwise builds might start failing with different versions of debhelper, when
the autodetection logic is changed. debhelper can also only do the right thing
if all the necessary dependencies are present. debhelper also only functions
in the context of a Debian package.
Ognibuild
Ognibuild is a new tool that figures
out the build system in use by an upstream project, as well as the other
dependencies it needs. This information can then be used to invoke said build
system, or to e.g. add missing build dependencies to a Debian package.
Ognibuild uses a variety of techniques to work out what the dependencies for an
upstream package are:
Extracting dependencies and other requirements declared in build system
metadata (e.g. setup.py)
Attempting builds and parsing build logs for missing dependencies (repeating
until the build succeeds), calling out to buildlog-consultant
Once it is determined which dependencies are missing, they can be resolved in a
variety of ways. Apt can be invoked to install missing dependencies on Debian
systems (optionally in a chroot) or ecosystem-specific tools can be used to do
so (e.g. pypi or cpan).
Instead of installing packages, the tool can also simply
inform the user about the missing packages and commands to install them, or
update a Debian package appropriately (this is what deb-fix-build does).
The target audience of ognibuild are people who need to (possibly from
automation) build a variety of projects from different ecosystems or users who
are looking to just install a project from source. Developers who are just
hacking on e.g. a Python project are better off directly invoking the
ecosystem-native tools rather than a wrapper like ognibuild.
Supported ecosystems
(Partially) supported ecosystems currently include:
Combinations of make and autoconf, automake or CMake
Python, including fetching packages from pypi
Perl, including fetching packages from cpan
Haskell, including fetching from hackage
Ninja/Meson
Maven
Rust, including fetching packages from crates.io
PHP Pear
R, including fetching packages from CRAN and Bioconductor
For a full list, see the README.
Usage
Ognibuild provides a couple of top-level subcommands that will seem familiar to anybody who has used a couple of other build systems:
ogni clean - remove build artifacts
ogni dist - create a dist tarball
ogni build - build the project in the current directory
ogni test - run the test suite
ogni install - install the project somewhere
ogni info - display project information including discovered build system and dependencies
ogni exec - run an arbitrary command but attempt to resolve issues like missing dependencies
These tools all take a couple of common options:
—resolve=apt|auto|native
Specifies how to resolve any missing dependencies:
apt: install the appropriate dependency using apt
native: install dependencies using native tools like pip or cpan
auto: invoke either apt or native package install, depending on whether
the current user is allowed to invoke apt
—schroot=name
Run inside of a schroot.
—explain
do not make any changes but tell the user which native on apt packages they could install.
There are also subcommand-specific options, e.g. to install to a specific directory on restrict which tests are run.
Examples
Creating a dist tarball
1
2
3
4
5
6
7
8
9
% git clone https://github.com/dulwich/dulwich
% cd dulwich
% ogni --schroot=unstable-amd64-sbuild dist
…
Writing dulwich-0.20.21/setup.cfg
creating dist
Creating tar archive
removing 'dulwich-0.20.21' (and everything under it)
Found new tarball dulwich-0.20.21.tar.gz in /var/run/schroot/mount/unstable-amd64-sbuild-974d32d7-6f10-4e77-8622-b6a091857e85/build/tmpucazj7j7/package/dist.
Installing ldb from source, resolving dependencies using apt
1
2
3
4
5
6
7
8
9
% wget https://download.samba.org/pub/ldb/ldb-2.3.0.tar.gz
% tar xvfz ldb-2.3.0.tar.gz
% cd ldb-2.3.0
% ogni install --prefix=/tmp/ldb
…
+ install /tmp/ldb/include/ldb.h (from include/ldb.h)
…
Waf: Leaving directory `/tmp/ldb-2.3.0/bin/default'
'install' finished successfully (11.395s)
Running all tests from XML::LibXML::LazyBuilder
1
2
3
4
5
6
% wget ``https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz`_ `_
% tar xvfz XML-LibXML-LazyBuilder-0.08.tar.gz
Cd XML-LibXML-LazyBuilder-0.08
% ogni test
…
Current Status
ognibuild is still in its early stages, but works well enough that it can
detect and invoke the build system for most of the upstream projects packaged
in Debian. If there are buildsystems that it currently lacks support for or
other issues, then I’d welcome any bug reports.
[Less]
|
Posted
almost 3 years
ago
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid
... [More]
packages maintained in
Git. This post is part of a series about the progress of the
Janitor.
The upstream ontologist is a project that extracts metadata about upstream projects in a consistent format. It does this with a combination of heuristics and reading ecosystem-specific metadata files, such as Python’s setup.py, rust’s Cargo.toml as well as e.g. scanning README files.
Supported Data Sources
It will extract information from a wide variety of sources, including:
Python package metadata (PKG-INFO, setup.py, setup.cfg, pyproject.toml)
package.json
composer.json
package.xml
Perl package metadata (dist.ini, META.json, META.yml, Makefile.PL)
Perl POD files
GNU configure files
R DESCRIPTION files
Rust Cargo.toml
maven pom.xml
metainfo.xml
.git/config
SECURITY.md
DOAP
Haskell cabal files
Ruby gemspec files
go.mod
README{,.rst,.md} files
Debian packaging metadata (debian/watch, debian/control, debian/rules, debian/get-orig-source.sh, debian/copyright, debian/patches)
Supported Fields
Fields that it currently provides include:
Homepage: homepage URL
Name: name of the upstream project
Contact: contact address of some sort of the upstream (e-mail, mailing list URL)
Repository: VCS URL
Repository-Browse: Web URL for viewing the VCS
Bug-Database: Bug database URL (for web viewing, generally)
Bug-Submit: URL to use to submit new bugs (either on the web or an e-mail address)
Screenshots: List of URLs with screenshots
Archive: Archive used - e.g. SourceForge
Security-Contact: e-mail or URL with instructions for reporting security issues
Documentation: Link to documentation on the web:
Wiki: Wiki URL
Summary: one-line description of the project
Description: longer description of the project
License: Single line license description (e.g. "GPL 2.0") as declared in the metadata[1]
Copyright: List of copyright holders
Version: Current upstream version
Security-MD: URL to markdown file with security policy
All data fields have a “certainty” associated with them (“certain”, “confident”, “likely” or “possible”), which gets set depending on how the data was derived or where it was found. If multiple possible values were found for a specific field, then the value with the highest certainty is taken.
Interface
The ontologist provides a high-level Python API as well as two command-line tools that can write output in two different formats:
guess-upstream-metadata writes DEP-12-like YAML output
autodoap writes DOAP files
For example, running guess-upstream-metadata on dulwich:
% guess-upstream-metadata
:2: (INFO/1) Duplicate implicit target name: "contributing".
Name: dulwich
Repository: https://www.dulwich.io/code/
X-Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md
X-Version: 0.20.21
Bug-Database: https://github.com/dulwich/dulwich/issues
X-Summary: Python Git Library
X-Description: |
This is the Dulwich project.
It aims to provide an interface to git repos (both local and remote) that
doesn't call out to git directly but instead uses pure Python.
X-License: Apache License, version 2 or GNU General Public License, version 2 or later.
Bug-Submit: https://github.com/dulwich/dulwich/issues/new
Lintian-Brush
lintian-brush can update DEP-12-style debian/upstream/metadata files that hold information about the upstream project that is packaged as well as the Homepage in the debian/control file based on information provided by the upstream ontologist. By default, it only imports data with the highest certainty - you can override this by specifying the --uncertain command-line flag.
[1]
Obviously this won't be able to describe the full licensing situation for many projects. Projects like scancode-toolkit are more appropriate for that.
[Less]
|
Posted
almost 3 years
ago
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid
... [More]
packages maintained in
Git. This post is part of a series about the progress of the Janitor.
The upstream ontologist is a project that extracts metadata about upstream projects in a consistent format. It does this with a combination of heuristics and reading ecosystem-specific metadata files, such as Python’s setup.py, rust’s Cargo.toml as well as e.g. scanning README files.
Supported Data Sources
It will extract information from a wide variety of sources, including:
Python package metadata (PKG-INFO, setup.py, setup.cfg, pyproject.toml)
package.json
composer.json
package.xml
Perl package metadata (dist.ini, META.json, META.yml, Makefile.PL)
Perl POD files
GNU configure files
R DESCRIPTION files
Rust Cargo.toml
maven pom.xml
metainfo.xml
.git/config
SECURITY.md
DOAP
Haskell cabal files
Ruby gemspec files
go.mod
README{,.rst,.md} files
Debian packaging metadata (debian/watch, debian/control, debian/rules, debian/get-orig-source.sh, debian/copyright, debian/patches)
Supported Fields
Fields that it currently provides include:
Homepage: homepage URL
Name: name of the upstream project
Contact: contact address of some sort of the upstream (e-mail, mailing list URL)
Repository: VCS URL
Repository-Browse: Web URL for viewing the VCS
Bug-Database: Bug database URL (for web viewing, generally)
Bug-Submit: URL to use to submit new bugs (either on the web or an e-mail address)
Screenshots: List of URLs with screenshots
Archive: Archive used - e.g. SourceForge
Security-Contact: e-mail or URL with instructions for reporting security issues
Documentation: Link to documentation on the web:
Wiki: Wiki URL
Summary: one-line description of the project
Description: longer description of the project
License: Single line license description (e.g. “GPL 2.0”) as declared in the metadata[1]
Copyright: List of copyright holders
Version: Current upstream version
Security-MD: URL to markdown file with security policy
All data fields have a “certainty” associated with them (“certain”, “confident”, “likely” or “possible”), which gets set depending on how the data was derived or where it was found. If multiple possible values were found for a specific field, then the value with the highest certainty is taken.
Interface
The ontologist provides a high-level Python API as well as two command-line tools that can write output in two different formats:
guess-upstream-metadata writes DEP-12-like YAML output
autodoap writes DOAP files
For example, running guess-upstream-metadata on dulwich:
% guess-upstream-metadata
:2: (INFO/1) Duplicate implicit target name: "contributing".
Name: dulwich
Repository: https://www.dulwich.io/code/
X-Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md
X-Version: 0.20.21
Bug-Database: https://github.com/dulwich/dulwich/issues
X-Summary: Python Git Library
X-Description: |
This is the Dulwich project.
It aims to provide an interface to git repos (both local and remote) that
doesn't call out to git directly but instead uses pure Python.
X-License: Apache License, version 2 or GNU General Public License, version 2 or later.
Bug-Submit: https://github.com/dulwich/dulwich/issues/new
Lintian-Brush
lintian-brush can update DEP-12-style debian/upstream/metadata files that hold information about the upstream project that is packaged as well as the Homepage in the debian/control file based on information provided by the upstream ontologist. By default, it only imports data with the highest certainty - you can override this by specifying the —uncertain command-line flag.
[1]
Obviously this won’t be able to describe the full licensing situation for many projects. Projects like scancode-toolkit are more appropriate for that.
[Less]
|
Posted
almost 3 years
ago
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid
... [More]
packages maintained in
Git. This post is part of a series about the progress of the
Janitor.
In my last blogpost, I introduced the buildlog
consultant - a tool that can
identify many reasons why a Debian build failed.
For example, here’s a fragment of a build log where the Build-Depends lack
python3-setuptools:
849
850
851
852
853
854
855
856
857
858
dpkg-buildpackage: info: host architecture amd64
fakeroot debian/rules clean
dh clean --with python3,sphinxdoc --buildsystem=pybuild
dh_auto_clean -O--buildsystem=pybuild
I: pybuild base:232: python3.9 setup.py clean
Traceback (most recent call last):
File "/<>/setup.py", line 2, in
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'
E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean
The buildlog consultant can identify the line in bold as being key, and interprets it:
% analyse-sbuild-log --json ~/build.log
{
"stage": "build",
"section": "Build",
"lineno": 857,
"kind": "missing-python-module",
"details": {"module": "setuptools", "python_version": 3, "minimum_version": null}
}
Automatically acting on buildlog problems
A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends.
Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools:
% apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py
python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py
The deb-fix-build command automates these steps:
It builds the package using sbuild; if the package successfully builds then it just exits successfully
It tries to identify the problem by looking through the build log; if it can't or if it's a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code
It tries to find a dependency that can address the problem
It updates Build-Depends in debian/control or Depends in debian/tests/control
Go to step 1
This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again.
For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this:
% deb-fix-build
Using output directory /tmp/tmpyz0nkgqq
Using sbuild chroot unstable-amd64-sbuild
Using fixers: …
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4')
Using apt-file to search apt contents
Adding build dependency: python3-setuptools-scm (>= 4)
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None)
Adding build dependency: python3-toml
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’].
And in our Git repository, we see these changes as well:
% git log -p
commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master)
Author: Jelmer Vernooij
Date: Sun Apr 4 02:35:56 2021 +0100
Add missing build dependency on python3-toml.
diff --git a/debian/control b/debian/control
index 5b854dc..3b27b73 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1
Author: Jelmer Vernooij
Date: Sun Apr 4 02:35:34 2021 +0100
Add missing build dependency on python3-setuptools-scm (>= 4).
diff --git a/debian/control b/debian/control
index a476cc2..5b854dc 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
Using deb-fix-build
You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements
for using it are that:
The package is maintained in Git
A sbuild schroot is available for use
Caveats
deb-fix-build is fairly easy to understand, and if it doesn't work then you're no worse off than you were without it - you'll have to add your own Build-Depends.
That said, there are a couple of things to keep in mind:
At the moment, it doesn't distinguish between general, Arch or Indep Build-Depends.
It can only add dependencies for things that are actually in the archive
Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn't always get it right
[Less]
|
Posted
almost 3 years
ago
In my last blogpost, I introduced the buildlog
consultant - a tool that can
identify many reasons why a Debian build failed.
For example, here’s a fragment of a build log where the Build-Depends lack
python3-setuptools:
849
850
851
852
853
854
855
... [More]
856
857
858
dpkg-buildpackage: info: host architecture amd64
fakeroot debian/rules clean
dh clean --with python3,sphinxdoc --buildsystem=pybuild
dh_auto_clean -O--buildsystem=pybuild
I: pybuild base:232: python3.9 setup.py clean
Traceback (most recent call last):
File "/<>/setup.py", line 2, in
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'
E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean
The buildlog consultant can identify the line in bold as being key, and interprets it:
% analyse-sbuild-log --json ~/build.log
{"stage": "build", "section": "Build", "lineno": 857, "kind": "missing-python-module", "details": {"module": "setuptools", "python_version": 3, "minimum_version": null}}
Automatically acting on buildlog problems
A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends.
Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools:
% apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py
python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py
The deb-fix-build command automates these steps:
It builds the package using sbuild; if the package successfully builds then it just exits successfully
It tries to identify the problem by looking through the build log; if it can't or if it's a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code
It tries to find a dependency that can address the problem
It updates Build-Depends in debian/control or Depends in debian/tests/control
Go to step 1
This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again.
For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this:
% deb-fix-build
Using output directory /tmp/tmpyz0nkgqq
Using sbuild chroot unstable-amd64-sbuild
Using fixers: …
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4')
Using apt-file to search apt contents
Adding build dependency: python3-setuptools-scm (>= 4)
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None)
Adding build dependency: python3-toml
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’].
And in our Git repository, we see these changes as well:
% git log -p
commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master)
Author: Jelmer Vernooij
Date: Sun Apr 4 02:35:56 2021 +0100
Add missing build dependency on python3-toml.
diff --git a/debian/control b/debian/control
index 5b854dc..3b27b73 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1
Author: Jelmer Vernooij
Date: Sun Apr 4 02:35:34 2021 +0100
Add missing build dependency on python3-setuptools-scm (>= 4).
diff --git a/debian/control b/debian/control
index a476cc2..5b854dc 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
Using deb-fix-build
You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements
for using it are that:
The package is maintained in Git
A sbuild schroot is available for use
Caveats
deb-fix-build is fairly easy to understand, and if it doesn't work then you're no worse off than you were without it - you'll have to add your own Build-Depends.
That said, there are a couple of things to keep in mind:
At the moment, it doesn't distinguish between general, Arch or Indep Build-Depends.
It can only add dependencies for things that are actually in the archive
Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn't always get it right
[Less]
|
Posted
almost 3 years
ago
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid
... [More]
packages maintained in
Git. This post is part of a series about the progress of the Janitor.
In my last blogpost, I introduced the buildlog
consultant - a tool that can
identify many reasons why a Debian build failed.
For example, here’s a fragment of a build log where the Build-Depends lack
python3-setuptools:
849
850
851
852
853
854
855
856
857
858
dpkg-buildpackage: info: host architecture amd64
fakeroot debian/rules clean
dh clean --with python3,sphinxdoc --buildsystem=pybuild
dh_auto_clean -O--buildsystem=pybuild
I: pybuild base:232: python3.9 setup.py clean
Traceback (most recent call last):
File "/<>/setup.py", line 2, in
from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'
E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean
The buildlog consultant can identify the line in bold as being key, and interprets it:
% analyse-sbuild-log --json ~/build.log
{
"stage": "build",
"section": "Build",
"lineno": 857,
"kind": "missing-python-module",
"details": {"module": "setuptools", "python_version": 3, "minimum_version": null}
}
Automatically acting on buildlog problems
A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends.
Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools:
% apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py
python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py
The deb-fix-build command automates these steps:
It builds the package using sbuild; if the package successfully builds then it just exits successfully
It tries to identify the problem by looking through the build log; if it can’t or if it’s a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code
It tries to find a dependency that can address the problem
It updates Build-Depends in debian/control or Depends in debian/tests/control
Go to step 1
This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again.
For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this:
% deb-fix-build
Using output directory /tmp/tmpyz0nkgqq
Using sbuild chroot unstable-amd64-sbuild
Using fixers: …
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4')
Using apt-file to search apt contents
Adding build dependency: python3-setuptools-scm (>= 4)
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None)
Adding build dependency: python3-toml
Building debian packages, running 'sbuild --no-clean-source -A -s -v'.
Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’].
And in our Git repository, we see these changes as well:
% git log -p
commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master)
Author: Jelmer Vernooij
Date: Sun Apr 4 02:35:56 2021 +0100
Add missing build dependency on python3-toml.
diff --git a/debian/control b/debian/control
index 5b854dc..3b27b73 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1
Author: Jelmer Vernooij
Date: Sun Apr 4 02:35:34 2021 +0100
Add missing build dependency on python3-setuptools-scm (>= 4).
diff --git a/debian/control b/debian/control
index a476cc2..5b854dc 100644
--- a/debian/control
+++ b/debian/control
@@ -1,6 +1,6 @@
Rules-Requires-Root: no
Standards-Version: 4.5.1
-Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel
+Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4)
Testsuite: autopkgtest-pkg-python
Source: python-saneyaml
Priority: optional
Using deb-fix-build
You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements
for using it are that:
The package is maintained in Git
A sbuild schroot is available for use
Caveats
deb-fix-build is fairly easy to understand, and if it doesn’t work then you’re no worse off than you were without it - you’ll have to add your own Build-Depends.
That said, there are a couple of things to keep in mind:
At the moment, it doesn’t distinguish between general, Arch or Indep Build-Depends.
It can only add dependencies for things that are actually in the archive
Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn’t always get it right
[Less]
|
Posted
almost 3 years
ago
Reading build logs
Build logs for Debian packages can be quite long and difficult
for a human to read. Anybody who has looked at these logs
trying to figure out why a build failed will have spent
time scrolling through them and skimming for certain
... [More]
phrases
(lines starting with “error:” for example). In many cases,
you can spot the problem in the last 10 or 20 lines of
output – but it’s also quite common that the error is somewhere
at the beginning of many pages of error output.
The buildlog consultant
The buildlog consultant project attempts to aid in this
process by parsing sbuild and non-Debian (e.g. the output
of “make”) build logs and trying to identify the key line
that explains why a build failed. It can then either display
this specific line, or a fragment of the log around surrounding
the key line.
Classification
In addition to finding the key line explaining the failure,
it can also classify and parse the error in many cases
and return a result code and some metadata.
For example, in a failed build
of gnss-sdr that has produced 2119 lines of output,
the reason for the failure is that
log4cpp is missing – which is on line 641:
634
635
636
637
638
639
640
641
642
643
644
645
646
647
-- Required GNU Radio Component: ANALOG missing!
-- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND)
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS
LOG4CPP_LIBRARIES)
CMake Error at CMakeLists.txt:593 (message):
*** Log4cpp is required to build gnss-sdr
-- Configuring incomplete, errors occurred!
See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/
CMakeOutput.log".
See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/
CMakeError.log".
In this case, the buildlog consultant can both figure out
line was problematic and what the problem was:
% analyse-sbuild-log build.log
Failed stage: build
Section: build
Failed line: 641:
*** Log4cpp is required to build gnss-sdr
Error: Missing dependency: Log4cpp
Or, if you'd like to do something else with the output, use JSON output:
% analyse-sbuild-log --json build.log
{"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}}
How it works
The consultant does some structured parsing (most notably
it can parse the sections from a sbuild log), but otherwise
is a large set of carefully crafted regular expressions
and heuristics. It doesn’t always find the problem, but
has proven to be fairly accurate. It is constantly improved
as part of the Debian Janitor project, and
that exposes it to a wide variety of different errors.
You can see the classification and error detection in action
on the result codes
page of the Janitor.
Using the buildlog consultant
You can get the buildlog consultant from either pip or Debian
unstable (package: python3-buildlog-consultant ).
The buildlog consultant comes with two scripts – analyse-build-log and analyse-sbuild-log, for analysing
build logs and sbuild logs respectively.
[Less]
|
Posted
almost 3 years
ago
Reading build logs
Build logs for Debian packages can be quite long and difficult
for a human to read. Anybody who has looked at these logs
trying to figure out why a build failed will have spent
time scrolling through them and skimming for certain
... [More]
phrases
(lines starting with “error:” for example). In many cases,
you can spot the problem in the last 10 or 20 lines of
output – but it’s also quite common that the error is somewhere
at the beginning of many pages of error output.
The buildlog consultant
The buildlog consultant project attempts to aid in this
process by parsing sbuild and non-Debian (e.g. the output
of “make”) build logs and trying to identify the key line
that explains why a build failed. It can then either display
this specific line, or a fragment of the log around surrounding
the key line.
Classification
In addition to finding the key line explaining the failure,
it can also classify and parse the error in many cases
and return a result code and some metadata.
For example, in a failed build
of gnss-sdr that has produced 2119 lines of output,
the reason for the failure is that
log4cpp is missing – which is on line 641:
634
635
636
637
638
639
640
641
642
643
644
645
646
647
-- Required GNU Radio Component: ANALOG missing!
-- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND)
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS
LOG4CPP_LIBRARIES)
CMake Error at CMakeLists.txt:593 (message):
*** Log4cpp is required to build gnss-sdr
-- Configuring incomplete, errors occurred!
See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/
CMakeOutput.log".
See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/
CMakeError.log".
In this case, the buildlog consultant can both figure out
line was problematic and what the problem was:
% analyse-sbuild-log build.log
Failed stage: build
Section: build
Failed line: 641:
*** Log4cpp is required to build gnss-sdr
Error: Missing dependency: Log4cpp
Or, if you'd like to do something else with the output, use JSON output:
% analyse-sbuild-log --json build.log
{"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}}
How it works
The consultant does some structured parsing (most notably
it can parse the sections from a sbuild log), but otherwise
is a large set of carefully crafted regular expressions
and heuristics. It doesn’t always find the problem, but
has proven to be fairly accurate. It is constantly improved
as part of the Debian Janitor project, and
that exposes it to a wide variety of different errors.
You can see the classification and error detection in action
on the result codes
page of the Janitor.
Using the buildlog consultant
You can get the buildlog consultant from either pip or Debian
unstable (package: python3-buildlog-consultant ).
The buildlog consultant comes with two scripts – analyse-build-log and analyse-sbuild-log, for analysing
build logs and sbuild logs respectively.
[Less]
|
Posted
almost 3 years
ago
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid
... [More]
packages maintained in
Git. This post is part of a series about the progress of the Janitor.
Reading build logs
Build logs for Debian packages can be quite long and difficult
for a human to read. Anybody who has looked at these logs
trying to figure out why a build failed will have spent
time scrolling through them and skimming for certain phrases
(lines starting with “error:” for example). In many cases,
you can spot the problem in the last 10 or 20 lines of
output – but it’s also quite common that the error is somewhere
at the beginning of many pages of error output.
The buildlog consultant
The buildlog consultant project attempts to aid in this
process by parsing sbuild and non-Debian (e.g. the output
of “make”) build logs and trying to identify the key line
that explains why a build failed. It can then either display
this specific line, or a fragment of the log around surrounding
the key line.
Classification
In addition to finding the key line explaining the failure,
it can also classify and parse the error in many cases
and return a result code and some metadata.
For example, in a failed build
of gnss-sdr that has produced 2119 lines of output,
the reason for the failure is that
log4cpp is missing – which is on line 641:
634
635
636
637
638
639
640
641
642
643
644
645
646
647
-- Required GNU Radio Component: ANALOG missing!
-- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND)
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
-- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS
LOG4CPP_LIBRARIES)
CMake Error at CMakeLists.txt:593 (message):
*** Log4cpp is required to build gnss-sdr
-- Configuring incomplete, errors occurred!
See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/
CMakeOutput.log".
See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/
CMakeError.log".
In this case, the buildlog consultant can both figure out
line was problematic and what the problem was:
% analyse-sbuild-log build.log
Failed stage: build
Section: build
Failed line: 641:
*** Log4cpp is required to build gnss-sdr
Error: Missing dependency: Log4cpp
Or, if you’d like to do something else with the output, use JSON output:
% analyse-sbuild-log --json build.log
{"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}}
How it works
The consultant does some structured parsing (most notably
it can parse the sections from a sbuild log), but otherwise
is a large set of carefully crafted regular expressions
and heuristics. It doesn’t always find the problem, but
has proven to be fairly accurate. It is constantly improved
as part of the Debian Janitor project, and
that exposes it to a wide variety of different errors.
You can see the classification and error detection in action
on the result codes
page of the Janitor.
Using the buildlog consultant
You can get the buildlog consultant from either pip or Debian
unstable (package: python3-buildlog-consultant ).
The buildlog consultant comes with two scripts – analyse-build-log and analyse-sbuild-log, for analysing
build logs and sbuild logs respectively.
[Less]
|
Posted
about 3 years
ago
TL;DR: There should be an option, taproot=lockintrue, which allows users to set lockin-on-timeout to true. It should not be the default, though.
As stated in my previous post, we need actual consensus, not simply the appearance of consensus. I’m
... [More]
pretty sure we have that for taproot, but I would like a template we can use in future without endless debate each time.
Giving every group a chance to openly signal for (or against!) gives us the most robust assurance that we actually have consensus. Being able to signal opposition is vital, since everyone can lie anyway; making opposition difficult just reduces the reliability of the signal.
Developers should not activate. They’ve tried to assure themselves that there’s broad approval of the change, but that’s not really a transferable proof. We should be concerned about about future corruption, insanity, or groupthink. Moreover, even the perception that developers can set the rules will lead to attempts to influence them as Bitcoin becomes more important. As a (non-Bitcoin-core) developer I can’t think of a worse hell myself, nor do we want to attract developers who want to be influenced!
Miner activation is actually brilliant. It’s easy for everyone to count, and majority miner enforcement is sufficient to rely on the new rules. But its real genius is that miners are most directly vulnerable to the economic majority of users: in a fork they have to pick sides continuously knowing that if they are wrong, they will immediately suffer economically through missed opportunity cost.
Of course, economic users are ultimately in control. Any system which doesn’t explicitly encode that is fragile; nobody would argue that fair elections are unnecessary because if people were really dissatisfied they could always overthrow the government themselves! We should make it as easy for them to exercise this power as possible: this means not requiring them to run unvetted or home-brew modifications which will place them at more risk, so developers need to supply this option (setting it should also change the default User-Agent string, for signalling purposes). It shouldn’t be an upgrade either (which inevitably comes with other changes). Such a default-off option provides both a simple method, and a Schelling point for the lockinontimeout parameters. It also means much less chance of this power being required: “Si vis pacem, para bellum“.
This triumverate model may seem familiar, being widely used in various different governance systems. It seems the most robust to me, and is very close to what we have evolved into already. Formalizing it reduces uncertainty for any future changes, as well. [Less]
|