I Use This!
Very High Activity

News

Analyzed about 19 hours ago. based on code collected 2 days ago.
Posted almost 3 years ago
The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid ... [More] packages maintained in Git. This post is part of a series about the progress of the Janitor. The FOSS world uses a wide variety of different build tools; given a git repository or tarball, it can be hard to figure out how to build and install a piece of software. Humans will generally know what build tool a project is using when they check out a project from git, or they can read the README. And even then, the answer may not always be straightforward to everybody. For automation, there is no obvious place to figure out how to build or install a project. Debian For Debian packages, Debian maintainers generally will have determined that the appropriate tools to invoke are, and added appropriate invocations to debian/rules. This is really nice when rebuilding all of Debian - one can just invoke debian/rules - a consistent interface - and it will in turn invoke the right tools to build the package, meeting a long list of requirements. With newer versions of debhelper and most common build systems, debhelper can figure a lot of this out automatically - the maintainer just has to add the appropriate build and run time dependencies. However, debhelper needs to be consistent in its behaviour per compat level - otherwise builds might start failing with different versions of debhelper, when the autodetection logic is changed. debhelper can also only do the right thing if all the necessary dependencies are present. debhelper also only functions in the context of a Debian package. Ognibuild Ognibuild is a new tool that figures out the build system in use by an upstream project, as well as the other dependencies it needs. This information can then be used to invoke said build system, or to e.g. add missing build dependencies to a Debian package. Ognibuild uses a variety of techniques to work out what the dependencies for an upstream package are: Extracting dependencies and other requirements declared in build system metadata (e.g. setup.py) Attempting builds and parsing build logs for missing dependencies (repeating until the build succeeds), calling out to buildlog-consultant Once it is determined which dependencies are missing, they can be resolved in a variety of ways. Apt can be invoked to install missing dependencies on Debian systems (optionally in a chroot) or ecosystem-specific tools can be used to do so (e.g. pypi or cpan). Instead of installing packages, the tool can also simply inform the user about the missing packages and commands to install them, or update a Debian package appropriately (this is what deb-fix-build does). The target audience of ognibuild are people who need to (possibly from automation) build a variety of projects from different ecosystems or users who are looking to just install a project from source. Developers who are just hacking on e.g. a Python project are better off directly invoking the ecosystem-native tools rather than a wrapper like ognibuild. Supported ecosystems (Partially) supported ecosystems currently include: Combinations of make and autoconf, automake or CMake Python, including fetching packages from pypi Perl, including fetching packages from cpan Haskell, including fetching from hackage Ninja/Meson Maven Rust, including fetching packages from crates.io PHP Pear R, including fetching packages from CRAN and Bioconductor For a full list, see the README. Usage Ognibuild provides a couple of top-level subcommands that will seem familiar to anybody who has used a couple of other build systems: ogni clean - remove build artifacts ogni dist - create a dist tarball ogni build - build the project in the current directory ogni test - run the test suite ogni install - install the project somewhere ogni info - display project information including discovered build system and dependencies ogni exec - run an arbitrary command but attempt to resolve issues like missing dependencies These tools all take a couple of common options: —resolve=apt|auto|native Specifies how to resolve any missing dependencies: apt: install the appropriate dependency using apt native: install dependencies using native tools like pip or cpan auto: invoke either apt or native package install, depending on whether the current user is allowed to invoke apt —schroot=name Run inside of a schroot. —explain do not make any changes but tell the user which native on apt packages they could install. There are also subcommand-specific options, e.g. to install to a specific directory on restrict which tests are run. Examples Creating a dist tarball 1 2 3 4 5 6 7 8 9 % git clone https://github.com/dulwich/dulwich % cd dulwich % ogni --schroot=unstable-amd64-sbuild dist … Writing dulwich-0.20.21/setup.cfg creating dist Creating tar archive removing 'dulwich-0.20.21' (and everything under it) Found new tarball dulwich-0.20.21.tar.gz in /var/run/schroot/mount/unstable-amd64-sbuild-974d32d7-6f10-4e77-8622-b6a091857e85/build/tmpucazj7j7/package/dist. Installing ldb from source, resolving dependencies using apt 1 2 3 4 5 6 7 8 9 % wget https://download.samba.org/pub/ldb/ldb-2.3.0.tar.gz % tar xvfz ldb-2.3.0.tar.gz % cd ldb-2.3.0 % ogni install --prefix=/tmp/ldb … + install /tmp/ldb/include/ldb.h (from include/ldb.h) … Waf: Leaving directory `/tmp/ldb-2.3.0/bin/default' 'install' finished successfully (11.395s) Running all tests from XML::LibXML::LazyBuilder 1 2 3 4 5 6 % wget ``https://cpan.metacpan.org/authors/id/T/TO/TORU/XML-LibXML-LazyBuilder-0.08.tar.gz`_ `_ % tar xvfz XML-LibXML-LazyBuilder-0.08.tar.gz Cd XML-LibXML-LazyBuilder-0.08 % ogni test … Current Status ognibuild is still in its early stages, but works well enough that it can detect and invoke the build system for most of the upstream projects packaged in Debian. If there are buildsystems that it currently lacks support for or other issues, then I’d welcome any bug reports. [Less]
Posted almost 3 years ago
The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid ... [More] packages maintained in Git. This post is part of a series about the progress of the Janitor. The upstream ontologist is a project that extracts metadata about upstream projects in a consistent format. It does this with a combination of heuristics and reading ecosystem-specific metadata files, such as Python’s setup.py, rust’s Cargo.toml as well as e.g. scanning README files. Supported Data Sources It will extract information from a wide variety of sources, including: Python package metadata (PKG-INFO, setup.py, setup.cfg, pyproject.toml) package.json composer.json package.xml Perl package metadata (dist.ini, META.json, META.yml, Makefile.PL) Perl POD files GNU configure files R DESCRIPTION files Rust Cargo.toml maven pom.xml metainfo.xml .git/config SECURITY.md DOAP Haskell cabal files Ruby gemspec files go.mod README{,.rst,.md} files Debian packaging metadata (debian/watch, debian/control, debian/rules, debian/get-orig-source.sh, debian/copyright, debian/patches) Supported Fields Fields that it currently provides include: Homepage: homepage URL Name: name of the upstream project Contact: contact address of some sort of the upstream (e-mail, mailing list URL) Repository: VCS URL Repository-Browse: Web URL for viewing the VCS Bug-Database: Bug database URL (for web viewing, generally) Bug-Submit: URL to use to submit new bugs (either on the web or an e-mail address) Screenshots: List of URLs with screenshots Archive: Archive used - e.g. SourceForge Security-Contact: e-mail or URL with instructions for reporting security issues Documentation: Link to documentation on the web: Wiki: Wiki URL Summary: one-line description of the project Description: longer description of the project License: Single line license description (e.g. "GPL 2.0") as declared in the metadata[1] Copyright: List of copyright holders Version: Current upstream version Security-MD: URL to markdown file with security policy All data fields have a “certainty” associated with them (“certain”, “confident”, “likely” or “possible”), which gets set depending on how the data was derived or where it was found. If multiple possible values were found for a specific field, then the value with the highest certainty is taken. Interface The ontologist provides a high-level Python API as well as two command-line tools that can write output in two different formats: guess-upstream-metadata writes DEP-12-like YAML output autodoap writes DOAP files For example, running guess-upstream-metadata on dulwich: % guess-upstream-metadata :2: (INFO/1) Duplicate implicit target name: "contributing". Name: dulwich Repository: https://www.dulwich.io/code/ X-Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md X-Version: 0.20.21 Bug-Database: https://github.com/dulwich/dulwich/issues X-Summary: Python Git Library X-Description: | This is the Dulwich project. It aims to provide an interface to git repos (both local and remote) that doesn't call out to git directly but instead uses pure Python. X-License: Apache License, version 2 or GNU General Public License, version 2 or later. Bug-Submit: https://github.com/dulwich/dulwich/issues/new Lintian-Brush lintian-brush can update DEP-12-style debian/upstream/metadata files that hold information about the upstream project that is packaged as well as the Homepage in the debian/control file based on information provided by the upstream ontologist. By default, it only imports data with the highest certainty - you can override this by specifying the --uncertain command-line flag. [1] Obviously this won't be able to describe the full licensing situation for many projects. Projects like scancode-toolkit are more appropriate for that. [Less]
Posted almost 3 years ago
The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid ... [More] packages maintained in Git. This post is part of a series about the progress of the Janitor. The upstream ontologist is a project that extracts metadata about upstream projects in a consistent format. It does this with a combination of heuristics and reading ecosystem-specific metadata files, such as Python’s setup.py, rust’s Cargo.toml as well as e.g. scanning README files. Supported Data Sources It will extract information from a wide variety of sources, including: Python package metadata (PKG-INFO, setup.py, setup.cfg, pyproject.toml) package.json composer.json package.xml Perl package metadata (dist.ini, META.json, META.yml, Makefile.PL) Perl POD files GNU configure files R DESCRIPTION files Rust Cargo.toml maven pom.xml metainfo.xml .git/config SECURITY.md DOAP Haskell cabal files Ruby gemspec files go.mod README{,.rst,.md} files Debian packaging metadata (debian/watch, debian/control, debian/rules, debian/get-orig-source.sh, debian/copyright, debian/patches) Supported Fields Fields that it currently provides include: Homepage: homepage URL Name: name of the upstream project Contact: contact address of some sort of the upstream (e-mail, mailing list URL) Repository: VCS URL Repository-Browse: Web URL for viewing the VCS Bug-Database: Bug database URL (for web viewing, generally) Bug-Submit: URL to use to submit new bugs (either on the web or an e-mail address) Screenshots: List of URLs with screenshots Archive: Archive used - e.g. SourceForge Security-Contact: e-mail or URL with instructions for reporting security issues Documentation: Link to documentation on the web: Wiki: Wiki URL Summary: one-line description of the project Description: longer description of the project License: Single line license description (e.g. “GPL 2.0”) as declared in the metadata[1] Copyright: List of copyright holders Version: Current upstream version Security-MD: URL to markdown file with security policy All data fields have a “certainty” associated with them (“certain”, “confident”, “likely” or “possible”), which gets set depending on how the data was derived or where it was found. If multiple possible values were found for a specific field, then the value with the highest certainty is taken. Interface The ontologist provides a high-level Python API as well as two command-line tools that can write output in two different formats: guess-upstream-metadata writes DEP-12-like YAML output autodoap writes DOAP files For example, running guess-upstream-metadata on dulwich: % guess-upstream-metadata :2: (INFO/1) Duplicate implicit target name: "contributing". Name: dulwich Repository: https://www.dulwich.io/code/ X-Security-MD: https://github.com/dulwich/dulwich/tree/HEAD/SECURITY.md X-Version: 0.20.21 Bug-Database: https://github.com/dulwich/dulwich/issues X-Summary: Python Git Library X-Description: | This is the Dulwich project. It aims to provide an interface to git repos (both local and remote) that doesn't call out to git directly but instead uses pure Python. X-License: Apache License, version 2 or GNU General Public License, version 2 or later. Bug-Submit: https://github.com/dulwich/dulwich/issues/new Lintian-Brush lintian-brush can update DEP-12-style debian/upstream/metadata files that hold information about the upstream project that is packaged as well as the Homepage in the debian/control file based on information provided by the upstream ontologist. By default, it only imports data with the highest certainty - you can override this by specifying the —uncertain command-line flag. [1] Obviously this won’t be able to describe the full licensing situation for many projects. Projects like scancode-toolkit are more appropriate for that. [Less]
Posted almost 3 years ago
The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid ... [More] packages maintained in Git. This post is part of a series about the progress of the Janitor. In my last blogpost, I introduced the buildlog consultant - a tool that can identify many reasons why a Debian build failed. For example, here’s a fragment of a build log where the Build-Depends lack python3-setuptools: 849 850 851 852 853 854 855 856 857 858 dpkg-buildpackage: info: host architecture amd64 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --buildsystem=pybuild dh_auto_clean -O--buildsystem=pybuild I: pybuild base:232: python3.9 setup.py clean Traceback (most recent call last): File "/<>/setup.py", line 2, in from setuptools import setup ModuleNotFoundError: No module named 'setuptools' E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean The buildlog consultant can identify the line in bold as being key, and interprets it: % analyse-sbuild-log --json ~/build.log { "stage": "build", "section": "Build", "lineno": 857, "kind": "missing-python-module", "details": {"module": "setuptools", "python_version": 3, "minimum_version": null} } Automatically acting on buildlog problems A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends. Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools: % apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py The deb-fix-build command automates these steps: It builds the package using sbuild; if the package successfully builds then it just exits successfully It tries to identify the problem by looking through the build log; if it can't or if it's a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code It tries to find a dependency that can address the problem It updates Build-Depends in debian/control or Depends in debian/tests/control Go to step 1 This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again. For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this: % deb-fix-build Using output directory /tmp/tmpyz0nkgqq Using sbuild chroot unstable-amd64-sbuild Using fixers: … Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4') Using apt-file to search apt contents Adding build dependency: python3-setuptools-scm (>= 4) Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None) Adding build dependency: python3-toml Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’]. And in our Git repository, we see these changes as well: % git log -p commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master) Author: Jelmer Vernooij Date: Sun Apr 4 02:35:56 2021 +0100 Add missing build dependency on python3-toml. diff --git a/debian/control b/debian/control index 5b854dc..3b27b73 100644 --- a/debian/control +++ b/debian/control @@ -1,6 +1,6 @@ Rules-Requires-Root: no Standards-Version: 4.5.1 -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4) +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml Testsuite: autopkgtest-pkg-python Source: python-saneyaml Priority: optional commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1 Author: Jelmer Vernooij Date: Sun Apr 4 02:35:34 2021 +0100 Add missing build dependency on python3-setuptools-scm (>= 4). diff --git a/debian/control b/debian/control index a476cc2..5b854dc 100644 --- a/debian/control +++ b/debian/control @@ -1,6 +1,6 @@ Rules-Requires-Root: no Standards-Version: 4.5.1 -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4) Testsuite: autopkgtest-pkg-python Source: python-saneyaml Priority: optional Using deb-fix-build You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements for using it are that: The package is maintained in Git A sbuild schroot is available for use Caveats deb-fix-build is fairly easy to understand, and if it doesn't work then you're no worse off than you were without it - you'll have to add your own Build-Depends. That said, there are a couple of things to keep in mind: At the moment, it doesn't distinguish between general, Arch or Indep Build-Depends. It can only add dependencies for things that are actually in the archive Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn't always get it right [Less]
Posted almost 3 years ago
In my last blogpost, I introduced the buildlog consultant - a tool that can identify many reasons why a Debian build failed. For example, here’s a fragment of a build log where the Build-Depends lack python3-setuptools: 849 850 851 852 853 854 855 ... [More] 856 857 858 dpkg-buildpackage: info: host architecture amd64 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --buildsystem=pybuild dh_auto_clean -O--buildsystem=pybuild I: pybuild base:232: python3.9 setup.py clean Traceback (most recent call last): File "/<>/setup.py", line 2, in from setuptools import setup ModuleNotFoundError: No module named 'setuptools' E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean The buildlog consultant can identify the line in bold as being key, and interprets it: % analyse-sbuild-log --json ~/build.log {"stage": "build", "section": "Build", "lineno": 857, "kind": "missing-python-module", "details": {"module": "setuptools", "python_version": 3, "minimum_version": null}} Automatically acting on buildlog problems A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends. Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools: % apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py The deb-fix-build command automates these steps: It builds the package using sbuild; if the package successfully builds then it just exits successfully It tries to identify the problem by looking through the build log; if it can't or if it's a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code It tries to find a dependency that can address the problem It updates Build-Depends in debian/control or Depends in debian/tests/control Go to step 1 This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again. For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this: % deb-fix-build Using output directory /tmp/tmpyz0nkgqq Using sbuild chroot unstable-amd64-sbuild Using fixers: … Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4') Using apt-file to search apt contents Adding build dependency: python3-setuptools-scm (>= 4) Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None) Adding build dependency: python3-toml Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’]. And in our Git repository, we see these changes as well: % git log -p commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master) Author: Jelmer Vernooij Date: Sun Apr 4 02:35:56 2021 +0100 Add missing build dependency on python3-toml. diff --git a/debian/control b/debian/control index 5b854dc..3b27b73 100644 --- a/debian/control +++ b/debian/control @@ -1,6 +1,6 @@ Rules-Requires-Root: no Standards-Version: 4.5.1 -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4) +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml Testsuite: autopkgtest-pkg-python Source: python-saneyaml Priority: optional commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1 Author: Jelmer Vernooij Date: Sun Apr 4 02:35:34 2021 +0100 Add missing build dependency on python3-setuptools-scm (>= 4). diff --git a/debian/control b/debian/control index a476cc2..5b854dc 100644 --- a/debian/control +++ b/debian/control @@ -1,6 +1,6 @@ Rules-Requires-Root: no Standards-Version: 4.5.1 -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4) Testsuite: autopkgtest-pkg-python Source: python-saneyaml Priority: optional Using deb-fix-build You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements for using it are that: The package is maintained in Git A sbuild schroot is available for use Caveats deb-fix-build is fairly easy to understand, and if it doesn't work then you're no worse off than you were without it - you'll have to add your own Build-Depends. That said, there are a couple of things to keep in mind: At the moment, it doesn't distinguish between general, Arch or Indep Build-Depends. It can only add dependencies for things that are actually in the archive Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn't always get it right [Less]
Posted almost 3 years ago
The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid ... [More] packages maintained in Git. This post is part of a series about the progress of the Janitor. In my last blogpost, I introduced the buildlog consultant - a tool that can identify many reasons why a Debian build failed. For example, here’s a fragment of a build log where the Build-Depends lack python3-setuptools: 849 850 851 852 853 854 855 856 857 858 dpkg-buildpackage: info: host architecture amd64 fakeroot debian/rules clean dh clean --with python3,sphinxdoc --buildsystem=pybuild dh_auto_clean -O--buildsystem=pybuild I: pybuild base:232: python3.9 setup.py clean Traceback (most recent call last): File "/<>/setup.py", line 2, in from setuptools import setup ModuleNotFoundError: No module named 'setuptools' E: pybuild pybuild:353: clean: plugin distutils failed with: exit code=1: python3.9 setup.py clean The buildlog consultant can identify the line in bold as being key, and interprets it: % analyse-sbuild-log --json ~/build.log { "stage": "build", "section": "Build", "lineno": 857, "kind": "missing-python-module", "details": {"module": "setuptools", "python_version": 3, "minimum_version": null} } Automatically acting on buildlog problems A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends. Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools: % apt-file search /usr/lib/python3/dist-packages/setuptools/__init__.py python3-setuptools: /usr/lib/python3/dist-packages/setuptools/__init__.py The deb-fix-build command automates these steps: It builds the package using sbuild; if the package successfully builds then it just exits successfully It tries to identify the problem by looking through the build log; if it can’t or if it’s a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code It tries to find a dependency that can address the problem It updates Build-Depends in debian/control or Depends in debian/tests/control Go to step 1 This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again. For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this: % deb-fix-build Using output directory /tmp/tmpyz0nkgqq Using sbuild chroot unstable-amd64-sbuild Using fixers: … Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4') Using apt-file to search apt contents Adding build dependency: python3-setuptools-scm (>= 4) Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None) Adding build dependency: python3-toml Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Built 0.5.2-1- changes files at [‘saneyaml_0.5.2-1_amd64.changes’]. And in our Git repository, we see these changes as well: % git log -p commit 5a1715f4c7273b042818fc75702f2284034c7277 (HEAD -> master) Author: Jelmer Vernooij Date: Sun Apr 4 02:35:56 2021 +0100 Add missing build dependency on python3-toml. diff --git a/debian/control b/debian/control index 5b854dc..3b27b73 100644 --- a/debian/control +++ b/debian/control @@ -1,6 +1,6 @@ Rules-Requires-Root: no Standards-Version: 4.5.1 -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4) +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4), python3-toml Testsuite: autopkgtest-pkg-python Source: python-saneyaml Priority: optional commit f03047da80fcd8468ee231fbc4cf8488d7a0acd1 Author: Jelmer Vernooij Date: Sun Apr 4 02:35:34 2021 +0100 Add missing build dependency on python3-setuptools-scm (>= 4). diff --git a/debian/control b/debian/control index a476cc2..5b854dc 100644 --- a/debian/control +++ b/debian/control @@ -1,6 +1,6 @@ Rules-Requires-Root: no Standards-Version: 4.5.1 -Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel +Build-Depends: debhelper-compat (= 12), dh-sequence-python3, python3-all, python3-setuptools (>= 50), python3-wheel, python3-setuptools-scm (>= 4) Testsuite: autopkgtest-pkg-python Source: python-saneyaml Priority: optional Using deb-fix-build You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements for using it are that: The package is maintained in Git A sbuild schroot is available for use Caveats deb-fix-build is fairly easy to understand, and if it doesn’t work then you’re no worse off than you were without it - you’ll have to add your own Build-Depends. That said, there are a couple of things to keep in mind: At the moment, it doesn’t distinguish between general, Arch or Indep Build-Depends. It can only add dependencies for things that are actually in the archive Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn’t always get it right [Less]
Posted almost 3 years ago
Reading build logs Build logs for Debian packages can be quite long and difficult for a human to read. Anybody who has looked at these logs trying to figure out why a build failed will have spent time scrolling through them and skimming for certain ... [More] phrases (lines starting with “error:” for example). In many cases, you can spot the problem in the last 10 or 20 lines of output – but it’s also quite common that the error is somewhere at the beginning of many pages of error output. The buildlog consultant The buildlog consultant project attempts to aid in this process by parsing sbuild and non-Debian (e.g. the output of “make”) build logs and trying to identify the key line that explains why a build failed. It can then either display this specific line, or a fragment of the log around surrounding the key line. Classification In addition to finding the key line explaining the failure, it can also classify and parse the error in many cases and return a result code and some metadata. For example, in a failed build of gnss-sdr that has produced 2119 lines of output, the reason for the failure is that log4cpp is missing – which is on line 641: 634 635 636 637 638 639 640 641 642 643 644 645 646 647 -- Required GNU Radio Component: ANALOG missing! -- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND) -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) -- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS LOG4CPP_LIBRARIES) CMake Error at CMakeLists.txt:593 (message): *** Log4cpp is required to build gnss-sdr -- Configuring incomplete, errors occurred! See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeOutput.log". See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeError.log". In this case, the buildlog consultant can both figure out line was problematic and what the problem was: % analyse-sbuild-log build.log Failed stage: build Section: build Failed line: 641: *** Log4cpp is required to build gnss-sdr Error: Missing dependency: Log4cpp Or, if you'd like to do something else with the output, use JSON output: % analyse-sbuild-log --json build.log {"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}} How it works The consultant does some structured parsing (most notably it can parse the sections from a sbuild log), but otherwise is a large set of carefully crafted regular expressions and heuristics. It doesn’t always find the problem, but has proven to be fairly accurate. It is constantly improved as part of the Debian Janitor project, and that exposes it to a wide variety of different errors. You can see the classification and error detection in action on the result codes page of the Janitor. Using the buildlog consultant You can get the buildlog consultant from either pip or Debian unstable (package: python3-buildlog-consultant ). The buildlog consultant comes with two scripts – analyse-build-log and analyse-sbuild-log, for analysing build logs and sbuild logs respectively. [Less]
Posted almost 3 years ago
Reading build logs Build logs for Debian packages can be quite long and difficult for a human to read. Anybody who has looked at these logs trying to figure out why a build failed will have spent time scrolling through them and skimming for certain ... [More] phrases (lines starting with “error:” for example). In many cases, you can spot the problem in the last 10 or 20 lines of output – but it’s also quite common that the error is somewhere at the beginning of many pages of error output. The buildlog consultant The buildlog consultant project attempts to aid in this process by parsing sbuild and non-Debian (e.g. the output of “make”) build logs and trying to identify the key line that explains why a build failed. It can then either display this specific line, or a fragment of the log around surrounding the key line. Classification In addition to finding the key line explaining the failure, it can also classify and parse the error in many cases and return a result code and some metadata. For example, in a failed build of gnss-sdr that has produced 2119 lines of output, the reason for the failure is that log4cpp is missing – which is on line 641: 634 635 636 637 638 639 640 641 642 643 644 645 646 647 -- Required GNU Radio Component: ANALOG missing! -- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND) -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) -- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS LOG4CPP_LIBRARIES) CMake Error at CMakeLists.txt:593 (message): *** Log4cpp is required to build gnss-sdr -- Configuring incomplete, errors occurred! See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeOutput.log". See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeError.log". In this case, the buildlog consultant can both figure out line was problematic and what the problem was: % analyse-sbuild-log build.log Failed stage: build Section: build Failed line: 641: *** Log4cpp is required to build gnss-sdr Error: Missing dependency: Log4cpp Or, if you'd like to do something else with the output, use JSON output: % analyse-sbuild-log --json build.log {"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}} How it works The consultant does some structured parsing (most notably it can parse the sections from a sbuild log), but otherwise is a large set of carefully crafted regular expressions and heuristics. It doesn’t always find the problem, but has proven to be fairly accurate. It is constantly improved as part of the Debian Janitor project, and that exposes it to a wide variety of different errors. You can see the classification and error detection in action on the result codes page of the Janitor. Using the buildlog consultant You can get the buildlog consultant from either pip or Debian unstable (package: python3-buildlog-consultant ). The buildlog consultant comes with two scripts – analyse-build-log and analyse-sbuild-log, for analysing build logs and sbuild logs respectively. [Less]
Posted almost 3 years ago
The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid ... [More] packages maintained in Git. This post is part of a series about the progress of the Janitor. Reading build logs Build logs for Debian packages can be quite long and difficult for a human to read. Anybody who has looked at these logs trying to figure out why a build failed will have spent time scrolling through them and skimming for certain phrases (lines starting with “error:” for example). In many cases, you can spot the problem in the last 10 or 20 lines of output – but it’s also quite common that the error is somewhere at the beginning of many pages of error output. The buildlog consultant The buildlog consultant project attempts to aid in this process by parsing sbuild and non-Debian (e.g. the output of “make”) build logs and trying to identify the key line that explains why a build failed. It can then either display this specific line, or a fragment of the log around surrounding the key line. Classification In addition to finding the key line explaining the failure, it can also classify and parse the error in many cases and return a result code and some metadata. For example, in a failed build of gnss-sdr that has produced 2119 lines of output, the reason for the failure is that log4cpp is missing – which is on line 641: 634 635 636 637 638 639 640 641 642 643 644 645 646 647 -- Required GNU Radio Component: ANALOG missing! -- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND) -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) -- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS LOG4CPP_LIBRARIES) CMake Error at CMakeLists.txt:593 (message): *** Log4cpp is required to build gnss-sdr -- Configuring incomplete, errors occurred! See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeOutput.log". See also "/<>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeError.log". In this case, the buildlog consultant can both figure out line was problematic and what the problem was: % analyse-sbuild-log build.log Failed stage: build Section: build Failed line: 641: *** Log4cpp is required to build gnss-sdr Error: Missing dependency: Log4cpp Or, if you’d like to do something else with the output, use JSON output: % analyse-sbuild-log --json build.log {"stage": "build", "section": "Build", "lineno": 641, "kind": "missing-dependency", "details": {"name": "Log4cpp""}} How it works The consultant does some structured parsing (most notably it can parse the sections from a sbuild log), but otherwise is a large set of carefully crafted regular expressions and heuristics. It doesn’t always find the problem, but has proven to be fairly accurate. It is constantly improved as part of the Debian Janitor project, and that exposes it to a wide variety of different errors. You can see the classification and error detection in action on the result codes page of the Janitor. Using the buildlog consultant You can get the buildlog consultant from either pip or Debian unstable (package: python3-buildlog-consultant ). The buildlog consultant comes with two scripts – analyse-build-log and analyse-sbuild-log, for analysing build logs and sbuild logs respectively. [Less]
Posted about 3 years ago
TL;DR: There should be an option, taproot=lockintrue, which allows users to set lockin-on-timeout to true. It should not be the default, though. As stated in my previous post, we need actual consensus, not simply the appearance of consensus. I’m ... [More] pretty sure we have that for taproot, but I would like a template we can use in future without endless debate each time. Giving every group a chance to openly signal for (or against!) gives us the most robust assurance that we actually have consensus. Being able to signal opposition is vital, since everyone can lie anyway; making opposition difficult just reduces the reliability of the signal. Developers should not activate. They’ve tried to assure themselves that there’s broad approval of the change, but that’s not really a transferable proof. We should be concerned about about future corruption, insanity, or groupthink. Moreover, even the perception that developers can set the rules will lead to attempts to influence them as Bitcoin becomes more important. As a (non-Bitcoin-core) developer I can’t think of a worse hell myself, nor do we want to attract developers who want to be influenced! Miner activation is actually brilliant. It’s easy for everyone to count, and majority miner enforcement is sufficient to rely on the new rules. But its real genius is that miners are most directly vulnerable to the economic majority of users: in a fork they have to pick sides continuously knowing that if they are wrong, they will immediately suffer economically through missed opportunity cost. Of course, economic users are ultimately in control. Any system which doesn’t explicitly encode that is fragile; nobody would argue that fair elections are unnecessary because if people were really dissatisfied they could always overthrow the government themselves! We should make it as easy for them to exercise this power as possible: this means not requiring them to run unvetted or home-brew modifications which will place them at more risk, so developers need to supply this option (setting it should also change the default User-Agent string, for signalling purposes). It shouldn’t be an upgrade either (which inevitably comes with other changes). Such a default-off option provides both a simple method, and a Schelling point for the lockinontimeout parameters. It also means much less chance of this power being required: “Si vis pacem, para bellum“. This triumverate model may seem familiar, being widely used in various different governance systems. It seems the most robust to me, and is very close to what we have evolved into already. Formalizing it reduces uncertainty for any future changes, as well. [Less]