Posted
over 1 year
ago
by
Alberto Mardegan
While developing MiTubo I've recently felt
the need of parsing HTML pages: the first problem I wanted to solve was
implementing proper RSS feed detection when the user entered a website URL into
MiTubo's search box, so that MiTubo would parse the
... [More]
site's HTML, look for
URLs in the HEAD section, and let the user
subscribe to any video feeds found there.
A quick search in the internet did not provide a clear answer: I found a Qt
HTML parser in (stalled) development, and
a few other C++ or C parsers (among the latters,
lexbor
is the most inspiring), but all of them seem to take the approach of parsing
the HTML file into a DOM tree, while I was hoping to find a lightweight
SAX-like parser. Pretty much like Python's
html.parser.
Anyway, I don't remember how it happened, but at a certain point I found myself
looking at html.parser source code, and I was surprised to see how compact it
was (apart, of course, for the long list of character references for the HTML
entities!). Upon a closer look, it also appeared that the code was not making
much use of Python's dynamic typing, so, I thought, maybe I could give it a try
to rewrite that into a Qt class. And a few hours later
QHtmlParser was born.
As this post's title suggests, the process of rewriting html.parser with Qt
was quite straightforward, and the nice thing about it is that I didn't have to
spend any time reading the HTML standard or trying to figure out how to
implement the parser: I just had to translate Python code into C++ code, and
thanks to the nice API of QString (which in many ways resembles Python's — or
vice versa) this was not too hard. I even left most of the original code
comments untouched, and reused quite a few tests from the test suite.
It was time well spent. :-)
If you think you might need an HTML parser for your Qt application, you are
welcome to give it a try. It's not a library, just a set of files that you can
import into your project; for the time being I only have a build file for
QBS, but I'll happily accept
contributions to make it easier to use QHtmlParser with projects built using
other build systems. You can see
here
the changes I made in MiTubo to start using it and detect RSS feed in a
webpage's HEAD.
That's all for now. And in case you missed the link before, you can find
QHtmlParser here. [Less]
|
Posted
over 1 year
ago
by
Alberto Mardegan
While developing MiTubo I've recently felt
the need of parsing HTML pages: the first problem I wanted to solve was
implementing proper RSS feed detection when the user entered a website URL into
MiTubo's search box, so that MiTubo would parse the
... [More]
site's HTML, look for
URLs in the HEAD section, and let the user
subscribe to any video feeds found there.
A quick search in the internet did not provide a clear answer: I found a Qt
HTML parser in (stalled) development, and
a few other C++ or C parsers (among the latters,
lexbor
is the most inspiring), but all of them seem to take the approach of parsing
the HTML file into a DOM tree, while I was hoping to find a lightweight
SAX-like parser. Pretty much like Python's
html.parser.
Anyway, I don't remember how it happened, but at a certain point I found myself
looking at html.parser source code, and I was surprised to see how compact it
was (apart, of course, for the long list of character references for the HTML
entities!). Upon a closer look, it also appeared that the code was not making
much use of Python's dynamic typing, so, I thought, maybe I could give it a try
to rewrite that into a Qt class. And a few hours later
QHtmlParser was born.
As this post's title suggests, the process of rewriting html.parser with Qt
was quite straightforward, and the nice thing about it is that I didn't have to
spend any time reading the HTML standard or trying to figure out how to
implement the parser: I just had to translate Python code into C++ code, and
thanks to the nice API of QString (which in many ways resembles Python's — or
vice versa) this was not too hard. I even left most of the original code
comments untouched, and reused quite a few tests from the test suite.
It was time well spent. :-)
If you think you might need an HTML parser for your Qt application, you are
welcome to give it a try. It's not a library, just a set of files that you can
import into your project; for the time being I only have a build file for
QBS, but I'll happily accept
contributions to make it easier to use QHtmlParser with projects built using
other build systems. You can see
here
the changes I made in MiTubo to start using it and detect RSS feed in a
webpage's HEAD.
That's all for now. And in case you missed the link before, you can find
QHtmlParser here.0 0 [Less]
|
Posted
over 1 year
ago
by
Alberto Mardegan
While developing MiTubo I've recently felt
the need of parsing HTML pages: the first problem I wanted to solve was
implementing proper RSS feed detection when the user entered a website URL into
MiTubo's search box, so that MiTubo would parse the
... [More]
site's HTML, look for
URLs in the HEAD section, and let the user
subscribe to any video feeds found there.
A quick search in the internet did not provide a clear answer: I found a Qt
HTML parser in (stalled) development, and
a few other C++ or C parsers (among the latters,
lexbor
is the most inspiring), but all of them seem to take the approach of parsing
the HTML file into a DOM tree, while I was hoping to find a lightweight
SAX-like parser. Pretty much like Python's
html.parser.
Anyway, I don't remember how it happened, but at a certain point I found myself
looking at html.parser source code, and I was surprised to see how compact it
was (apart, of course, for the long list of character references for the HTML
entities!). Upon a closer look, it also appeared that the code was not making
much use of Python's dynamic typing, so, I thought, maybe I could give it a try
to rewrite that into a Qt class. And a few hours later
QHtmlParser was born.
As this post's title suggests, the process of rewriting html.parser with Qt
was quite straightforward, and the nice thing about it is that I didn't have to
spend any time reading the HTML standard or trying to figure out how to
implement the parser: I just had to translate Python code into C++ code, and
thanks to the nice API of QString (which in many ways resembles Python's — or
vice versa) this was not too hard. I even left most of the original code
comments untouched, and reused quite a few tests from the test suite.
It was time well spent. :-)
If you think you might need an HTML parser for your Qt application, you are
welcome to give it a try. It's not a library, just a set of files that you can
import into your project; for the time being I only have a build file for
QBS, but I'll happily accept
contributions to make it easier to use QHtmlParser with projects built using
other build systems. You can see
here
the changes I made in MiTubo to start using it and detect RSS feed in a
webpage's HEAD.
That's all for now. And in case you missed the link before, you can find
QHtmlParser here.
0 0 [Less]
|
Posted
over 1 year
ago
by
Enrique Ocaña González
I was a happy user of Del.icio.us for many years until the service closed. Then I moved my links to Google Bookmarks, which offered basically the same functionality (at least for my needs): link storage with title, tags and comments. I’ve carefully
... [More]
tagged and filed more than 2500 links since I started, and I’ve learnt to appreciate the usefulness of searching by tag to find again some precious information that was valuable to me in the past.
Google Bookmarks is a very old and simple service that “just works”. Sometimes it looked as if Google had just forgotten about it and let it run for years without anybody noticing… until now. It’s closing on September 2021.
I didn’t want to lose all my links, still need a link database searchable by tags and don’t want to be locked-in again in a similar service that might close in some years, so I wrote my own super-simple alternative to it. It’s called bs, sort of bookmark search.
The usage can’t be simpler, just add the tag you want to look for and it will print a list of links that have that tag:
$ bs webassembly
title = Canvas filled three ways: JS, WebAssembly and WebGL | Compile
url = https://compile.fi/canvas-filled-three-ways-js-webassembly-and-webgl/
tags = canvas,graphics,html5,wasm,webassembly,webgl
date = 2020-02-18 16:48:56
comment =
title = Compiling to WebAssembly: It’s Happening! ★ Mozilla Hacks – the Web developer blog
url = https://hacks.mozilla.org/2015/12/compiling-to-webassembly-its-happening/
tags = asm.js,asmjs,emscripten,llvm,toolchain,web,webassembly
date = 2015-12-18 09:14:35
comment =
If you call the tools without parameters, it will prompt data to insert a new link or edit it if the entered url matches a preexisting one:
$ bs
url: https://compile.fi/canvas-filled-three-ways-js-webassembly-and-webgl/
title: Canvas filled three ways: JS, WebAssembly and WebGL | Compile
tags: canvas,graphics,html5,wasm,webassembly,webgl
comment:
The data is stored in an sqlite database and I’ve written some JavaScript snippets to import the Delicious exported bookmarks file and the Google Bookmarks exported bookmarks file. Those snippets are meant to be copypasted in the JavaScript console of your browser while you have the exported bookmarks html file open on it. They’ll generate SQL sentences that will populate the database for the first time with your preexisting data.
By now the tool doesn’t allow to delete bookmarks (I haven’t had the need yet) and I still need to find a way to simplify its usage through the browser with a bookmarklet to ease adding new bookmarks automatically. But that’s a task for other day. By now I have enough just by knowing that my bookmarks are now safe.
Enjoy!
0 0 [Less]
|
Posted
over 1 year
ago
by
Enrique Ocaña González
I was a happy user of Del.icio.us for many years until the service closed. Then I moved my links to Google Bookmarks, which offered basically the same functionality (at least for my needs): link storage with title, tags and comments. I’ve carefully
... [More]
tagged and filed more than 2500 links since I started, and I’ve learnt to appreciate the usefulness of searching by tag to find again some precious information that was valuable to me in the past.
Google Bookmarks is a very old and simple service that “just works”. Sometimes it looked as if Google had just forgotten about it and let it run for years without anybody noticing… until now. It’s closing on September 2021.
I didn’t want to lose all my links, still need a link database searchable by tags and don’t want to be locked-in again in a similar service that might close in some years, so I wrote my own super-simple alternative to it. It’s called bs, sort of bookmark search.
The usage can’t be simpler, just add the tag you want to look for and it will print a list of links that have that tag:
$ bs webassembly
title = Canvas filled three ways: JS, WebAssembly and WebGL | Compile
url = https://compile.fi/canvas-filled-three-ways-js-webassembly-and-webgl/
tags = canvas,graphics,html5,wasm,webassembly,webgl
date = 2020-02-18 16:48:56
comment =
title = Compiling to WebAssembly: It’s Happening! ★ Mozilla Hacks – the Web developer blog
url = https://hacks.mozilla.org/2015/12/compiling-to-webassembly-its-happening/
tags = asm.js,asmjs,emscripten,llvm,toolchain,web,webassembly
date = 2015-12-18 09:14:35
comment =
If you call the tools without parameters, it will prompt data to insert a new link or edit it if the entered url matches a preexisting one:
$ bs
url: https://compile.fi/canvas-filled-three-ways-js-webassembly-and-webgl/
title: Canvas filled three ways: JS, WebAssembly and WebGL | Compile
tags: canvas,graphics,html5,wasm,webassembly,webgl
comment:
The data is stored in an sqlite database and I’ve written some JavaScript snippets to import the Delicious exported bookmarks file and the Google Bookmarks exported bookmarks file. Those snippets are meant to be copypasted in the JavaScript console of your browser while you have the exported bookmarks html file open on it. They’ll generate SQL sentences that will populate the database for the first time with your preexisting data.
By now the tool doesn’t allow to delete bookmarks (I haven’t had the need yet) and I still need to find a way to simplify its usage through the browser with a bookmarklet to ease adding new bookmarks automatically. But that’s a task for other day. By now I have enough just by knowing that my bookmarks are now safe.
Enjoy!
[UPDATE: 2020-09-08]
I’ve now coded an alternate variant of the database client that can be hosted on any web server with PHP and SQLite3. The bookmarks can now be managed from a browser in a centralized way, in a similar fashion as you could before with Google Bookmarks and Delicious. As you can see in the screenshot, the style resembles Google Bookmarks in some way.
You can easily create a quick search / search engine link in Firefox and Chrome (I use “d” as keyword, a tradition from the Delicious days, so that if I type “d debug” in the browser search bar it will look for that tag in the bookmark search page). Also, the 0 0 [Less]
|
Posted
over 1 year
ago
by
Enrique Ocaña González
I was a happy user of Del.icio.us for many years until the service closed. Then I moved my links to Google Bookmarks, which offered basically the same functionality (at least for my needs): link storage with title, tags and comments. I’ve carefully
... [More]
tagged and filed more than 2500 links since I started, and I’ve learnt to appreciate the usefulness of searching by tag to find again some precious information that was valuable to me in the past.
Google Bookmarks is a very old and simple service that “just works”. Sometimes it looked as if Google had just forgotten about it and let it run for years without anybody noticing… until now. It’s closing on September 2021.
I didn’t want to lose all my links, still need a link database searchable by tags and don’t want to be locked-in again in a similar service that might close in some years, so I wrote my own super-simple alternative to it. It’s called bs, sort of bookmark search.
The usage can’t be simpler, just add the tag you want to look for and it will print a list of links that have that tag:
$ bs webassembly
title = Canvas filled three ways: JS, WebAssembly and WebGL | Compile
url = https://compile.fi/canvas-filled-three-ways-js-webassembly-and-webgl/
tags = canvas,graphics,html5,wasm,webassembly,webgl
date = 2020-02-18 16:48:56
comment =
title = Compiling to WebAssembly: It’s Happening! ★ Mozilla Hacks – the Web developer blog
url = https://hacks.mozilla.org/2015/12/compiling-to-webassembly-its-happening/
tags = asm.js,asmjs,emscripten,llvm,toolchain,web,webassembly
date = 2015-12-18 09:14:35
comment =
If you call the tools without parameters, it will prompt data to insert a new link or edit it if the entered url matches a preexisting one:
$ bs
url: https://compile.fi/canvas-filled-three-ways-js-webassembly-and-webgl/
title: Canvas filled three ways: JS, WebAssembly and WebGL | Compile
tags: canvas,graphics,html5,wasm,webassembly,webgl
comment:
The data is stored in an sqlite database and I’ve written some JavaScript snippets to import the Delicious exported bookmarks file and the Google Bookmarks exported bookmarks file. Those snippets are meant to be copypasted in the JavaScript console of your browser while you have the exported bookmarks html file open on it. They’ll generate SQL sentences that will populate the database for the first time with your preexisting data.
By now the tool doesn’t allow to delete bookmarks (I haven’t had the need yet) and I still need to find a way to simplify its usage through the browser with a bookmarklet to ease adding new bookmarks automatically. But that’s a task for other day. By now I have enough just by knowing that my bookmarks are now safe.
Enjoy!
[UPDATE: 2020-09-08]
I’ve now coded an alternate variant of the database client that can be hosted on any web server with PHP and SQLite3. The bookmarks can now be managed from a browser in a centralized way, in a similar fashion as you could before with Google Bookmarks and Delicious. As you can see in the screenshot, the style resembles Google Bookmarks in some way.
You can easily create a quick search / search engine link in Firefox and Chrome (I use “d” as keyword, a tradition from the Delicious days, so that if I type “d debug” in the browser search bar it will look for that tag in the bookmark search page). Also, the 0 0 [Less]
|
Posted
over 1 year
ago
by
Thomas Perl
As part of a summer clean-up of the desk drawers, I pulled out the N800 and N9 and ported my game Loonies 8192 to these devices. Since those are "proper" Linux devices, one can compile things directly on-device (just install gcc from the SDK repos)
... [More]
, and with SSH, it's easy to type on a real keyboard.Anyway, you can install the game via the landing pages:For the N800, make sure "maemo Extras" is enabled so it will find libsdl1.2 if it's not already installed. Head over to https://loonies.thp.io/n800/ on the device and download the deb, it will be installed by Application manager.For the N9, make sure you have n9repomirror installed (again, so libsdl1.2 can be installed if necessary). Enable third party applications in Settings, Applications, Installations. Then head over to https://loonies.thp.io/n9/ on the device and download the deb, selecting after the download is finished will ask you to install it.The N9 version is also available on openrepos.net.And don't forget that the game is also available for DOS, various consoles and handheld consoles as well as on Windows. All of the builds are available on itch.io.0 0 [Less]
|
Posted
almost 2 years
ago
by
Enrique Ocaña González
This is the last post of the series showing interesting debugging tools, I hope you have found it useful. Don’t miss the custom scripts at the bottom to process GStreamer logs, help you highlight the interesting parts and find the root cause of
... [More]
difficult bugs. Here are also the previous posts of the series:
GStreamer WebKit debugging tricks using GDB (1/2)
GStreamer WebKit debugging tricks using GDB (2/2)
GStreamer WebKit debugging by instrumenting source code (1/3)
GStreamer WebKit debugging by instrumenting source code (2/3)
GStreamer WebKit debugging by instrumenting source code (3/3)
GStreamer WebKit debugging by using external tools (1/2)
How to debug pkgconfig
When pkg-config finds the PKG_CONFIG_DEBUG_SPEW env var, it explains all the steps used to resolve the packages:
PKG_CONFIG_DEBUG_SPEW=1 /usr/bin/pkg-config --libs x11
This is useful to know why a particular package isn’t found and what are the default values for PKG_CONFIG_PATH when it’s not defined. For example:
Adding directory '/usr/local/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/local/lib/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/local/share/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/lib/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/share/pkgconfig' from PKG_CONFIG_PATH
If we have tuned PKG_CONFIG_PATH, maybe we also want to add the default paths. For example:
SYSROOT=~/sysroot-x86-64
export PKG_CONFIG_PATH=${SYSROOT}/usr/local/lib/pkgconfig:${SYSROOT}/usr/lib/pkgconfig
# Add also the standard pkg-config paths to find libraries in the system
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/local/lib/x86_64-linux-gnu/pkgconfig:\
/usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:\
/usr/lib/pkgconfig:/usr/share/pkgconfig
# This tells pkg-config where the "system" pkg-config dir is. This is useful when cross-compiling for other
# architecture, to avoid pkg-config using the system .pc files and mixing host and target libraries
export PKG_CONFIG_LIBDIR=${SYSROOT}/usr/lib
# This could have been used for cross compiling:
#export PKG_CONFIG_SYSROOT_DIR=${SYSROOT}
Man in the middle proxy for WebKit
Sometimes it’s useful to use our own modified/unminified files with a 3rd party service we don’t control. Mitmproxy can be used as a man-in-the-middle proxy, but I haven’t tried it personally yet. What I have tried (with WPE) is this:
Add an /etc/hosts entry to point the host serving the files we want to change to an IP address controlled by us.
Configure a web server to provide the files in the expected path.
Modify the ResourceRequestBase constructor to change the HTTPS requests to HTTP when the hostname matches the target:
ResourceRequestBase(const URL& url, ResourceRequestCachePolicy policy)
: m_url(url)
, m_timeoutInterval(s_defaultTimeoutInterval)
...
, m_isAppBound(false)
{
if (m_url.host().toStringWithoutCopying().containsIgnoringASCIICase(String("out-of-control-service.com"))
&& m_url.protocol().containsIgnoringASCIICase(String("https"))) {
printf("### %s: URL %s detected, changing from https to http\n",
__PRETTY_FUNCTION__, m_url.string().utf8().data());
fflush(stdout);
m_url.setProtocol(String("http"));
}
}
:bulb: Pro tip: If you have to debug minified/obfuscated JavaScript code and don’t have a deobfuscated version to use in a man-in-the-middle fashion, use http://www.jsnice.org/ to deobfuscate it and get meaningful variable names.
Bandwidth control for a dependent device
If your computer has a “shared internet connection” enabled in Network Manager and provides access to a dependent device , you can control the bandwidth offered to that device. This is useful to trigger quality changes on adaptive streaming videos from services out of your control.
This can be done using tc, the Traffic Control tool from the Linux kernel. You can use this script to automate the process (edit it to suit to your needs).
Useful scripts to process GStreamer logs
I use these scripts in my daily job to look for strange patterns in GStreamer logs that help me to find the cause of the bugs I’m debugging:
h: Highlights each expression in the command line in a different color.
mgrep: Greps (only) for the lines with the expressions in the command line and highlights each expression in a different color.
filter-time: Gets a subset of the log lines between a start and (optionally) an end GStreamer log timestamp.
highlight-threads: Highlights each thread in a GStreamer log with a different color. That way it’s easier to follow a thread with the naked eye.
remove-ansi-colors: Removes the color codes from a colored GStreamer log.
aha: ANSI-HTML-Adapter converts plain text with color codes to HTML, so you can share your GStreamer logs from a web server (eg: for bug discussion). Available in most distros.
gstbuffer-leak-analyzer: Analyzes a GStreamer log and shows unbalances in the creation/destruction of GstBuffer and GstMemory objects.
0 0 [Less]
|
Posted
almost 2 years
ago
by
Enrique Ocaña González
This is the last post of the series showing interesting debugging tools, I hope you have found it useful. Don’t miss the custom scripts at the bottom to process GStreamer logs, help you highlight the interesting parts and find the root cause of
... [More]
difficult bugs. Here are also the previous posts of the series:
GStreamer WebKit debugging tricks using GDB (1/2)
GStreamer WebKit debugging tricks using GDB (2/2)
GStreamer WebKit debugging by instrumenting source code (1/3)
GStreamer WebKit debugging by instrumenting source code (2/3)
GStreamer WebKit debugging by instrumenting source code (3/3)
GStreamer WebKit debugging by using external tools (1/2)
How to debug pkgconfig
When pkg-config finds the PKG_CONFIG_DEBUG_SPEW env var, it explains all the steps used to resolve the packages:
PKG_CONFIG_DEBUG_SPEW=1 /usr/bin/pkg-config --libs x11
This is useful to know why a particular package isn’t found and what are the default values for PKG_CONFIG_PATH when it’s not defined. For example:
Adding directory '/usr/local/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/local/lib/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/local/share/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/lib/x86_64-linux-gnu/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/lib/pkgconfig' from PKG_CONFIG_PATH
Adding directory '/usr/share/pkgconfig' from PKG_CONFIG_PATH
If we have tuned PKG_CONFIG_PATH, maybe we also want to add the default paths. For example:
SYSROOT=~/sysroot-x86-64
export PKG_CONFIG_PATH=${SYSROOT}/usr/local/lib/pkgconfig:${SYSROOT}/usr/lib/pkgconfig
# Add also the standard pkg-config paths to find libraries in the system
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/local/lib/x86_64-linux-gnu/pkgconfig:\
/usr/local/lib/pkgconfig:/usr/local/share/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:\
/usr/lib/pkgconfig:/usr/share/pkgconfig
# This tells pkg-config where the "system" pkg-config dir is. This is useful when cross-compiling for other
# architecture, to avoid pkg-config using the system .pc files and mixing host and target libraries
export PKG_CONFIG_LIBDIR=${SYSROOT}/usr/lib
# This could have been used for cross compiling:
#export PKG_CONFIG_SYSROOT_DIR=${SYSROOT}
Man in the middle proxy for WebKit
Sometimes it’s useful to use our own modified/unminified files with a 3rd party service we don’t control. Mitmproxy can be used as a man-in-the-middle proxy, but I haven’t tried it personally yet. What I have tried (with WPE) is this:
Add an /etc/hosts entry to point the host serving the files we want to change to an IP address controlled by us.
Configure a web server to provide the files in the expected path.
Modify the ResourceRequestBase constructor to change the HTTPS requests to HTTP when the hostname matches the target:
ResourceRequestBase(const URL& url, ResourceRequestCachePolicy policy)
: m_url(url)
, m_timeoutInterval(s_defaultTimeoutInterval)
...
, m_isAppBound(false)
{
if (m_url.host().toStringWithoutCopying().containsIgnoringASCIICase(String("out-of-control-service.com"))
&& m_url.protocol().containsIgnoringASCIICase(String("https"))) {
printf("### %s: URL %s detected, changing from https to http\n",
__PRETTY_FUNCTION__, m_url.string().utf8().data());
fflush(stdout);
m_url.setProtocol(String("http"));
}
}
:bulb: Pro tip: If you have to debug minified/obfuscated JavaScript code and don’t have a deobfuscated version to use in a man-in-the-middle fashion, use http://www.jsnice.org/ to deobfuscate it and get meaningful variable names.
Bandwidth control for a dependent device
If your computer has a “shared internet connection” enabled in Network Manager and provides access to a dependent device , you can control the bandwidth offered to that device. This is useful to trigger quality changes on adaptive streaming videos from services out of your control.
This can be done using tc, the Traffic Control tool from the Linux kernel. You can use this script to automate the process (edit it to suit to your needs).
Useful scripts to process GStreamer logs
I use these scripts in my daily job to look for strange patterns in GStreamer logs that help me to find the cause of the bugs I’m debugging:
h: Highlights each expression in the command line in a different color.
mgrep: Greps (only) for the lines with the expressions in the command line and highlights each expression in a different color.
filter-time: Gets a subset of the log lines between a start and (optionally) an end GStreamer log timestamp.
highlight-threads: Highlights each thread in a GStreamer log with a different color. That way it’s easier to follow a thread with the naked eye.
remove-ansi-colors: Removes the color codes from a colored GStreamer log.
aha: ANSI-HTML-Adapter converts plain text with color codes to HTML, so you can share your GStreamer logs from a web server (eg: for bug discussion). Available in most distros.
gstbuffer-leak-analyzer: Analyzes a GStreamer log and shows unbalances in the creation/destruction of GstBuffer and GstMemory objects.
0 0 [Less]
|
Posted
almost 2 years
ago
by
Enrique Ocaña González
In this new post series, I’ll show you how both existing and ad-hoc tools can be helpful to find the root cause of some problems. Here are also the older posts of this series in case you find them useful:
GStreamer WebKit debugging tricks using
... [More]
GDB (1/2)
GStreamer WebKit debugging tricks using GDB (2/2)
GStreamer WebKit debugging by instrumenting source code (1/3)
GStreamer WebKit debugging by instrumenting source code (2/3)
GStreamer WebKit debugging by instrumenting source code (3/3)
Use strace to know which config/library files are used by a program
If you’re becoming crazy supposing that the program should use some config and it seems to ignore it, just use strace to check what config files, libraries or other kind of files is the program actually using. Use the grep rules you need to refine the search:
$ strace -f -e trace=%file nano 2> >(grep 'nanorc')
access("/etc/nanorc", R_OK) = 0
access("/usr/share/nano/javascript.nanorc", R_OK) = 0
access("/usr/share/nano/gentoo.nanorc", R_OK) = 0
...
Know which process is killing another one
First, try to strace -e trace=signal -p 1234 the killed process.
If that doesn’t work (eg: because it’s being killed with the uncatchable SIGKILL signal), then you can resort to modifying the kernel source code (signal.c) to log the calls to kill():
SYSCALL_DEFINE2(kill, pid_t, pid, int, sig)
{
struct task_struct *tsk_p;
...
/* Log SIGKILL */
if (sig & 0x1F == 9) {
tsk_p = find_task_by_vpid(pid);
if (tsk_p) {
printk(KERN_DEBUG "Sig: %d from pid: %d (%s) to pid: %d (%s)\n",
sig, current->pid, current->comm, pid, tsk_p->comm);
} else {
printk(KERN_DEBUG "Sig: %d from pid: %d (%s) to pid: %d\n",
sig, current->pid, current->comm, pid);
}
}
...
}
Wrap gcc/ld/make to tweak build parameters
If you ever find yourself with little time in front of a stubborn build system and, no matter what you try, you can’t get the right flags to the compiler, think about putting something (a wrapper) between the build system and the compiler. Example for g++:
#!/bin/bash
main() {
# Build up arg[] array with all options to be passed
# to subcommand.
i=0
for opt in "$@"; do
case "$opt" in
-O2) ;; # Removes this option
*)
arg[i]="$opt" # Keeps the others
i=$((i+1))
;;
esac
done
EXTRA_FLAGS="-O0" # Adds extra option
echo "g++ ${EXTRA_FLAGS} ${arg[@]}" # >> /tmp/build.log # Logs the command
/usr/bin/ccache g++ ${EXTRA_FLAGS} "${arg[@]}" # Runs the command
}
main "$@"
Make sure that the wrappers appear earlier than the real commands in your PATH.
The make wrapper can also call remake instead. Remake is fully compatible with make but has features to help debugging compilation and makefile errors.
Analyze the structure of MP4 data
The ISOBMFF Box Structure Viewer online tool allows you to upload an MP4 file and explore its structure.
0 0 [Less]
|