Maemo

In this post I show some more useful debugging tricks. Check also the other posts of the series: GStreamer WebKit debugging tricks using GDB (1/2) GStreamer WebKit debugging tricks using GDB (2/2) GStreamer WebKit debugging by instrumenting ... [More] source code (1/3) Print current thread id The thread id is generated by Linux and can take values higher than 1-9, just like PIDs. This thread number is useful to know which function calls are issued by the same thread, avoiding confusion between threads. #include #include #include printf("%s [%d]\n", __PRETTY_FUNCTION__, syscall(SYS_gettid)); fflush(stdout); Debug GStreamer thread locks We redefine the GST_OBJECT_LOCK/UNLOCK/TRYLOCK macros to print the calls, compare locks against unlocks, and see who’s not releasing its lock: #include "wtf/Threading.h" #define GST_OBJECT_LOCK(obj) do { \ printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \ g_mutex_lock(GST_OBJECT_GET_LOCK(obj)); \ } while (0) #define GST_OBJECT_UNLOCK(obj) do { \ printf("### [UNLOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \ g_mutex_unlock(GST_OBJECT_GET_LOCK(obj)); \ } while (0) #define GST_OBJECT_TRYLOCK(obj) ({ \ gboolean result = g_mutex_trylock(GST_OBJECT_GET_LOCK(obj)); \ if (result) { \ printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \ } \ result; \ }) Warning: The statement expression that allows the TRYLOCK macro to return a value will only work on GCC. There’s a way to know which thread has taken a lock in glib/GStreamer using gdb. First locate the stalled thread: (gdb) thread (gdb) bt #2 0x74f07416 in pthread_mutex_lock () #3 0x7488aec6 in gst_pad_query () #4 0x6debebf2 in autoplug_query_allocation () (gdb) frame 3 #3 0x7488aec6 in gst_pad_query ([email protected]=0x54a9b8, ...) 4058 GST_PAD_STREAM_LOCK (pad); Now get the process id (PID) and use the pthread_mutex_t structure to print the Linux thread id that has acquired the lock: (gdb) call getpid() $30 = 6321 (gdb) p ((pthread_mutex_t*)pad.stream_rec_lock.p)->__data.__owner $31 = 6368 (gdb) thread find 6321.6368 Thread 21 has target id 'Thread 6321.6368' Trace function calls (poor developer version) If you’re using C++, you can define a tracer class. This is for webkit, but you get the idea: #define MYTRACER() MyTracer(__PRETTY_FUNCTION__); class MyTracer { public: MyTracer(const gchar* functionName) : m_functionName(functionName) { printf("### %s : begin %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout); } virtual ~MyTracer() { printf("### %s : end %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout); } private: String m_functionName; }; And use it like this in all the functions you want to trace: void somefunction() { MYTRACER(); // Some other code... } The constructor will log when the execution flow enters into the function and the destructor will log when the flow exits. Setting breakpoints from C In the C code, just call raise(SIGINT) (simulate CTRL+C, normally the program would finish). And then, in a previously attached gdb, after breaking and having debugging all you needed, just continue the execution by ignoring the signal or just plainly continuing: (gdb) signal 0 (gdb) continue There’s a way to do the same but attaching gdb after the raise. Use raise(SIGSTOP) instead (simulate CTRL+Z). Then attach gdb, locate the thread calling raise and switch to it: (gdb) thread apply all bt [now search for "raise" in the terminal log] Thread 36 (Thread 1977.2033): #1 0x74f5b3f2 in raise () from /home/enrique/buildroot/output2/staging/lib/libpthread.so.0 (gdb) thread 36 Now, from a terminal, send a continuation signal: kill -SIGCONT 1977. Finally instruct gdb to single-step only the current thread (IMPORTANT!) and run some steps until all the raises have been processed: (gdb) set scheduler-locking on (gdb) next // Repeat several times... Know the name of a GStreamer function stored in a pointer at runtime Just use this macro: GST_DEBUG_FUNCPTR_NAME(func) Detecting memory leaks in WebKit RefCountedLeakCounter is a tool class that can help to debug reference leaks by printing this kind of messages when WebKit exits: LEAK: 2 XMLHttpRequest LEAK: 25 CachedResource LEAK: 3820 WebCoreNode To use it you have to modify the particular class you want to debug: Include wtf/RefCountedLeakCounter.h DEFINE_DEBUG_ONLY_GLOBAL(WTF::RefCountedLeakCounter, myClassCounter, ("MyClass")); In the constructor: myClassCounter.increment() In the destructor: myClassCounter.decrement() 0 0 [Less]

In this post I show some more useful debugging tricks. Check also the other posts of the series: GStreamer WebKit debugging tricks using GDB (1/2) GStreamer WebKit debugging tricks using GDB (2/2) GStreamer WebKit debugging by instrumenting ... [More] source code (1/3) Print current thread id The thread id is generated by Linux and can take values higher than 1-9, just like PIDs. This thread number is useful to know which function calls are issued by the same thread, avoiding confusion between threads. #include #include #include printf("%s [%d]\n", __PRETTY_FUNCTION__, syscall(SYS_gettid)); fflush(stdout); Debug GStreamer thread locks We redefine the GST_OBJECT_LOCK/UNLOCK/TRYLOCK macros to print the calls, compare locks against unlocks, and see who’s not releasing its lock: #include "wtf/Threading.h" #define GST_OBJECT_LOCK(obj) do { \ printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \ g_mutex_lock(GST_OBJECT_GET_LOCK(obj)); \ } while (0) #define GST_OBJECT_UNLOCK(obj) do { \ printf("### [UNLOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \ g_mutex_unlock(GST_OBJECT_GET_LOCK(obj)); \ } while (0) #define GST_OBJECT_TRYLOCK(obj) ({ \ gboolean result = g_mutex_trylock(GST_OBJECT_GET_LOCK(obj)); \ if (result) { \ printf("### [LOCK] %s [%p]\n", __PRETTY_FUNCTION__, &Thread::current()); fflush(stdout); \ } \ result; \ }) Warning: The statement expression that allows the TRYLOCK macro to return a value will only work on GCC. There’s a way to know which thread has taken a lock in glib/GStreamer using gdb. First locate the stalled thread: (gdb) thread (gdb) bt #2 0x74f07416 in pthread_mutex_lock () #3 0x7488aec6 in gst_pad_query () #4 0x6debebf2 in autoplug_query_allocation () (gdb) frame 3 #3 0x7488aec6 in gst_pad_query ([email protected]=0x54a9b8, ...) 4058 GST_PAD_STREAM_LOCK (pad); Now get the process id (PID) and use the pthread_mutex_t structure to print the Linux thread id that has acquired the lock: (gdb) call getpid() $30 = 6321 (gdb) p ((pthread_mutex_t*)pad.stream_rec_lock.p)->__data.__owner $31 = 6368 (gdb) thread find 6321.6368 Thread 21 has target id 'Thread 6321.6368' Trace function calls (poor developer version) If you’re using C++, you can define a tracer class. This is for webkit, but you get the idea: #define MYTRACER() MyTracer(__PRETTY_FUNCTION__); class MyTracer { public: MyTracer(const gchar* functionName) : m_functionName(functionName) { printf("### %s : begin %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout); } virtual ~MyTracer() { printf("### %s : end %d\n", m_functionName.utf8().data(), currentThread()); fflush(stdout); } private: String m_functionName; }; And use it like this in all the functions you want to trace: void somefunction() { MYTRACER(); // Some other code... } The constructor will log when the execution flow enters into the function and the destructor will log when the flow exits. Setting breakpoints from C In the C code, just call raise(SIGINT) (simulate CTRL+C, normally the program would finish). And then, in a previously attached gdb, after breaking and having debugging all you needed, just continue the execution by ignoring the signal or just plainly continuing: (gdb) signal 0 (gdb) continue There’s a way to do the same but attaching gdb after the raise. Use raise(SIGSTOP) instead (simulate CTRL+Z). Then attach gdb, locate the thread calling raise and switch to it: (gdb) thread apply all bt [now search for "raise" in the terminal log] Thread 36 (Thread 1977.2033): #1 0x74f5b3f2 in raise () from /home/enrique/buildroot/output2/staging/lib/libpthread.so.0 (gdb) thread 36 Now, from a terminal, send a continuation signal: kill -SIGCONT 1977. Finally instruct gdb to single-step only the current thread (IMPORTANT!) and run some steps until all the raises have been processed: (gdb) set scheduler-locking on (gdb) next // Repeat several times... Know the name of a GStreamer function stored in a pointer at runtime Just use this macro: GST_DEBUG_FUNCPTR_NAME(func) Detecting memory leaks in WebKit RefCountedLeakCounter is a tool class that can help to debug reference leaks by printing this kind of messages when WebKit exits: LEAK: 2 XMLHttpRequest LEAK: 25 CachedResource LEAK: 3820 WebCoreNode To use it you have to modify the particular class you want to debug: Include wtf/RefCountedLeakCounter.h DEFINE_DEBUG_ONLY_GLOBAL(WTF::RefCountedLeakCounter, myClassCounter, ("MyClass")); In the constructor: myClassCounter.increment() In the destructor: myClassCounter.decrement() 0 0 [Less]

This is the continuation of the GStreamer WebKit debugging tricks post series. In the next three posts, I’ll focus on what we can get by doing some little changes to the source code for debugging purposes (known as “instrumenting”), but before, you ... [More] might want to check the previous posts of the series: GStreamer WebKit debugging tricks using GDB (1/2) GStreamer WebKit debugging tricks using GDB (2/2) Know all the env vars read by a program by using LD_PRELOAD to intercept libc calls // File getenv.c // To compile: gcc -shared -Wall -fPIC -o getenv.so getenv.c -ldl // To use: export LD_PRELOAD="./getenv.so", then run any program you want // See http://www.catonmat.net/blog/simple-ld-preload-tutorial-part-2/ #define _GNU_SOURCE #include #include // This function will take the place of the original getenv() in libc char *getenv(const char *name) { printf("Calling getenv(\"%s\")\n", name); char *(*original_getenv)(const char*); original_getenv = dlsym(RTLD_NEXT, "getenv"); return (*original_getenv)(name); } See the breakpoints with command example to know how to get the same using gdb. Check also Zan’s libpine for more features. Track lifetime of GObjects by LD_PRELOADing gobject-list The gobject-list project, written by Thibault Saunier, is a simple LD_PRELOAD library for tracking the lifetime of GObjects. When loaded into an application, it prints a list of living GObjects on exiting the application (unless the application crashes), and also prints reference count data when it changes. SIGUSR1 or SIGUSR2 can be sent to the application to trigger printing of more information. Overriding the behaviour of a debugging macro The usual debugging macros aren’t printing messages? Redefine them to make what you want: #undef LOG_MEDIA_MESSAGE #define LOG_MEDIA_MESSAGE(...) do { \ printf("LOG %s: ", __PRETTY_FUNCTION__); \ printf(__VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while(0) This can be done to enable asserts on demand in WebKit too: #undef ASSERT #define ASSERT(assertion) \ (!(assertion) ? \ (WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, #assertion), \ CRASH()) : \ (void)0) #undef ASSERT_NOT_REACHED #define ASSERT_NOT_REACHED() do { \ WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, 0); \ CRASH(); \ } while (0) It may be interesting to enable WebKit LOG() and GStreamer GST_DEBUG() macros only on selected files: #define LOG(channel, msg, ...) do { \ printf("%s: ", #channel); \ printf(msg, ## __VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while (false) #define _GST_DEBUG(msg, ...) do { \ printf("### %s: ", __PRETTY_FUNCTION__); \ printf(msg, ## __VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while (false) Note all the preprocessor trickery used here: First arguments (channel, msg) are captured intependently The remaining args are captured in __VA_ARGS__ do while(false) is a trick to avoid {braces} and make the code block work when used in if/then/else one-liners #channel expands LOG(MyChannel,....) as printf("%s: ", "MyChannel"). It’s called “stringification”. ## __VA_ARGS__ expands the variable argument list as a comma-separated list of items, but if the list is empty, it eats the comma after “msg”, preventing syntax errors Print the compile-time type of an expression Use typeid().name(). Filter the ouput through c++filt -t: std::vector v; printf("Type: %s\n", typeid(v.begin()).name()); Abusing the compiler to know all the places where a function is called If you want to know all the places from where the GstClockTime toGstClockTime(float time) function is called, you can convert it to a template function and use static_assert on a wrong datatype like this (in the .h): template GstClockTime toGstClockTime(float time) { static_assert(std::is_integral::value, "Don't call toGstClockTime(float)!"); return 0; } Note that T=float is different to integer (is_integral). It has nothing to do with the float time parameter declaration. You will get compile-time errors like this on every place the function is used: WebKitMediaSourceGStreamer.cpp:474:87: required from here GStreamerUtilities.h:84:43: error: static assertion failed: Don't call toGstClockTime(float)! Use pragma message to print values at compile time Sometimes is useful to know if a particular define is enabled: #include #define _STR(x) #x #define STR(x) _STR(x) #pragma message "Int max is " STR(INT_MAX) #ifdef WHATEVER #pragma message "Compilation goes by here" #else #pragma message "Compilation goes by there" #endif ... The code above would generate this output: test.c:6:9: note: #pragma message: Int max is 0x7fffffff #pragma message "Int max is " STR(INT_MAX) ^~~~~~~ test.c:11:9: note: #pragma message: Compilation goes by there #pragma message "Compilation goes by there" ^~~~~~~ 0 0 [Less]

This is the continuation of the GStreamer WebKit debugging tricks post series. In the next three posts, I’ll focus on what we can get by doing some little changes to the source code for debugging purposes (known as “instrumenting”), but before, you ... [More] might want to check the previous posts of the series: GStreamer WebKit debugging tricks using GDB (1/2) GStreamer WebKit debugging tricks using GDB (2/2) Know all the env vars read by a program by using LD_PRELOAD to intercept libc calls // File getenv.c // To compile: gcc -shared -Wall -fPIC -o getenv.so getenv.c -ldl // To use: export LD_PRELOAD="./getenv.so", then run any program you want // See http://www.catonmat.net/blog/simple-ld-preload-tutorial-part-2/ #define _GNU_SOURCE #include #include // This function will take the place of the original getenv() in libc char *getenv(const char *name) { printf("Calling getenv(\"%s\")\n", name); char *(*original_getenv)(const char*); original_getenv = dlsym(RTLD_NEXT, "getenv"); return (*original_getenv)(name); } See the breakpoints with command example to know how to get the same using gdb. Check also Zan’s libpine for more features. Track lifetime of GObjects by LD_PRELOADing gobject-list The gobject-list project, written by Thibault Saunier, is a simple LD_PRELOAD library for tracking the lifetime of GObjects. When loaded into an application, it prints a list of living GObjects on exiting the application (unless the application crashes), and also prints reference count data when it changes. SIGUSR1 or SIGUSR2 can be sent to the application to trigger printing of more information. Overriding the behaviour of a debugging macro The usual debugging macros aren’t printing messages? Redefine them to make what you want: #undef LOG_MEDIA_MESSAGE #define LOG_MEDIA_MESSAGE(...) do { \ printf("LOG %s: ", __PRETTY_FUNCTION__); \ printf(__VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while(0) This can be done to enable asserts on demand in WebKit too: #undef ASSERT #define ASSERT(assertion) \ (!(assertion) ? \ (WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, #assertion), \ CRASH()) : \ (void)0) #undef ASSERT_NOT_REACHED #define ASSERT_NOT_REACHED() do { \ WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, 0); \ CRASH(); \ } while (0) It may be interesting to enable WebKit LOG() and GStreamer GST_DEBUG() macros only on selected files: #define LOG(channel, msg, ...) do { \ printf("%s: ", #channel); \ printf(msg, ## __VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while (false) #define _GST_DEBUG(msg, ...) do { \ printf("### %s: ", __PRETTY_FUNCTION__); \ printf(msg, ## __VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while (false) Note all the preprocessor trickery used here: First arguments (channel, msg) are captured intependently The remaining args are captured in __VA_ARGS__ do while(false) is a trick to avoid {braces} and make the code block work when used in if/then/else one-liners #channel expands LOG(MyChannel,....) as printf("%s: ", "MyChannel"). It’s called “stringification”. ## __VA_ARGS__ expands the variable argument list as a comma-separated list of items, but if the list is empty, it eats the comma after “msg”, preventing syntax errors Print the compile-time type of an expression Use typeid().name(). Filter the ouput through c++filt -t: std::vector v; printf("Type: %s\n", typeid(v.begin()).name()); Abusing the compiler to know all the places where a function is called If you want to know all the places from where the GstClockTime toGstClockTime(float time) function is called, you can convert it to a template function and use static_assert on a wrong datatype like this (in the .h): template GstClockTime toGstClockTime(float time) { static_assert(std::is_integral::value, "Don't call toGstClockTime(float)!"); return 0; } Note that T=float is different to integer (is_integral). It has nothing to do with the float time parameter declaration. You will get compile-time errors like this on every place the function is used: WebKitMediaSourceGStreamer.cpp:474:87: required from here GStreamerUtilities.h:84:43: error: static assertion failed: Don't call toGstClockTime(float)! Use pragma message to print values at compile time Sometimes is useful to know if a particular define is enabled: #include #define _STR(x) #x #define STR(x) _STR(x) #pragma message "Int max is " STR(INT_MAX) #ifdef WHATEVER #pragma message "Compilation goes by here" #else #pragma message "Compilation goes by there" #endif ... The code above would generate this output: test.c:6:9: note: #pragma message: Int max is 0x7fffffff #pragma message "Int max is " STR(INT_MAX) ^~~~~~~ test.c:11:9: note: #pragma message: Compilation goes by there #pragma message "Compilation goes by there" ^~~~~~~ 0 0 [Less]

This is the continuation of the GStreamer WebKit debugging tricks post series. In the next three posts, I’ll focus on what we can get by doing some little changes to the source code for debugging purposes (known as “instrumenting”), but before, you ... [More] might want to check the previous posts of the series: GStreamer WebKit debugging tricks using GDB (1/2) GStreamer WebKit debugging tricks using GDB (2/2) Know all the env vars read by a program by using LD_PRELOAD to intercept libc calls // File getenv.c // To compile: gcc -shared -Wall -fPIC -o getenv.so getenv.c -ldl // To use: export LD_PRELOAD="./getenv.so", then run any program you want // See http://www.catonmat.net/blog/simple-ld-preload-tutorial-part-2/ #define _GNU_SOURCE #include #include // This function will take the place of the original getenv() in libc char *getenv(const char *name) { printf("Calling getenv(\"%s\")\n", name); char *(*original_getenv)(const char*); original_getenv = dlsym(RTLD_NEXT, "getenv"); return (*original_getenv)(name); } See the breakpoints with command example to know how to get the same using gdb. Check also Zan’s libpine for more features. Track lifetime of GObjects by LD_PRELOADing gobject-list The gobject-list project, written by Thibault Saunier, is a simple LD_PRELOAD library for tracking the lifetime of GObjects. When loaded into an application, it prints a list of living GObjects on exiting the application (unless the application crashes), and also prints reference count data when it changes. SIGUSR1 or SIGUSR2 can be sent to the application to trigger printing of more information. Overriding the behaviour of a debugging macro The usual debugging macros aren’t printing messages? Redefine them to make what you want: #undef LOG_MEDIA_MESSAGE #define LOG_MEDIA_MESSAGE(...) do { \ printf("LOG %s: ", __PRETTY_FUNCTION__); \ printf(__VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while(0) This can be done to enable asserts on demand in WebKit too: #undef ASSERT #define ASSERT(assertion) \ (!(assertion) ? \ (WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, #assertion), \ CRASH()) : \ (void)0) #undef ASSERT_NOT_REACHED #define ASSERT_NOT_REACHED() do { \ WTFReportAssertionFailure(__FILE__, __LINE__, WTF_PRETTY_FUNCTION, 0); \ CRASH(); \ } while (0) It may be interesting to enable WebKit LOG() and GStreamer GST_DEBUG() macros only on selected files: #define LOG(channel, msg, ...) do { \ printf("%s: ", #channel); \ printf(msg, ## __VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while (false) #define _GST_DEBUG(msg, ...) do { \ printf("### %s: ", __PRETTY_FUNCTION__); \ printf(msg, ## __VA_ARGS__); \ printf("\n"); \ fflush(stdout); \ } while (false) Note all the preprocessor trickery used here: First arguments (channel, msg) are captured intependently The remaining args are captured in __VA_ARGS__ do while(false) is a trick to avoid {braces} and make the code block work when used in if/then/else one-liners #channel expands LOG(MyChannel,....) as printf("%s: ", "MyChannel"). It’s called “stringification”. ## __VA_ARGS__ expands the variable argument list as a comma-separated list of items, but if the list is empty, it eats the comma after “msg”, preventing syntax errors Print the compile-time type of an expression Use typeid().name(). Filter the ouput through c++filt -t: std::vector v; printf("Type: %s\n", typeid(v.begin()).name()); Abusing the compiler to know all the places where a function is called If you want to know all the places from where the GstClockTime toGstClockTime(float time) function is called, you can convert it to a template function and use static_assert on a wrong datatype like this (in the .h): template GstClockTime toGstClockTime(float time) { static_assert(std::is_integral::value, "Don't call toGstClockTime(float)!"); return 0; } Note that T=float is different to integer (is_integral). It has nothing to do with the float time parameter declaration. You will get compile-time errors like this on every place the function is used: WebKitMediaSourceGStreamer.cpp:474:87: required from here GStreamerUtilities.h:84:43: error: static assertion failed: Don't call toGstClockTime(float)! Use pragma message to print values at compile time Sometimes is useful to know if a particular define is enabled: #include #define _STR(x) #x #define STR(x) _STR(x) #pragma message "Int max is " STR(INT_MAX) #ifdef WHATEVER #pragma message "Compilation goes by here" #else #pragma message "Compilation goes by there" #endif ... The code above would generate this output: test.c:6:9: note: #pragma message: Int max is 0x7fffffff #pragma message "Int max is " STR(INT_MAX) ^~~~~~~ test.c:11:9: note: #pragma message: Compilation goes by there #pragma message "Compilation goes by there" ^~~~~~~ 0 0 [Less]

This post is a continuation of a series of blog posts about the most interesting debugging tricks I’ve found while working on GStreamer WebKit on embedded devices. These are the other posts of the series published so far: GStreamer WebKit ... [More] debugging tricks using GDB (1/2) Print corrupt stacktraces In some circumstances you may get stacktraces that eventually stop because of missing symbols or corruption (?? entries). #3 0x01b8733c in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) However, you can print the stack in a useful way that gives you leads about what was next in the stack: For i386: x/256wa $esp For x86_64: x/256ga $rsp For ARM 32 bit: x/256wa $sp You may want to enable asm-demangle: set print asm-demangle Example output, the 3 last lines give interesting info: 0x7ef85550: 0x1b87400 0x2 0x0 0x1b87400 0x7ef85560: 0x0 0x1b87140 0x1b87140 0x759e88a4 0x7ef85570: 0x1b87330 0x759c71a9 0x140c 0x1b87330 0x7ef85580: 0x759e88a4 0x7ef855b4 0x0 0x7ef855b4 ... 0x7ef85830: 0x76dbd6c4 0x4 0x3 0x1bfeb50 0x7ef85840: 0x0 0x76d59268 0x75135374 0x75135374 0x7ef85850: 0x76dbd6c4 0x1b7e300 0x1d651d0 0x75151b74 More info: 1 Sometimes the symbol names aren’t printed in the stack memdump. You can do this trick to iterate the stack and print the symbols found there (take with a grain of salt!): (gdb) set $i = 0 (gdb) p/a *((void**)($sp + 4*$i++)) [Press ENTER multiple times to repeat the command] $46 = 0xb6f9fb17 <_dl_lookup_symbol_x+250> $58 = 0xb40a9001 $142 = 0xb40a877b $154 = 0xb65a93d5 $164 = 0xb65ab4e5 ... Many times it’s just a matter of gdb not having loaded the unstripped version of the library. /proc//smaps and info proc mappings can help to locate the library providing the missing symbol. Then we can load it by hand. For instance, for this backtrace: #0 0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 #1 0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 #2 0x6cfd0d60 in ?? () In a shell, we examine smaps and find out that the unknown piece of code comes from libgstomx: $ cat /proc/715/smaps ... 6cfc1000-6cff8000 r-xp 00000000 b3:02 785380 /usr/lib/gstreamer-1.0/libgstomx.so ... Now we load the unstripped .so in gdb and we’re able to see the new symbol afterwards: (gdb) add-symbol-file /home/enrique/buildroot-wpe/output/build/gst-omx-custom/omx/.libs/libgstomx.so 0x6cfc1000 (gdb) bt #0 0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 #1 0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 #2 0x6cfd0d60 in gst_omx_video_dec_loop (self=0x6e0c8130) at gstomxvideodec.c:1311 #3 0x6e0c8130 in ?? () Useful script to prepare the add-symbol-file: cat /proc/715/smaps | grep '[.]so' | sed -e 's/-[0-9a-f]*//' | { while read ADDR _ _ _ _ LIB; do echo "add-symbol-file $LIB 0x$ADDR"; done; } More info: 1 The “figuring out corrupt ARM stacktraces” post has some additional info about how to use addr2line to translate memory addresses to function names on systems with a hostile debugging environment. Debugging a binary without debug symbols There are times when there’s just no way to get debug symbols working, or where we’re simply debugging on a release version of the software. In those cases, we must directly debug the assembly code. The gdb text user interface (TUI) can be used to examine the disassebled code and the CPU registers. It can be enabled with these commands: layout asm layout regs set print asm-demangle Some useful keybindings in this mode: Arrows: scroll the disassemble window CTRL+p/n: Navigate history (previously done with up/down arrows) CTRL+b/f: Go backward/forward one character (previously left/right arrows) CTRL+d: Delete character (previously “Del” key) CTRL+a/e: Go to the start/end of the line This screenshot shows how we can infer that an empty RefPtr is causing a crash in some WebKit code. Wake up an unresponsive gdb on ARM Sometimes, when you continue (‘c’) execution on ARM there’s no way to stop it again unless a breakpoint is hit. But there’s a trick to retake the control: just send a harmless signal to the process. kill -SIGCONT 1234 Know which GStreamer thread id matches with each gdb thread Sometimes you need to match threads in the GStreamer logs with threads in a running gdb session. The simplest way is to ask it to GThread for each gdb thread: (gdb) set output-radix 16 (gdb) thread apply all call g_thread_self() This will print a list of gdb threads and GThread*. We only need to find the one we’re looking for. Generate a pipeline dump from gdb If we have a pointer to the pipeline object, we can call the function that dumps the pipeline: (gdb) call gst_debug_bin_to_dot_file_with_ts((GstBin*)0x15f0078, GST_DEBUG_GRAPH_SHOW_ALL, "debug") 0 0 [Less]

This post is a continuation of a series of blog posts about the most interesting debugging tricks I’ve found while working on GStreamer WebKit on embedded devices. These are the other posts of the series published so far: GStreamer WebKit ... [More] debugging tricks using GDB (1/2) Print corrupt stacktraces In some circumstances you may get stacktraces that eventually stop because of missing symbols or corruption (?? entries). #3 0x01b8733c in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) However, you can print the stack in a useful way that gives you leads about what was next in the stack: For i386: x/256wa $esp For x86_64: x/256ga $rsp For ARM 32 bit: x/256wa $sp You may want to enable asm-demangle: set print asm-demangle Example output, the 3 last lines give interesting info: 0x7ef85550: 0x1b87400 0x2 0x0 0x1b87400 0x7ef85560: 0x0 0x1b87140 0x1b87140 0x759e88a4 0x7ef85570: 0x1b87330 0x759c71a9 0x140c 0x1b87330 0x7ef85580: 0x759e88a4 0x7ef855b4 0x0 0x7ef855b4 ... 0x7ef85830: 0x76dbd6c4 0x4 0x3 0x1bfeb50 0x7ef85840: 0x0 0x76d59268 0x75135374 0x75135374 0x7ef85850: 0x76dbd6c4 0x1b7e300 0x1d651d0 0x75151b74 More info: 1 Sometimes the symbol names aren’t printed in the stack memdump. You can do this trick to iterate the stack and print the symbols found there (take with a grain of salt!): (gdb) set $i = 0 (gdb) p/a *((void**)($sp + 4*$i++)) [Press ENTER multiple times to repeat the command] $46 = 0xb6f9fb17 <_dl_lookup_symbol_x+250> $58 = 0xb40a9001 $142 = 0xb40a877b $154 = 0xb65a93d5 $164 = 0xb65ab4e5 ... Many times it’s just a matter of gdb not having loaded the unstripped version of the library. /proc//smaps and info proc mappings can help to locate the library providing the missing symbol. Then we can load it by hand. For instance, for this backtrace: #0 0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 #1 0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 #2 0x6cfd0d60 in ?? () In a shell, we examine smaps and find out that the unknown piece of code comes from libgstomx: $ cat /proc/715/smaps ... 6cfc1000-6cff8000 r-xp 00000000 b3:02 785380 /usr/lib/gstreamer-1.0/libgstomx.so ... Now we load the unstripped .so in gdb and we’re able to see the new symbol afterwards: (gdb) add-symbol-file /home/enrique/buildroot-wpe/output/build/gst-omx-custom/omx/.libs/libgstomx.so 0x6cfc1000 (gdb) bt #0 0x740ad3fc in syscall () from /home/enrique/buildroot-wpe/output/staging/lib/libc.so.6 #1 0x74375c44 in g_cond_wait () from /home/enrique/buildroot-wpe/output/staging/usr/lib/libglib-2.0.so.0 #2 0x6cfd0d60 in gst_omx_video_dec_loop (self=0x6e0c8130) at gstomxvideodec.c:1311 #3 0x6e0c8130 in ?? () Useful script to prepare the add-symbol-file: cat /proc/715/smaps | grep '[.]so' | sed -e 's/-[0-9a-f]*//' | { while read ADDR _ _ _ _ LIB; do echo "add-symbol-file $LIB 0x$ADDR"; done; } More info: 1 The “figuring out corrupt ARM stacktraces” post has some additional info about how to use addr2line to translate memory addresses to function names on systems with a hostile debugging environment. Debugging a binary without debug symbols There are times when there’s just no way to get debug symbols working, or where we’re simply debugging on a release version of the software. In those cases, we must directly debug the assembly code. The gdb text user interface (TUI) can be used to examine the disassebled code and the CPU registers. It can be enabled with these commands: layout asm layout regs set print asm-demangle Some useful keybindings in this mode: Arrows: scroll the disassemble window CTRL+p/n: Navigate history (previously done with up/down arrows) CTRL+b/f: Go backward/forward one character (previously left/right arrows) CTRL+d: Delete character (previously “Del” key) CTRL+a/e: Go to the start/end of the line This screenshot shows how we can infer that an empty RefPtr is causing a crash in some WebKit code. Wake up an unresponsive gdb on ARM Sometimes, when you continue (‘c’) execution on ARM there’s no way to stop it again unless a breakpoint is hit. But there’s a trick to retake the control: just send a harmless signal to the process. kill -SIGCONT 1234 Know which GStreamer thread id matches with each gdb thread Sometimes you need to match threads in the GStreamer logs with threads in a running gdb session. The simplest way is to ask it to GThread for each gdb thread: (gdb) set output-radix 16 (gdb) thread apply all call g_thread_self() This will print a list of gdb threads and GThread*. We only need to find the one we’re looking for. Generate a pipeline dump from gdb If we have a pointer to the pipeline object, we can call the function that dumps the pipeline: (gdb) call gst_debug_bin_to_dot_file_with_ts((GstBin*)0x15f0078, GST_DEBUG_GRAPH_SHOW_ALL, "debug") 0 0 [Less]

I’ve been developing and debugging desktop and mobile applications on embedded devices over the last decade or so. The main part of this period I’ve been focused on the multimedia side of the WebKit ports using GStreamer, an area that is a mix of C ... [More] (glib, GObject and GStreamer) and C++ (WebKit). Over these years I’ve had to work on ARM embedded devices (mobile phones, set-top-boxes, Raspberry Pi using buildroot) where most of the environment aids and tools we take for granted on a regular x86 Linux desktop just aren’t available. In these situations you have to be imaginative and find your own way to get the work done and debug the issues you find in along the way. I’ve been writing down the most interesting tricks I’ve found in this journey and I’m sharing them with you in a series of 7 blog posts, one per week. Most of them aren’t mine, and the ones I learnt in the begining of my career can even seem a bit naive, but I find them worth to share anyway. I hope you find them as useful as I do. Breakpoints with command You can break on a place, run some command and continue execution. Useful to get logs: break getenv command # This disables scroll continue messages # and supresses output silent set pagination off p (char*)$r0 continue end break grl-xml-factory.c:2720 if (data != 0) command call grl_source_get_id(data->source) # $ is the last value in the history, the result of # the previous call call grl_media_set_source (send_item->media, $) call grl_media_serialize_extended (send_item->media, GRL_MEDIA_SERIALIZE_FULL) continue end This idea can be combined with watchpoints and applied to trace reference counting in GObjects and know from which places the refcount is increased and decreased. Force execution of an if branch Just wait until the if chooses a branch and then jump to the other one: 6 if (i > 3) { (gdb) next 7 printf("%d > 3\n", i); (gdb) break 9 (gdb) jump 9 9 printf("%d <= 3\n", i); (gdb) next 5 <= 3 Debug glib warnings If you get a warning message like this: W/GLib-GObject(18414): g_object_unref: assertion `G_IS_OBJECT (object)' failed the functions involved are: g_return_if_fail_warning(), which calls to g_log(). It’s good to set a breakpoint in any of the two: break g_log Another method is to export G_DEBUG=fatal_criticals, which will convert all the criticals in crashes, which will stop the debugger. Debug GObjects If you want to inspect the contents of a GObjects that you have in a reference… (gdb) print web_settings $1 = (WebKitWebSettings *) 0x7fffffffd020 you can dereference it… (gdb) print *web_settings $2 = {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0} even if it’s an untyped gpointer… (gdb) print user_data (void *) 0x7fffffffd020 (gdb) print *((WebKitWebSettings *)(user_data)) {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0} To find the type, you can use GType: (gdb) call (char*)g_type_name( ((GTypeInstance*)0x70d1b038)->g_class->g_type ) $86 = 0x2d7e14 "GstOMXH264Dec-omxh264dec" Instantiate C++ object from gdb (gdb) call malloc(sizeof(std::string)) $1 = (void *) 0x91a6a0 (gdb) call ((std::string*)0x91a6a0)->basic_string() (gdb) call ((std::string*)0x91a6a0)->assign("Hello, World") $2 = (std::basic_string, std::allocator > &) @0x91a6a0: {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x91a6f8 "Hello, World"}} (gdb) call SomeFunctionThatTakesAConstStringRef(*(const std::string*)0x91a6a0) See: 1 and 2 0 0 [Less]

I’ve been developing and debugging desktop and mobile applications on embedded devices over the last decade or so. The main part of this period I’ve been focused on the multimedia side of the WebKit ports using GStreamer, an area that is a mix of C ... [More] (glib, GObject and GStreamer) and C++ (WebKit). Over these years I’ve had to work on ARM embedded devices (mobile phones, set-top-boxes, Raspberry Pi using buildroot) where most of the environment aids and tools we take for granted on a regular x86 Linux desktop just aren’t available. In these situations you have to be imaginative and find your own way to get the work done and debug the issues you find in along the way. I’ve been writing down the most interesting tricks I’ve found in this journey and I’m sharing them with you in a series of 7 blog posts, one per week. Most of them aren’t mine, and the ones I learnt in the begining of my career can even seem a bit naive, but I find them worth to share anyway. I hope you find them as useful as I do. Breakpoints with command You can break on a place, run some command and continue execution. Useful to get logs: break getenv command # This disables scroll continue messages # and supresses output silent set pagination off p (char*)$r0 continue end break grl-xml-factory.c:2720 if (data != 0) command call grl_source_get_id(data->source) # $ is the last value in the history, the result of # the previous call call grl_media_set_source (send_item->media, $) call grl_media_serialize_extended (send_item->media, GRL_MEDIA_SERIALIZE_FULL) continue end This idea can be combined with watchpoints and applied to trace reference counting in GObjects and know from which places the refcount is increased and decreased. Force execution of an if branch Just wait until the if chooses a branch and then jump to the other one: 6 if (i > 3) { (gdb) next 7 printf("%d > 3\n", i); (gdb) break 9 (gdb) jump 9 9 printf("%d <= 3\n", i); (gdb) next 5 <= 3 Debug glib warnings If you get a warning message like this: W/GLib-GObject(18414): g_object_unref: assertion `G_IS_OBJECT (object)' failed the functions involved are: g_return_if_fail_warning(), which calls to g_log(). It’s good to set a breakpoint in any of the two: break g_log Another method is to export G_DEBUG=fatal_criticals, which will convert all the criticals in crashes, which will stop the debugger. Debug GObjects If you want to inspect the contents of a GObjects that you have in a reference… (gdb) print web_settings $1 = (WebKitWebSettings *) 0x7fffffffd020 you can dereference it… (gdb) print *web_settings $2 = {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0} even if it’s an untyped gpointer… (gdb) print user_data (void *) 0x7fffffffd020 (gdb) print *((WebKitWebSettings *)(user_data)) {parent_instance = {g_type_instance = {g_class = 0x18}, ref_count = 0, qdata = 0x0}, priv = 0x0} To find the type, you can use GType: (gdb) call (char*)g_type_name( ((GTypeInstance*)0x70d1b038)->g_class->g_type ) $86 = 0x2d7e14 "GstOMXH264Dec-omxh264dec" Instantiate C++ object from gdb (gdb) call malloc(sizeof(std::string)) $1 = (void *) 0x91a6a0 (gdb) call ((std::string*)0x91a6a0)->basic_string() (gdb) call ((std::string*)0x91a6a0)->assign("Hello, World") $2 = (std::basic_string, std::allocator > &) @0x91a6a0: {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x91a6f8 "Hello, World"}} (gdb) call SomeFunctionThatTakesAConstStringRef(*(const std::string*)0x91a6a0) See: 1 and 2 0 0 [Less]

This depends on Bounce (the N900 .deb) and SDL 1.2 being installed. Google "bounce_1.0.0_armel.deb" for the former, and use n9repomirror for the latter.This is available on OpenRepos: https://openrepos.net/content/thp/rebounce Also on ... [More] TMO: https://talk.maemo.org/showthread.php?t=101160 Also on Twitter: https://twitter.com/thp4/status/1359758278620758019 Source code (copy'n'paste this into a file named "rebounce.c", then run it using your shell): #if 0 gcc -Os -shared -fPIC -lSDL -o librebounce.so rebounce.c LD_PRELOAD=$(pwd)/librebounce.so /opt/bounce/bin/bounce exit 0 #endif /**  * reBounce -- softfp-to-hardfp LD_PRELOAD hack for Bounce on N9  *  * Bounce was a really nice 2009 tech demo on the N900. This  * makes this tech demo work on an N9 by translating the calls  * that use floating point arguments to the hardfp ABI. It also  * fixes input using SDL1.2 to get sensor values from sensorfw.  *  * Known issues: Audio is muted on startup until mute is toggled.  *  * 2021-02-11 Thomas Perl  **/ #define _GNU_SOURCE #include #include #include #include #define SFP __attribute__((pcs("aapcs"))) typedef unsigned int GLuint; static void * sensor_thread(void *user_data) {     SDL_Init(SDL_INIT_JOYSTICK | SDL_INIT_VIDEO);     SDL_Joystick *accelerometer = SDL_JoystickOpen(0);     while (1) {         SDL_JoystickUpdate();         float x = 0.053888f * SDL_JoystickGetAxis(accelerometer, 0);         float y = 0.053888f * SDL_JoystickGetAxis(accelerometer, 1);         float z = 0.053888f * SDL_JoystickGetAxis(accelerometer, 2);         FILE *out = fopen("/dev/shm/bounce.sensor.tmp", "wb");         fprintf(out, "%f %f %f\n", -y, x, z);         fclose(out);         rename("/dev/shm/bounce.sensor.tmp", "/dev/shm/bounce.sensor");         SDL_Delay(10);     }     return NULL; } FILE * fopen(const char *filename, const char *mode) {     FILE *(*fopen_orig)(const char *, const char *) = dlsym(RTLD_NEXT, "fopen");     if (strcmp(filename, "/sys/class/i2c-adapter/i2c-3/3-001d/coord") == 0) {         static int sensor_inited = 0;         if (!sensor_inited) {             sensor_inited = 1;             pthread_t thread;             pthread_create(&thread, NULL, sensor_thread, NULL);         }         filename = "/dev/shm/bounce.sensor";     }     return fopen_orig(filename, mode); } #define f_f(name) float SFP name(float x) { return ((float (*)(float))dlsym(RTLD_NEXT, #name))(x); } #define d_d(name) double SFP name(double x) { return ((double (*)(double))dlsym(RTLD_NEXT, #name))(x); } #define f_ff(name) float SFP name(float x, float y) { return ((float (*)(float, float))dlsym(RTLD_NEXT, #name))(x, y); } #define d_dd(name) double SFP name(double x, double y) { return ((double (*)(double, double))dlsym(RTLD_NEXT, #name))(x, y); } f_f(sinhf) f_f(coshf) f_f(tanhf) f_f(asinf) f_f(acosf) f_f(atanf) f_f(sinf) f_f(cosf) f_f(tanf) f_f(expf) f_f(logf) f_f(log10f) f_f(ceilf) f_f(floorf) d_d(log) d_d(sin) f_ff(atan2f) f_ff(fmodf) d_dd(atan2) d_dd(pow) d_dd(fmod) double SFP frexp(double value, int *exp) {     return ((double (*)(double, int *))dlsym(RTLD_NEXT, "frexp"))(value, exp); } double SFP ldexp(double x, int n) {     return ((double (*)(double, int))dlsym(RTLD_NEXT, "ldexp"))(x, n); } double SFP modf(double value, double *iptr) {     return ((double (*)(double, double *))dlsym(RTLD_NEXT, "modf"))(value, iptr); } void SFP glClearColor(float r, float g, float b, float a) {     ((void (*)(float, float, float, float))dlsym(RTLD_NEXT, "glClearColor"))(r, g, b, a); } void SFP glUniform4f(GLuint location, float v0, float v1, float v2, float v3) {     ((void (*)(GLuint, float, float, float, float))dlsym(RTLD_NEXT, "glUniform4f"))(location, v0, v1, v2, v3); } void SFP glUniform1f(GLuint location, float v0) {     ((void (*)(GLuint, float))dlsym(RTLD_NEXT, "glUniform1f"))(location, v0); } 0 0 [Less]