Subject:
QDialog randomly misplaced invisible b/c QDesktopWidget::availableGeometry() reads _NET_WORKAREA wrong, patch


Introduction

After migrating a legacy application from jessie to buster without making major changes we experienced QDialogs and QMessageBoxes randomly showing up outside the visible area. 
In most of the occurences the invisible dialog didn't have focus despite being modal, rendering the GUI frozen from the user's perspective.


Bugs

We finally tracked it down to QDesktopWidget::availableGeometry(int screen) parsing the response of XGetWindowProperty(_NET_WORKAREA) as array of 4 64 bit integers despite format==32 clearly guaranteeing 32 bit integers.

This 'compresses' the 4 good values received in the first 2 values of the QRect workArea c'tor, x and y, and pulls random memory content in the other 2 values, width and heigth.

As a consequence QMessageBoxPrivate::updateSize() calculated hardLimit, softLimit, width and height to crazy negative values like -123456789.
That whole method, QMessageBoxPrivate::updateSize(), has some wild heuristics and should not use negative values, but that's not the root cause.

Depending on the window manager and system configuration the bad code path in QDesktopWidget::availableGeometry(int) might be used or not.
It's always used in our customer's setup that facilitates   matchbox-window-manager -use_titlebar no   for a kiosk-like user experience.


Patches

Our main patch, qdesktopwidget_fix_availableGeometry.diff, fixes the main error described above, in src/gui/kernel/qdesktopwidget_x11.cpp:304.

Our side patch qmessagebox_ironize_updatesize.diff makes QMessageBoxPrivate::updateSize() ignore non-positive softLimits and hardLimits.
We're not 100% sure it's a good idea to apply that. Concerns src/gui/dialogs/qmessagebox.cpp lines 345, 349, 375, 390.


Non-Patches

On the assumption that such code is often copied'n'change we grepped for other uses of XGetWindowProperty() (there are 51), quickly (not too thouroghly) checked for similar mistakes and found serveral suspicious blocks of code.
This was done after 'apt-get source qt4-x11' to *.cpp under qt4-x11-4.8.7+dfsg/src/, i.e. after debian patches have been applied to the source code. Line numbers in upstream tarball might differ.

In QETWidget::translatePropertyEvent(), 
  src/gui/kernel/qapplication_x11.cpp:5008ff 
XGetWindowProperty(_KDE_NET_WM_FRAME_STRUT) seems to make a similar mistake, but I haven't researched wether the contradiction between '// struts are 4 longs' and the check 'format == 32' is a bug or maybe a legitimate-ish hack.

Other XGetWindowProperty() calls with suspicious result processing can be found in
  src/gui/kernel/qapplication_x11.cpp:2831, which someone has thought about with // quint32, but seems not in use
  src/gui/kernel/qapplication_x11.cpp:5027
  src/gui/kernel/qapplication_x11.cpp:5106
  src/gui/kernel/qapplication_x11.cpp:5165
  src/gui/kernel/qx11embed_x11.cpp:415 
  src/gui/kernel/qx11embed_x11.cpp:829 
  src/gui/kernel/qx11embed_x11.cpp:1673