The new VGBAnext 5.5.11, published today, fixes a bunch of issues in the Joystick Layout Editor. Among other things, the new Layout Editor will let you freely resize the display rectangle in both dimensions, something users requested. Do note that if you make that rectangle too high, the actual GameBoy screen will show up at the top of the rectangle. See below for all the changes.
Greatly improved the Layout Editor, fixing many bugs.
Layout Editor now lets you resize screen in both dimensions.
Screen rectangle will stay centered while resizing.
Buttons gravity will be restored when resetting layout.
Made states/screenshots deletion more reliable.
Fixed hangup on exit where Java code waits for non-existent native thread.