Hello Christoph
Sorry for the long response, but I was quite busy.
I looked at your source and as expected reads Filebrowser only the current folder information so there is no room for improvements in the most time-consuming part of the plugin. I did not get timeouts in some time so they could be coused by my provider. As I remember I have also moved some image folders from 2lang directory to userfiles/images folder so actually there were more images (and folders) in my cmsimple 1.6 version of the website than before.
Do you mean the filebrowser works fast when called from TinyMCE, but not when called from the admin menu?
Fortunately, there is no difference.
I have made some tests. 100 subfolders with 100 images
My image folder is a lot bigger than your test folder. But fortunately, most images are only used on 2lang site, a kind of archive of outdated information. This makes it reasonable to move unused images to 2lang folder. Those images are no longer available for Filebrowser (unless I make some hacks), but are still available for 2lang site. This solution is fine for my needs, and the only conclusion is that large number of imagess makes Filebrowser slow with the risk of timeouts and an efficient folder structure can be a part of the solution.