The maxThumbRows and maxThumbColumns configuration options are used only in Large Screen Mode (to determine the upper limit of how many thumbnails will appear on each page).
With maxThumbRows="11" and maxThumbColumns="3" there will be a maximum of 33 thumbnails on each page (and possibly less if the user's browser window is not large enough to accommodate this many thumbnails).
If, for example, you have 50 images in your gallery and your browser window is large enough to display a grid of 11 x 3, then there will be 33 thumbnails on the first thumbnail page and 17 on the next.
In Small Screen Mode, maxThumbRows and maxThumbColumns are not used and the thumbnails will fill whatever space is available.
So the max number in the grid is the rows x columns?
In Large Screen Mode, yes.
I would think the images would fill out the grid and not stop at the combo of rows and columns.
This is what should happen in Small Screen Mode.
If you find that things are not happening as expected, please post the URL to your gallery so that I can take a look and help further.