It is not possible to create a new custom button with its own functionality.
However, you could perhaps take advantage of the way that the 'Open Image' button works and re-appropriate it for your own requirements.
Try the following:
Set every image's linkURL to javascript: custom_function();.
Set every image's linkTarget to '_self'.
Set showOpenButton="TRUE" in your gallery's XML file.
Change the icon for the 'Open Image' button using the technique in the Using Custom Icons support section.
Change the toolip text for the 'Open Image' button using the languageList configuration option (so as not to confuse users).
When the 'Open Image' button is clicked, the custom_function() JavaScript function (which should be included in the HTML page which contains the gallery's embedding code) will be run.
(Please note that this will not work if using a baseUrl.)
Otherwise, you could create your own HTML button outside your gallery and have it interact with the gallery (if necessary) using the Juicebox-Pro API. (The Juicebox-Pro API requires Juicebox-Pro and is not available to Juicebox-Lite, the free version).