Interactive Swing & JavaFX rendering in notebooks

[mkhawam]

Summary

Render AWT/Swing components — and JavaFX (Node/JFXPanel) — in notebooks, both statically and interactively.

What's included

• Static snapshot. Any java.awt.Component renders as a PNG via display(component) or a bare component expression — it's painted off-screen into a BufferedImage and reuses the existing Image pipeline. Headless-safe (no native window is created).
• Interactive. displayInteractive(component) streams PNG frames of the component over a kernel-opened Jupyter comm and dispatches mouse/keyboard events back into it. Frames are coalesced (~30 fps) with duplicate-frame suppression, and a custom RepaintManager streams asynchronous repaints (animation timers, model changes).
• JavaFX, same pipeline. Pass a javafx.scene.Node (auto-wrapped) or a JFXPanel to displayInteractive(...). JavaFX is an optional, user-supplied dependency (e.g. via %maven) and is handled reflectively, so jjava-jupyter keeps no compile/link dependency on it.
• Graceful fallback. The interactive output also carries a static image/png, so frontends without the extension (or non-Lab frontends) show a still image instead of an error. Notebooks that never use Swing are unaffected — AWT is not initialized until a component is actually rendered.
• JupyterLab 4 extension under jupyterlab-extension/ (package jjava-swing): a mime renderer that mounts a <canvas> for the custom MIME type plus a notebook plugin that registers the comm target, draws streamed frames, and forwards events. Plus example notebooks for Swing and JavaFX.
• Tests (headless): click → ActionListener, typing → JTextField, frame streaming on bind, duplicate-frame skip, and the static-fallback bundle.

Design notes

• Off-screen rendering keeps Swing headless-safe.
• Keyboard is delivered via KeyboardFocusManager.redispatchEvent (Swing) and Event.fireEvent into the focused FX node (JavaFX), because synthetic key events would otherwise be swallowed off-screen (no Java window holds focus).
• A JFXPanel-containing tree is hosted in an off-screen displayable window so its input can be coordinate-mapped.
• Adds a CommManager.openComm(target, data, factory) overload so the kernel-initiated comm_open can carry a correlation token.

Limitations

Synthetic off-screen dispatch covers custom-painted panels, charts, buttons, text fields, tables, etc., but not features needing their own native windows or the real lightweight dispatcher: tooltips, popup menus, combo-box dropdowns, drag-and-drop, some focus-traversal edge cases. JavaFX additionally needs a display (or Monocle on a headless server) and modifications to the kernel.json to tell Javafx to run headless.

Try it

mvn clean package
cd jupyterlab-extension && npm install && npm run build && jupyter labextension develop --overwrite .

Then see jupyterlab-extension/examples/ (swing-demos.ipynb, javafx-demo.ipynb).

JButton b = new JButton("Click me");
b.addActionListener(e -> b.setText("clicked"));
displayInteractive(b);

[andrus]

Hi Mohamad, this looks pretty cool (though still need to try it out). Yeah, maybe we should try to digest it one piece at a time, starting with the most portable JupyterLab-independent part.

For instance, DFLib charts are currently rendered in Jupyter as HTML/JavaScript/CSS so I'd love to be able to create a JavaFX-based renderer that produces static images that e.g. can be displayed on GitHub, etc.

[mkhawam]

Per the suggestion to digest this one piece at a time, I've split out the most portable, JupyterLab-independent part into #123 — static image rendering for java.awt.Component and javafx.scene.Node (i.e. display(node) → a still PNG, viewable on GitHub/nbviewer). It has no frontend, no comms, and no dependency on JavaFX (optional/reflective). I'll keep this PR as the draft for the interactive streaming + JupyterLab extension to follow once the static core lands.

For interactive displaying however, it would be necessary for an extension to be made to juypyterlab. The extension is meant to be a client side renderer which adds some events which then get send to the swing/javafx application.