Static image rendering for Swing and JavaFX components

examples/static-image-rendering.ipynb

@@ -0,0 +1,68 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# Static image rendering (Swing & JavaFX)\n\nThis exercises PR #123: `display(component)` / `display(node)` \u2014 and a bare last-expression value \u2014\nrender a GUI component to a **still PNG**. No JupyterLab extension, no comms; the image shows in any\nfrontend and on GitHub/nbviewer.\n\n- **Swing/AWT** is headless-native (no display needed).\n- **JavaFX** is optional (pull it with `%maven`) and needs a display to snapshot \u2014 on a headless host run\n  the kernel under **Xvfb** (`xvfb-run -a jupyter lab`). Build FX nodes on the JavaFX thread (helper below).\n\nRebuild the kernel from the `static-image-rendering` branch and reinstall the kernelspec before running."
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Swing / AWT"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "import javax.swing.*;\nimport java.awt.*;\n\n// A laid-out Swing component renders as a static image.\nJButton button = new JButton(\"Hello from Swing\");\nbutton.setSize(220, 48);\ndisplay(button);"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "import javax.swing.*;\nimport java.awt.*;\n\n// Custom painting also works - here a tiny bar chart.\nJPanel chart = new JPanel() {\n    protected void paintComponent(Graphics g) {\n        super.paintComponent(g);\n        Graphics2D g2 = (Graphics2D) g;\n        g2.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);\n        int[] vals = {40, 90, 60, 120, 80};\n        int w = getWidth() / vals.length;\n        for (int i = 0; i < vals.length; i++) {\n            g2.setColor(Color.getHSBColor(i / (float) vals.length, 0.6f, 0.9f));\n            g2.fillRect(i * w + 8, getHeight() - vals[i] - 10, w - 16, vals[i]);\n        }\n        g2.setColor(Color.DARK_GRAY);\n        g2.drawString(\"Swing bar chart\", 8, 16);\n    }\n};\nchart.setBackground(Color.WHITE);\nchart.setSize(360, 180);\ndisplay(chart);"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## JavaFX (optional)\n\nPull OpenJFX (edit the classifier for your OS: `linux` | `mac` | `mac-aarch64` | `win`). JavaFX needs a\ndisplay; on a headless host run the kernel under Xvfb.\n\nAll JavaFX imports + the `fx(...)` helper live in the setup cell. The cells below pass the node **straight to\n`display(...)`** (no intermediate variable) \u2014 storing a JavaFX-typed top-level variable trips a JShell\nfield-resolution bug for runtime-loaded types."
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%maven org.openjfx:javafx-base:jar:linux:21.0.2\n%maven org.openjfx:javafx-graphics:jar:linux:21.0.2\n%maven org.openjfx:javafx-controls:jar:linux:21.0.2\n%maven org.openjfx:javafx-swing:jar:linux:21.0.2\n\nimport javafx.application.Platform;\nimport javafx.embed.swing.JFXPanel;\nimport javafx.scene.layout.Pane;\nimport javafx.scene.shape.Circle;\nimport javafx.scene.shape.Rectangle;\nimport javafx.scene.paint.Color;\nimport javafx.scene.chart.BarChart;\nimport javafx.scene.chart.CategoryAxis;\nimport javafx.scene.chart.NumberAxis;\nimport javafx.scene.chart.XYChart;\nimport java.util.concurrent.CountDownLatch;\nimport java.util.function.Supplier;\n\nnew JFXPanel(); // boots the JavaFX toolkit (needs a display / Xvfb)\n\n// Build a JavaFX node on the FX thread and return it (as Object - do not store it in a typed var).\nObject fx(Supplier<Object> builder) throws Exception {\n    Object[] holder = new Object[1];\n    CountDownLatch latch = new CountDownLatch(1);\n    Platform.runLater(() -> { try { holder[0] = builder.get(); } finally { latch.countDown(); } });\n    latch.await();\n    return holder[0];\n}\nSystem.out.println(\"JavaFX \" + System.getProperty(\"javafx.runtime.version\") + \" ready\");"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "// Pass the node straight to display(...) - no intermediate variable.\ndisplay(fx(() -> {\n    Pane p = new Pane();\n    p.setPrefSize(300, 200);\n    Rectangle r = new Rectangle(40, 50, 90, 90);\n    r.setFill(Color.ORANGE);\n    Circle c = new Circle(170, 100, 70, Color.CORNFLOWERBLUE);\n    p.getChildren().addAll(r, c);\n    return p;\n}));"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "display(fx(() -> {\n    BarChart<String, Number> bc = new BarChart<>(new CategoryAxis(), new NumberAxis());\n    bc.setTitle(\"JavaFX BarChart\");\n    XYChart.Series<String, Number> s = new XYChart.Series<>();\n    s.setName(\"demo\");\n    s.getData().add(new XYChart.Data<>(\"A\", 40));\n    s.getData().add(new XYChart.Data<>(\"B\", 90));\n    s.getData().add(new XYChart.Data<>(\"C\", 60));\n    bc.getData().add(s);\n    bc.setPrefSize(440, 300);\n    return bc;\n}));"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Java",
+   "language": "java",
+   "name": "java"
+  },
+  "language_info": {
+   "name": "java",
+   "file_extension": ".jshell",
+   "mimetype": "text/x-java-source"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jjava-jupyter/src/main/java/org/dflib/jjava/jupyter/kernel/BaseKernel.java

@@ -7,6 +7,8 @@
 import org.dflib.jjava.jupyter.kernel.display.DisplayData;
 import org.dflib.jjava.jupyter.kernel.display.Renderer;
 import org.dflib.jjava.jupyter.kernel.display.common.Image;
+import org.dflib.jjava.jupyter.kernel.display.common.JavaFx;
+import org.dflib.jjava.jupyter.kernel.display.common.Swing;
 import org.dflib.jjava.jupyter.kernel.display.common.Text;
 import org.dflib.jjava.jupyter.kernel.display.common.Url;
 import org.dflib.jjava.jupyter.kernel.history.HistoryEntry;
@@ -136,6 +138,8 @@ protected BaseKernel(
         this.executionCount = new AtomicInteger(1);
 
         Image.registerAll(this.renderer);
+        Swing.registerAll(this.renderer);
+        JavaFx.registerAll(this.renderer);
         Url.registerAll(this.renderer);
         Text.registerAll(this.renderer);
     }

jjava-jupyter/src/main/java/org/dflib/jjava/jupyter/kernel/display/Renderer.java

@@ -56,12 +56,23 @@ public class RenderRegistration<T> {
         private final Set<MIMEType> supported;
         private final Set<MIMEType> preferred;
         private final Set<Class<? extends T>> types;
+        private final Set<String> typeNames;
 
         public RenderRegistration(Class<? extends T> type) {
+            this();
+            this.types.add(type);
+        }
+
+        public RenderRegistration(String typeName) {
+            this();
+            this.typeNames.add(typeName);
+        }
+
+        private RenderRegistration() {
             this.supported = new LinkedHashSet<>();
             this.preferred = new LinkedHashSet<>();
             this.types = new LinkedHashSet<>();
-            this.types.add(type);
+            this.typeNames = new LinkedHashSet<>();
         }
 
         public RenderRegistration<T> supporting(MIMEType... types) {
@@ -96,22 +107,36 @@ public RenderRegistration<T> onType(Class<? extends T> type) {
         public void register(RenderFunction<T> function) {
             Set<MIMEType> supported = this.supported.isEmpty() ? DisplayDataRenderable.ANY : this.supported;
             Set<MIMEType> preferred = this.preferred.isEmpty() ? supported : this.preferred;
-            Renderer.this.register(supported, preferred, types, function);
+            if (!this.types.isEmpty())
+                Renderer.this.register(supported, preferred, types, function);
+            if (!this.typeNames.isEmpty())
+                Renderer.this.registerByName(supported, preferred, typeNames, function);
         }
     }
 
     private final Map<Class, List<RenderFunctionProps>> renderFunctions;
+    private final Map<String, List<RenderFunctionProps>> renderFunctionsByName;
     private final Map<String, MIMEType> suffixMappings;
 
     public Renderer() {
         this.renderFunctions = new HashMap<>();
+        this.renderFunctionsByName = new HashMap<>();
         this.suffixMappings = new HashMap<>();
     }
 
     public <T> RenderRegistration<T> createRegistration(Class<T> type) {
         return new RenderRegistration<>(type);
     }
 
+    /**
+     * Create a registration keyed by a fully-qualified class name rather than a {@link Class} instance. This matches a
+     * rendered value when any class or interface in its hierarchy has this name, regardless of which class loader loaded
+     * it — useful for optional, runtime-supplied types (e.g. JavaFX) that are not on the kernel's own class path.
+     */
+    public RenderRegistration<Object> createRegistration(String typeName) {
+        return new RenderRegistration<>(typeName);
+    }
+
     public <T> void register(Set<MIMEType> supported, Set<MIMEType> preferred, Set<Class<? extends T>> types, RenderFunction<T> function) {
         RenderFunctionProps props = new RenderFunctionProps(function, supported, preferred);
 
@@ -122,6 +147,21 @@ public <T> void register(Set<MIMEType> supported, Set<MIMEType> preferred, Set<C
         }));
     }
 
+    public <T> void registerByName(Set<MIMEType> supported, Set<MIMEType> preferred, Set<String> typeNames, RenderFunction<T> function) {
+        RenderFunctionProps props = new RenderFunctionProps(function, supported, preferred);
+
+        typeNames.forEach(name -> this.renderFunctionsByName.compute(name, (k, v) -> {
+            List<RenderFunctionProps> functions = v != null ? v : new ArrayList<>();
+            functions.add(props);
+            return functions;
+        }));
+    }
+
+    private List<RenderFunctionProps> lookup(Class type) {
+        List<RenderFunctionProps> byClass = this.renderFunctions.get(type);
+        return byClass != null ? byClass : this.renderFunctionsByName.get(type.getName());
+    }
+
     protected DisplayData finalizeDisplayData(DisplayData data, Object value) {
         if (!data.hasDataForType(MIMEType.TEXT_PLAIN))
             data.putText(String.valueOf(value));
@@ -171,7 +211,7 @@ public DisplayData render(Object value, Map<String, Object> params) {
         while (inheritedTypes.hasNext()) {
             Class type = inheritedTypes.next();
 
-            List<RenderFunctionProps> allRenderFunctionProps = this.renderFunctions.get(type);
+            List<RenderFunctionProps> allRenderFunctionProps = this.lookup(type);
             if (allRenderFunctionProps != null && !allRenderFunctionProps.isEmpty()) {
                 for (RenderFunctionProps renderFunctionProps : allRenderFunctionProps) {
                     RenderRequestTypes.Builder requestTypes = new RenderRequestTypes.Builder(this.suffixMappings::get);
@@ -257,7 +297,7 @@ public DisplayData renderAs(Object value, Map<String, Object> params, String...
         Iterator<Class> inheritedTypes = new InheritanceIterator(value.getClass());
         while (inheritedTypes.hasNext() && !requestTypes.isEmpty()) {
             Class type = inheritedTypes.next();
-            List<RenderFunctionProps> allRenderFunctionProps = this.renderFunctions.get(type);
+            List<RenderFunctionProps> allRenderFunctionProps = this.lookup(type);
             if (allRenderFunctionProps != null) {
                 for (RenderFunctionProps renderFunctionProps : allRenderFunctionProps) {
                     if (requestTypes.anyRequestedIsSupported(renderFunctionProps.getSupportedTypes())) {

jjava-jupyter/src/main/java/org/dflib/jjava/jupyter/kernel/display/common/JavaFx.java

@@ -0,0 +1,108 @@
+package org.dflib.jjava.jupyter.kernel.display.common;
+
+import org.dflib.jjava.jupyter.kernel.display.RenderContext;
+import org.dflib.jjava.jupyter.kernel.display.Renderer;
+import org.dflib.jjava.jupyter.kernel.display.mime.MIMEType;
+
+import java.awt.image.BufferedImage;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Renders a JavaFX {@code javafx.scene.Node} as a static image: it snapshots the node on the JavaFX Application Thread
+ * and converts the result to a {@link BufferedImage}, which is then encoded by {@link Image}. So {@code display(node)}
+ * (or a bare node expression) yields a still PNG that works anywhere a static image does (GitHub, nbviewer, ...).
+ *
+ * <p>Everything is done reflectively, so jjava-jupyter keeps <strong>no compile- or link-time dependency on
+ * JavaFX</strong>: JavaFX is optional and user-supplied (e.g. {@code %maven org.openjfx:javafx-controls:...}). The
+ * renderer is registered <strong>by class name</strong> so it matches user nodes regardless of which class loader loaded
+ * them.
+ *
+ * <p>Producing a snapshot starts the JavaFX toolkit, which needs a display; on a headless host run under Xvfb. If the
+ * toolkit can't start, rendering degrades to a short text message instead of failing the cell.
+ */
+public class JavaFx {
+
+    public static final String NODE_CLASS = "javafx.scene.Node";
+
+    private static final long SNAPSHOT_TIMEOUT_SECONDS = 15;
+
+    public static void registerAll(Renderer renderer) {
+        renderer.createRegistration(NODE_CLASS)
+                .preferring(Image.PNG)
+                .supporting(Image.JPEG)
+                .register(JavaFx::renderNode);
+    }
+
+    public static void renderNode(Object node, RenderContext context) {
+        try {
+            Image.renderImage(snapshot(node), context);
+        } catch (Throwable t) {
+            context.renderIfRequested(MIMEType.TEXT_PLAIN,
+                    () -> "Could not render JavaFX node as an image: " + rootMessage(t)
+                            + " (JavaFX needs a display; on a headless host run the kernel under Xvfb).");
+        }
+    }
+
+    private static BufferedImage snapshot(Object node) throws Exception {
+        ClassLoader cl = node.getClass().getClassLoader();
+        Class<?> platformClass = Class.forName("javafx.application.Platform", true, cl);
+        Class<?> snapshotParamsClass = Class.forName("javafx.scene.SnapshotParameters", true, cl);
+        Class<?> writableImageClass = Class.forName("javafx.scene.image.WritableImage", true, cl);
+        Class<?> imageClass = Class.forName("javafx.scene.image.Image", true, cl);
+        Class<?> swingFxUtilsClass = Class.forName("javafx.embed.swing.SwingFXUtils", true, cl);
+
+        startToolkit(platformClass);
+
+        Method runLater = platformClass.getMethod("runLater", Runnable.class);
+        Method snapshotMethod = node.getClass().getMethod("snapshot", snapshotParamsClass, writableImageClass);
+        Method fromFXImage = swingFxUtilsClass.getMethod("fromFXImage", imageClass, BufferedImage.class);
+
+        AtomicReference<BufferedImage> result = new AtomicReference<>();
+        AtomicReference<Throwable> failure = new AtomicReference<>();
+        CountDownLatch done = new CountDownLatch(1);
+
+        // snapshot() must run on the JavaFX Application Thread.
+        runLater.invoke(null, (Runnable) () -> {
+            try {
+                Object params = snapshotParamsClass.getConstructor().newInstance();
+                Object fxImage = snapshotMethod.invoke(node, params, null);
+                result.set((BufferedImage) fromFXImage.invoke(null, fxImage, null));
+            } catch (Throwable t) {
+                failure.set(t);
+            } finally {
+                done.countDown();
+            }
+        });
+
+        if (!done.await(SNAPSHOT_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {
+            throw new IllegalStateException("Timed out waiting for the JavaFX snapshot (is a display available?)");
+        }
+        if (failure.get() != null) {
+            throw new RuntimeException(failure.get());
+        }
+        return result.get();
+    }
+
+    private static void startToolkit(Class<?> platformClass) throws Exception {
+        try {
+            platformClass.getMethod("startup", Runnable.class).invoke(null, (Runnable) () -> { });
+        } catch (InvocationTargetException e) {
+            // IllegalStateException means the toolkit is already running, which is fine; anything else is fatal.
+            if (!(e.getCause() instanceof IllegalStateException)) {
+                throw e;
+            }
+        }
+    }
+
+    private static String rootMessage(Throwable t) {
+        Throwable cause = t;
+        while (cause.getCause() != null && cause.getCause() != cause) {
+            cause = cause.getCause();
+        }
+        return cause.getMessage() != null ? cause.getMessage() : cause.getClass().getSimpleName();
+    }
+}

jjava-jupyter/src/main/java/org/dflib/jjava/jupyter/kernel/display/common/Swing.java

@@ -0,0 +1,82 @@
+package org.dflib.jjava.jupyter.kernel.display.common;
+
+import org.dflib.jjava.jupyter.kernel.display.RenderContext;
+import org.dflib.jjava.jupyter.kernel.display.Renderer;
+import org.dflib.jjava.jupyter.kernel.display.mime.MIMEType;
+
+import java.awt.Component;
+import java.awt.Container;
+import java.awt.Dimension;
+import java.awt.Graphics2D;
+import java.awt.RenderingHints;
+import java.awt.image.BufferedImage;
+
+/**
+ * Renders an {@link java.awt.Component AWT/Swing component} as an image by painting it off-screen into a
+ * {@link BufferedImage}, which is then encoded by {@link Image}. This works without a display (headless) since no native
+ * window is ever created.
+ */
+public class Swing {
+    public static final MIMEType PNG = MIMEType.IMAGE_PNG;
+    public static final MIMEType JPEG = MIMEType.IMAGE_JPEG;
+    public static final MIMEType GIF = MIMEType.IMAGE_GIF;
+
+    public static void registerAll(Renderer renderer) {
+        renderer.createRegistration(Component.class)
+                .preferring(PNG)
+                .supporting(JPEG, GIF)
+                .register(Swing::renderComponent);
+    }
+
+    public static void renderComponent(Component component, RenderContext context) {
+        Image.renderImage(snapshot(component), context);
+    }
+
+    /**
+     * Paint a component using its current size, falling back to its {@link Component#getPreferredSize() preferred size}
+     * when it has not been laid out yet (which is the usual case for a component that was never added to a window).
+     */
+    public static BufferedImage snapshot(Component component) {
+        Dimension size = component.getSize();
+        if (size.width <= 0 || size.height <= 0) {
+            size = component.getPreferredSize();
+        }
+        return snapshot(component, size.width, size.height);
+    }
+
+    public static BufferedImage snapshot(Component component, int width, int height) {
+        width = Math.max(width, 1);
+        height = Math.max(height, 1);
+
+        component.setSize(width, height);
+        layoutTree(component);
+
+        BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
+        Graphics2D g = image.createGraphics();
+        try {
+            g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
+            g.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
+            // print() rather than paint(): it bypasses Swing's double-buffering, which is the right behaviour for
+            // off-screen rendering of a component that has no native peer.
+            component.print(g);
+        } finally {
+            g.dispose();
+        }
+        return image;
+    }
+
+    /**
+     * A component that was never realized in a window is not laid out, so its children have zero bounds and would not
+     * paint (nor hit-test). Forcing a layout pass over the whole subtree gives every child a position and size.
+     */
+    public static void layoutTree(Component component) {
+        synchronized (component.getTreeLock()) {
+            component.doLayout();
+            if (component instanceof Container) {
+                for (Component child : ((Container) component).getComponents()) {
+                    layoutTree(child);
+                }
+            }
+        }
+    }
+}