Help taking a screencapture in Java. It used to work

[darwerft]

I develop a Java application that must be able to see the screen for automation purposes.
I used to use the Robot Class in java to take screenshots and look for pixel Colors, however it stopped working at some point. I googled around for an hour and only found random comments speculating that it’s a security feature of Wayland or Xorg.

However I know an alternative must exist becuase other applications (Like Spectacle and Discord) are clearly still able to take screenshots. Does anyone know an alternative to the Robot Class in Java or have more information regarding this issue?

It is especially annoying since it just outright crashes the application so it took some time to debug the issue by slowly removing code until I found the guilty function call.

I use java-14-openjdk and Intellij idea and call createScreenCapture function on a Robot object, that should make it reproduceable. It is possible that it is a Bug in java-14-openjdk, please let me know and I will remove the post. Since java 14 is supported by Arch I thought it might also be the case for Manjaro, please correct me if im wrong.

[darwerft]

To any other poor soul that runs into this issue the only work around I could find was simulating pressing the print screen button and reading from clipboard, in kde this function works with spectacle to achieve this.

private BufferedImage getScreenCapture() {
    robot.keyPress(VK_SHIFT);
    robot.delay(40);
    robot.keyPress(VK_PRINTSCREEN);
    robot.delay(40);
    robot.keyRelease(VK_PRINTSCREEN);
    robot.delay(40);
    robot.keyRelease(VK_SHIFT);
    robot.delay(40);

    Transferable transferable = Toolkit.getDefaultToolkit().getSystemClipboard().getContents(null);
    try {
        return (BufferedImage) (Image) transferable.getTransferData(DataFlavor.imageFlavor);
    } catch (UnsupportedFlavorException | IOException e) {
        e.printStackTrace();
        try {
            Thread.sleep(1000);
        } catch (InterruptedException interruptedException) {
            interruptedException.printStackTrace();
        }
        //We try again
        return getScreenCapture();
    }
    //TODO: Enable when bug is fixed.
    //robot.createScreenCapture(new Rectangle(0, 0, screenSize.width, screenSize.height));
}

But really it sucks as it is much slower, polutes the clipboard and sends notifcations to the host that screenshot was taken.

Still very open to other solutions.

[darwerft]

Today I discovered that I wasnt using the javafx robot but the awt robot. Switching over to the javafx robot completely solved the problem. There was however some conversion work as the Colors are not the same in javafx and java awt.

It is interesting that some awt Robot functions like mouse/keyboard hooks work fine while others like the one above dont (however it used to work!!). I really wish someone could have saved me these hours of troubleshooting.

Hopefully someone will find these posts useful and I can save someone elses time.