Class Screens

java.lang.Object

uno.anahata.asi.agi.tool.ToolContext

uno.anahata.asi.agi.tool.AnahataToolkit

uno.anahata.asi.swing.toolkit.Screens

All Implemented Interfaces:

ContextProvider, PropertyChangeSource, Rebindable

public class Screens
extends AnahataToolkit

A hardware-aware toolkit for capturing high-fidelity screenshots of the host system's displays and individual application windows.

This toolkit leverages the UICapture utility to perform native screen scraping. It is primarily used by the ASI to "see" the user's current workspace or specific application states.

Author:

anahata

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Screens.SharedRegion	Represents a shared rectangular region of the screen.

Field Summary

Fields

Modifier and Type	Field	Description
private final List<Integer>	sharedDeviceIndexes	The list of physical device indexes currently being shared in multimodal turns.
private final List<Screens.SharedRegion>	sharedRegions	The list of rectangular regions currently being shared in multimodal turns.

Fields inherited from class AnahataToolkit

childrenProviders, propertyChangeSupport

Modifier and Type	Field	Description
protected List<ContextProvider>	childrenProviders	The list of child context providers managed by this toolkit.
protected PropertyChangeSupport	propertyChangeSupport	Support for firing property change events.

Fields inherited from class ToolContext

toolkit

Modifier and Type	Field	Description
protected JavaObjectToolkit	toolkit	The toolkit instance that owns this context.

Constructor Summary

Constructors

Constructor	Description
Screens()

Method Summary

Modifier and Type	Method	Description
String	captureRegion(int x, int y, int width, int height)	Captures a specific region of the primary screen.
int	getSharedCount()	Gets the total number of items currently being shared (displays plus custom regions).
List<String>	getSystemInstructions()	Gets a list of system instruction strings provided by this context provider.
void	populateMessage(RagMessage ragMessage)	Populates the given RagMessage with dynamic, just-in-time context parts.
List<String>	screenshotAllWindows()	Orchestrates a bulk capture of all visible application windows currently managed by the host's window manager.
String	startSharingRegion(int x, int y, int w, int h, String name)	Adds a specific rectangular region to the live share.
String	stopSharingRegion(String regionId)	Stops sharing a specific region by its ID.
String	takeScreenshot(int deviceIdx)	Captures a screenshot of a specific physical display device identified by its index.
String	toggleDeviceSharing(int deviceIdx)	Toggles sharing for a specific screen device.

Methods inherited from class AnahataToolkit

getChildrenProviders, getDescription, getId, getName, getParentProvider, getPropertyChangeSupport, initialize, isEffectivelyProviding, postActivate, rebind

Modifier and Type	Method	Description
List<ContextProvider>	getChildrenProviders()	Gets the list of immediate child context providers.
String	getDescription()	Gets a detailed description of what this context provider does.
String	getId()	Gets the unique identifier for this context provider.
String	getName()	Gets the human-readable name of this context provider.
ContextProvider	getParentProvider()	Gets the parent context provider in the hierarchy, if any.
PropertyChangeSupport	getPropertyChangeSupport()	Gets the PropertyChangeSupport instance for this object.
void	initialize()	Performs initial setup for a newly created toolkit instance.
boolean	isEffectivelyProviding()	Override to also take into consideration the AbstractToolkit.enabled
void	postActivate()	Callback method triggered by the Asi Container once the Agi session has been fully deserialized and associated to the ASI container.
void	rebind()	Rebinds the toolkit after deserialization.

Methods inherited from class ToolContext

addAttachment, addAttachment, addAttachment, addAttachment, error, error, error, getAgi, getApplicationMap, getAsiContainer, getAsiContainerMap, getCall, getExecutorService, getModelId, getModelMessage, getResourceManager, getResponse, getSessionMap, getThreadSafeLogger, getTool, getToolContext, getToolkit, getToolManager, getTurnMap, isToolExecution, log, peekResponse, runAsync

Modifier and Type	Method	Description
void	addAttachment(byte[] data, String mimeType)	Attaches a binary blob to the current tool's response.
void	addAttachment(byte[] data, String mimeType, Path sourcePath)	Attaches a binary blob to the current tool's response with source file path traceability.
void	addAttachment(File file)	Convenience method to attach a file to the current tool's response.
void	addAttachment(Path path)	Convenience method to attach a file from a Path to the current tool's response.
void	error(String message)	Adds an error message to the current tool's response.
void	error(String message, Throwable t)	Adds an error message and exception stack trace to the current tool's response.
void	error(Throwable t)	Adds an exceptions stack trace to the current tool's response.
Agi	getAgi()	Convenience method to get the parent agi container for the current execution.
Map	getApplicationMap()	Gets a static, application-scoped map for sharing objects across all containers in the current JVM.
AbstractAsiContainer	getAsiContainer()	Gets the current ASI container hosting this Agi container is running.
Map	getAsiContainerMap()	Gets a map for sharing objects across all sessions within the current ASI container.
JavaMethodToolCall	getCall()	Convenience method to get the current tool call that initiated this execution.
ExecutorService	getExecutorService()	Convenience method to get the executor service for the current agi session.
String	getModelId()	The id of model that is executing the tool.
AbstractModelMessage	getModelMessage()	The id of model that is executing the tool.
ResourceManager	getResourceManager()	Convenience method to get the application's ResourceManager.
JavaMethodToolResponse	getResponse()	Gets the current tool response from the thread-local context.
Map	getSessionMap()	Gets a shared map for storing state across turns within the current agi session.
Consumer<String>	getThreadSafeLogger()	Gets a thread-safe logging consumer that automatically propagates the current tool execution context whenever a message is logged.
JavaMethodTool	getTool()	Convenience method to get the metadata for the tool currently being executed.
ToolContext	getToolContext()	Returns a thread-safe snapshot of this context.
<T> T	getToolkit(Class<T> toolkitClass)	Retrieves the singleton instance of a registered toolkit class.
ToolManager	getToolManager()	Convenience method to get the application's global ToolManager.
Map	getTurnMap()	Gets a shared map for storing state across turns within the current agi session.
boolean	isToolExecution()	Checks if the current thread or this context instance is associated with an active tool execution.
void	log(String message)	Adds a standard log message to the current tool's response.
protected JavaMethodToolResponse	peekResponse()	Internal helper to retrieve the response if it exists, without throwing an exception.
void	runAsync(String taskName, Runnable task)	Executes a task asynchronously in the session's executor service, automatically propagating the current tool execution context to the background thread.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface ContextProvider

getAgi, getContextPosition, getFlattenedHierarchy, getFullyQualifiedId, getHeader, getIconId, getInstructionsTokenCount, getRagTokenCount, isProviding, setParentProvider, setProviding

Modifier and Type	Method	Description
Agi	getAgi()	Gets the parent AGI session this context provider is registered under.
default ContextPosition	getContextPosition()	Gets the designated position of this provider's content in the AI prompt.
default List<ContextProvider>	getFlattenedHierarchy(boolean providingOnly)	Gets a flattened list of this provider and all its descendants in the hierarchy.
default String	getFullyQualifiedId()	Gets the fully qualified ID of this provider, reflecting its position in the hierarchy.
default String	getHeader()	Generates a machine-readable header string for this context provider, used to identify the source of injected context in the final prompt.
default String	getIconId()	Gets an optional icon identifier for this provider.
default int	getInstructionsTokenCount()	Calculates the token count for the system instructions provided by this instance using the active model.
default int	getRagTokenCount()	Calculates the token count for the RAG content provided by this instance using the active model.
boolean	isProviding()	Checks if this context provider is currently active and providing context.
default void	setParentProvider(ContextProvider parent)	Sets the parent context provider for this instance.
default void	setProviding(boolean enabled)	Sets whether this context provider is enabled.

Field Details

sharedDeviceIndexes

private final List<Integer> sharedDeviceIndexes

The list of physical device indexes currently being shared in multimodal turns.

sharedRegions

private final List<Screens.SharedRegion> sharedRegions

The list of rectangular regions currently being shared in multimodal turns.

Constructor Details

Screens

public Screens()

Method Details

getSharedCount

public int getSharedCount()

Gets the total number of items currently being shared (displays plus custom regions).

Returns:

The total count of shared displays and regions.

getSystemInstructions

public List<String> getSystemInstructions()

Gets a list of system instruction strings provided by this context provider. These are typically prepended to the conversation as high-level guidance.

Returns:

A list of system instruction strings.

populateMessage

public void populateMessage(RagMessage ragMessage)

Populates the given RagMessage with dynamic, just-in-time context parts. These parts are appended to the end of the user's prompt (RAG).

Parameters:

ragMessage - The message to be augmented with context.

toggleDeviceSharing

public String toggleDeviceSharing(int deviceIdx)

Toggles sharing for a specific screen device.

Parameters:

deviceIdx - The index of the device.

Returns:

A status message.

startSharingRegion

public String startSharingRegion(int x,
 int y,
 int w,
 int h,
 String name)

Adds a specific rectangular region to the live share.

Parameters:

x - X coordinate.

y - Y coordinate.

w - Width.

h - Height.

name - Optional name for the region.

Returns:

A status message with the ID.

stopSharingRegion

public String stopSharingRegion(String regionId)

Stops sharing a specific region by its ID.

Parameters:

regionId - The UUID of the shared region.

Returns:

A status message.

takeScreenshot

public String takeScreenshot(int deviceIdx)
                      throws Exception

Captures a screenshot of a specific physical display device identified by its index.

The resulting image is automatically added as an attachment to the current tool response, making it immediately available to the model's vision system.

Parameters:

deviceIdx - The 0-based index of the graphics device (0 is usually the primary display).

Returns:

A descriptive status message confirming the capture and attachment.

Throws:

IOException - if the native capture operation fails.

Exception

screenshotAllWindows

public List<String> screenshotAllWindows()
                                  throws Exception

Orchestrates a bulk capture of all visible application windows currently managed by the host's window manager.

Each window is captured as a separate image and attached to the tool response. This is particularly useful for multi-window discovery tasks.

Returns:

A status message indicating the total number of windows captured.

Throws:

Exception - if the window enumeration or capture fails.

captureRegion

public String captureRegion(int x,
 int y,
 int width,
 int height)
                     throws Exception

Captures a specific region of the primary screen.

Parameters:

x - X coordinate of the top-left corner.

y - Y coordinate of the top-left corner.

width - Width of the region.

height - Height of the region.

Returns:

A status message with the file path.

Throws:

Exception - if the native capture operation fails.