diogo/aura_chi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

jtasso review

Browse Source
This commit is contained in:
Diogo Peralta Cordeiro 2025-04-09 03:39:18 +01:00
parent cac5284129
commit 899716319e
Signed by: diogo
GPG Key ID: 18D2D35001FBFAB0
2 changed files with 81 additions and 98 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
figures/bootstrap.drawio.svg
View File

File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 81 KiB

After

Width:  |  Height:  |  Size: 82 KiB

177
main.tex
View File

@ -132,7 +132,8 @@ We introduce AURA --- the \textbf{A}ugmented Reality \textbf{U}nified \textbf{R}
%% Keywords that describe your work. Will show as 'Index Terms' in journal
%% please capitalize first letter and insert punctuation after last keyword.
\keywords{Ubiquitous computing, multi-application interoperability, interactive spaces, spatial computing, augmented reality, application manifest, human-machine interaction, interaction design.}
\keywords{Human-centered computing --- Human computer interaction (HCI) --- Interaction paradigms --- Mixed / augmented reality;
Human-centered computing --- Ubiquitous and mobile computing --- Ubiquitous computing}
%% Copyright space is enabled by default as required by guidelines.
%% It is disabled by the 'review' option or via the following command:
@ -194,7 +195,7 @@ AURA builds upon concepts from software architecture, human-computer interaction
\subsection{Entity Component System Architecture in AR}
The Entity-Component-System (ECS) architecture separates data (components) from logic (systems), enabling modular, efficient representation of interactive environments. This pattern has seen widespread adoption in game development and is increasingly embraced in AR engines. For instance, Apple’s RealityKit employs an ECS-based design, where developers define entities and attach components (e.g., geometries, anchors, physics bodies) to them, with systems processing entities based on their components \cite{AppleRealityKitChapter9}. Similarly, Unity’s DOTS ECS framework improves runtime performance by processing large numbers of entities using data-oriented memory access patterns, including in AR/VR contexts \cite{UnityECSConcepts}.
The Entity-Component-System (ECS) architecture separates data (components) from logic (systems), enabling modular, efficient representation of interactive environments. This pattern has seen widespread adoption in game development and is increasingly embraced in AR engines. For instance, Apple's RealityKit employs an ECS-based design, where developers define entities and attach components (e.g., geometries, anchors, physics bodies) to them, with systems processing entities based on their components \cite{AppleRealityKitChapter9}. Similarly, Unity's DOTS ECS framework improves runtime performance by processing large numbers of entities using data-oriented memory access patterns, including in AR/VR contexts \cite{UnityECSConcepts}.
AURA draws on ECS principles to structure declarative AR applications in a runtime-independent way. Each AURA Entity represents a logical construct (e.g., a ``PhotoWall''), while Components encapsulate spatial or functional capabilities such as display surfaces, coordinate systems, or I/O devices. Unlike engine-level ECS systems, where entities and components exist in runtime code, AURA's ECS model is expressed declaratively through JSON-LD manifests. The runtime acts as the system layer: it matches component requirements to physical resources and mediates interactions, enabling applications to remain engine-neutral and spatially adaptive.
@ -236,16 +237,6 @@ In contrast, AURA introduces a runtime mediator that interprets resource require
\caption{The AURA bootstrap process. Applications declare abstract requirements, the system proposes compatible components, user feedback finalizes mappings (and may suggest that certain components are used for certain entities), application acknowledges the ambient announcing final mappings, and runtime execution begins.}\label{fig:aura_bootstrap}
\end{figure*}
\begin{figure*}
\centering
\includesvg[inkscapelatex=false, width=\linewidth]{figures/sarm_basic_storage_concept.drawio.svg}
\caption{The System Manifest defines Worlds ($\omega_1$, $\omega_2$) and their associated Ambients ($\alpha_1..\alpha_6$). Each Application is tied to a unique Ambient (1:1 relationship). When an Application (e.g., $A_1$) accesses a Component's storage, the associated Ambient (e.g., $\alpha_1$) mediates access to the correct scoped storage view (e.g., $C_1$'s $\omega_1$ data). Thus, $A_1$, $A_2$, and $A_3$ share the same view of $C_1$, while $A_4$ and $A_5$/$A_6$ operate over isolated data contexts.}\label{fig:shared_memory}
\end{figure*}
AURA provides a structured and extensible runtime model for AR systems by decoupling application behaviour from system-level resource management. It defines a declarative interface through which applications describe, among other aspects, their structure (via entities), spatial requirements (via components), and interaction patterns (via heuristics), while the system governs spatial safety, coordination, and execution. This section formalizes the core abstractions in AURA, presents the manifest structure, and details runtime semantics including spatial bootstrapping, component-based memory, and cross-application communication strategy.
\subsection{Fundamental Concepts and Vocabulary}\label{sec:core-concepts}
\begin{table*}
\caption{Comparison Between the Application and System Manifests.}\label{tab:manifest_comparison}
\begin{tabularx}{\textwidth}{l*{2}{X}}
@ -265,6 +256,16 @@ AURA provides a structured and extensible runtime model for AR systems by decoup
\end{tabularx}
\end{table*}
\begin{figure*}
\centering
\includesvg[inkscapelatex=false, width=\linewidth]{figures/sarm_basic_storage_concept.drawio.svg}
\caption{The System Manifest defines Worlds ($\omega_1$, $\omega_2$) and their associated Ambients ($\alpha_1..\alpha_6$). Each Application is tied to a unique Ambient (1:1 relationship). When an Application (e.g., $A_1$) accesses a Component's storage, the associated Ambient (e.g., $\alpha_1$) mediates access to the correct scoped storage view (e.g., $C_1$'s $\omega_1$ data). Thus, $A_1$, $A_2$, and $A_3$ share the same view of $C_1$, while $A_4$ and $A_5$/$A_6$ operate over isolated data contexts.}\label{fig:shared_memory}
\end{figure*}
AURA provides a structured and extensible runtime model for AR systems by decoupling application behaviour from system-level resource management. It defines a declarative interface through which applications describe, among other aspects, their structure (via entities), spatial requirements (via components), and interaction patterns (via heuristics), while the system governs spatial safety, coordination, and execution. This section formalizes the core abstractions in AURA, presents the manifest structure, and details runtime semantics including spatial bootstrapping, component-based memory, and cross-application communication strategy.
\subsection{Fundamental Concepts and Vocabulary}\label{sec:core-concepts}
AURA establishes a declarative and runtime-coordinated interface between applications and spatial environments, separating responsibilities between the application and the system. Each application must provide a static \textbf{application manifest} describing its structure, requirements, and expected behaviour. While the runtime does not strictly require it, we recommend that implementations of AURA maintain a dynamically generated and persistable \textbf{system manifest}, which reflects the physical environment and current component configuration. This supports introspection, debugging, and reproducibility across sessions. \Cref{tab:manifest_comparison} outlines the key differences between both manifest types.
AURA introduces a shared vocabulary to describe the relationships between abstract application logic and concrete spatial and system resources. These concepts enable multi-application environments to remain spatially coherent, interoperable, and conflict-free.
@ -282,18 +283,16 @@ An agent represents an active participant in the AR environment. Depending on th
An ambient defines a spatially and logically bounded context in which a single AR application is executed. Declared in the \textit{system manifest}, an ambient groups a set of components and agents under a coherent coordinate system and enforces application-level isolation. Each ambient is assigned exactly one application at runtime, ensuring exclusive access to its associated components for rendering and interaction. Ambients serve as the primary unit of spatial scoping in AURA: they restrict where an application can observe, modify, or produce output, and they mediate data flow between applications and the system. Although ambients are disjoint in terms of assigned components, they may participate in shared coordination via higher-level \textit{world}s.
\paragraph{World}
A world is a higher-level construct that groups multiple ambients into a shared spatial and semantic context. Declared in the \textit{system manifest}, a world defines global coordination rules --- such as data visibility, and access policies --- that apply across its ambients. Worlds enable interoperability between applications by allowing scoped observation of shared component storage and indirect event communication. While ambients enforce application isolation, worlds provide a mechanism for structured coexistence and interaction between applications that occupy distinct but related regions of space.
A world is a higher-level construct that groups multiple ambients into a shared spatial and semantic context. Declared in the \textit{system manifest}, a world defines global coordination rules --- such as data visibility, and access policies --- that apply across its ambients. Worlds enable interoperability between applications by allowing scoped observation of shared component storage and indirect event communication. While ambients enforce application isolation, worlds provide a mechanism for structured coexistence and communication between applications that occupy distinct but related regions of space.
\paragraph{Manifest}
The pair of declarations that form AURA's contract: the application manifest defines what an app wants to do, while the system manifest declares what is possible in the environment. Together, they define a binding between abstract logic and physical reality. An application manifest is static while the system manifest may be updated during runtime.
\subsection{Runtime Initialization and Spatial Matching}\label{sec:aura-bootstrap}
As shown in \cref{fig:aura_bootstrap}, execution in AURA begins with an application submitting its manifest. The system analyzes spatial, sensory, and interaction requirements and identifies candidate components within the physical environment. The user may optionally assist in selecting which components should back each application entity. However, the application is provided with the components and decides which will in fact use for each entity.
As illustrated in \cref{fig:aura_bootstrap}, execution in AURA begins when a user launches one or more applications --- potentially authored by different third-party developers --- and each submits its manifest to the AURA-based system. The system analyzes the spatial, sensory, and interaction requirements declared in the manifests and identifies suitable components in the physical space. The user may optionally assist in mapping these components to application entities, offering spatial preferences or intent. Based on this input, the system provides each application with an ambient that includes the available components and a proposed entity-to-component assignment. Each application then decides which of the assigned components to bind to its entities. At runtime, the system monitors spatial triggers, schedules component updates, routes input and output, and manages state transitions --- all according to the declarative contracts defined in the manifests and the final entity-to-component bindings.
Once mappings are finalized, the system transmits the relevant \textbf{ambient configuration} to the application. The application acknowledges the selected components and entity attribution, and execution begins. The runtime monitors spatial triggers, schedules component updates, routes input and output, and manages state transitions, all based on the declarative contract provided in the manifests.
This setup process ensures that:
This process ensures that:
\begin{itemize}
\item Component use is explicitly negotiated;
\item Runtime conflicts (e.g., overlapping visual output or sensor contention) are avoided up front;
@ -323,17 +322,19 @@ This setup process ensures that:
%This declarative startup enables safe multi-application coexistence, supports mixed automated and user-directed spatial decisions, and ensures clear boundary definitions for application behaviour.
\subsection{Runtime Safety Model}
\subsection{Ambient Isolation and Security Guarantees}\label{sec:aura-security}
AURA enforces strong isolation boundaries through its ambient-based runtime model:
AURA enforces strong isolation through its ambient-based execution model:
\begin{itemize}
\item Shared components must not be assigned to overlapping ambients;
\item Applications can only observe and write to components within their assigned ambient;
\item Outputs (e.g., visual rendering or actuator control) must target ambient-local components;
\item Access to shared data or sensor feeds from other ambients requires explicit world-scoped observation rights.
\item Outputs (e.g., visual rendering or actuator control) must target components currently available in the ambient;
\item Cross-ambient observation requires explicit world-scoped coordination.
\end{itemize}
This model draws from prior work on secure multi-app AR systems~\cite{Ruth2019ARsecurity}, which highlighted the risks of unregulated cross-app composition, including accidental interference and information leakage. AURA's design directly addresses these challenges by defining ambient boundaries for each application and restricting direct interaction across those boundaries unless structured through a world. This guarantees that rendering, state changes, and input access occur within well-defined scopes, preserving both visual consistency and data safety.
These constraints mitigate several threats outlined by Ruth et al.~\cite{Ruth2019ARsecurity}, including attempts by one user to (1) share unwanted or undesirable content, (2) see private content belonging to another user, or (3) manipulate another user's AR content. By scoping visibility and interaction to well-defined ambients, AURA provides strong default boundaries that prevent such interactions unless explicitly mediated.
AURA also focuses on isolating applications from one another by design: components are non-shareable across ambient instances unless explicitly mediated. This helps limit multi-app interference but does not yet include inter-app communication controls or shared-world consistency guarantees, which remain future work.
\subsection{Component-Based Memory Sharing}\label{sec:component-memory}
@ -345,7 +346,7 @@ AURA supports data exchange between applications by attaching shared memory to c
Each component in the system manifest include a structured data store. Applications can read or write to this memory under defined constraints, enabling indirect coordination. For example, placing a virtual photo on a surface may involve storing its metadata in the corresponding component's data structure.
\paragraph{Data Scoping}
When components are part of shared worlds, their memory can be exposed to multiple applications for observation. Writes, however, are ambient-scoped and conflict-free. See \cref{fig:shared_memory}.
When components are in an ambient that is contained in a world, their memory is shared with all the other ambients that refer to the same component in the same world. Writes, however, are ambient-scoped and conflict-free. See \cref{fig:shared_memory}.
\paragraph{Event-Driven Access}
Applications may subscribe to updates on component storage, such as when another application modifies a shared component. For example, an application monitoring a movable surface could receive events when another application moves or drops a virtual object on it in a different ambient on the same world.
@ -356,81 +357,77 @@ Applications can query current component storage or historical logs of changes,
\paragraph{Snapshot Consistency and System Responsibility}
The system --- not individual applications --- must be responsible for producing consistent snapshots of component memory. Since AURA does not provide global synchronization guarantees across applications, any attempt by applications to independently maintain histories of component updates could result in inconsistent event orderings. By delegating this responsibility to the runtime, AURA ensures a coherent view of temporal changes across the shared environment.
\subsection{Representing Virtual Objects and Interaction Semantics}\label{sec:virtual-objects}
\subsection{Interaction Semantics with Component-Based Memory}\label{sec:virtual-elements}
AURA distinguishes between system-managed spatial resources (components) and application-defined virtual entities. While virtual objects (e.g., icons, photos, avatars) are not directly declared in manifests, their behaviour is often tied to components via spatial anchoring or memory updates.
AURA distinguishes between system-managed omponents and application-defined virtual elements. Virtual elements --- such as photos, tools, avatars, or icons --- are not explicitly declared in manifests, but their behavior can be grounded and made interoperable by binding them to component-based shared memory. This model allows for spatial anchoring, inter-application observability, and indirect coordination across loosely coupled applications.
\paragraph{Data-Driven Object Movement}
Applications may model virtual object movement by updating memory in the relevant component stores:
\begin{itemize}
\item A virtual photo removed from a wall is removed from its surface component store and written to the agent's (hand) component's store.
\item Dropping the object on a table updates the table top component's store and clears it from the agent's (hand) component store.
\end{itemize}
\paragraph{Data-Driven Object Sharing Across Applications}
This approach enables multiple applications to recognize, interact with, or observe such objects without needing to coordinate directly.
Consider a scenario where a user is cooking in the kitchen using a recipe application projected onto the countertop. Simultaneously, their family is seated in the living room, browsing a photo gallery displayed on the dining table via a separate AR application. Both applications are deployed in distinct ambients --- one for the kitchen, one for the living room --- but are part of the same world (\textit{kitchnet}) and share access to a common physical component: a lightweight physical object (e.g., a serving plate or designated tag) declared in both ambients.
The user decides to share a recipe photo with the family. First, the recipe app copies the photo's metadata into the agent's (hand) component store. Then, the user transfers the data into the shared object's component store --- effectively ``placing'' the virtual photo onto the shared object. The user then physically tosses the object from the kitchen into the living room. Upon catching it, the family interacts with the photo gallery app. Since the gallery app can observe the shared component, it retrieves the updated memory and displays the photo. The family then drags the photo from the shared object onto the dining table, which becomes the new anchor for the virtual content.
\paragraph{Cross-Ambient Semantics via Worlds}
This interaction is only possible because both ambients --- kitchen and living room --- are part of the same AURA world. Within a world, shared components remain uniquely identified and consistently observable across ambients. Although the applications are logically separated, AURA's runtime sees the physical object as a single system-managed component. Thus, any update to its memory in one ambient becomes visible in the other, enabling seamless, asynchronous coordination. This declarative and spatially scoped approach to data exchange ensures interoperability without requiring direct app-to-app communication, supporting safe and modular interaction between independently authored applications.
\paragraph{Interoperability Constraints}
Applications may implement alternative handling for virtual objects (e.g., proprietary object representations or tracking). However, such approaches break the declarative guarantees of AURA and prevent meaningful interoperability. Shared object semantics should, whenever possible, leverage AURA's shared memory and manifest declarations.
While applications are free to use proprietary formats or tracking logic for virtual objects, doing so outside AURA's shared memory model undermines interoperability. To ensure system-level coordination and reuse of spatial context, application logic should rely on AURA-declared components and ambient-bound storage. By doing so, developers preserve the benefits of modularity, observability, and runtime-managed consistency.
\subsection{Cross-Application Interoperability Mechanisms}
\label{sec:xdg-portals}
\subsection{Cross-Application Interoperability Mechanisms} \label{sec:xdg-portals}
In addition to structured memory, we encourage the use of host-level \textbf{interoperability services}, such as the XDG Desktop Portals~\cite{XDGDesktopPortals}. These portals provide secure, permissioned channels for applications to request access to shared services, in a user-consented and sandboxed manner, without requiring tight integration.
Beyond structured, spatially scoped memory, AURA can benefit from integration with host-level \textbf{interoperability services} such as XDG Desktop Portals~\cite{XDGDesktopPortals}. These standardized interfaces enable applications to access system services --- like file dialogs, media capture, or clipboard operations --- in a sandboxed, user-mediated manner. They complement AURA's declarative model by enabling permissioned, OS-level interaction without requiring direct inter-application communication.
We envision the following AURA-specific uses for portals:
We envision the following AURA-specific extensions or use cases for portal-based services:
\begin{itemize}
\item \textbf{File and clipboard exchange:} An application may use a portal to export captured data (e.g., annotated photos) to the host system.
\item \textbf{Permissioned interaction bridges}: Applications may launch dialogs or service requests (e.g., sharing a file or data blob) via a system-defined portal agent.
\item \textbf{User-confirmed output redirection}: Applications may request visual output redirection (e.g., streaming to a shared surface) via a media-sharing portal interface.
\item \textbf{Window sharing:} Legacy applications without an AURA manifest may be embedded via portals and anchored as virtual agents or visual surfaces within an ambient.
\item \textbf{File and clipboard exchange:} Applications may export artifacts (e.g., annotated images, sensor logs) to the host filesystem or clipboard via standardized save or copy dialogs.
\item \textbf{Permissioned service bridging:} AURA applications can delegate cross-app requests (e.g., blob sharing, identity selection) to a system-defined portal agent, enabling interactions across app boundaries with explicit user consent.
\item \textbf{Visual output redirection:} Applications may request that rendered content be redirected to a different component or shared surface (e.g., projector, virtual screen), mediated by a media-sharing portal.
\item \textbf{Embedding legacy applications:} Apps that do not provide an AURA manifest --- such as conventional desktop programs --- may be wrapped via window-sharing portals and presented as visual agents or embedded surfaces within an ambient.
\end{itemize}
%This model preserves AURA's declarative and ambient-scoped design while extending its reach into general-purpose OS environments and user-facing application ecosystems.
This bridges the gap between tightly scoped in-AR coordination (via manifests and shared memory) and more general multi-app interoperability through user-facing OS-level services.
This hybrid model bridges the gap between AURA's tightly scoped, manifest-driven coordination and broader interoperability with traditional system services, allowing AR applications to coexist seamlessly with host applications and general-purpose OS environments.
\subsection{Heuristics and Observational State Modeling}\label{sec:heuristics}
AURA allows applications to define \textbf{heuristics} --- semantic abstractions over component-level variables --- to express observable states and their transitions. These definitions are included in the \textit{application manifest} and enable the system to assist applications by evaluating logical conditions on tracked variables and reporting state changes. This mechanism supports a declarative programming model where application logic can respond to high-level events such as ``object rising'' or ``hand approaching surface'' rather than continuously polling low-level signals.
AURA allows applications to define \textbf{heuristics} --- semantic abstractions over component-level variables --- to express observable states and their transitions. These definitions are included in the \textit{application manifest} and enable the system to assist applications by evaluating logical conditions on tracked variables and notifying them of relevant state changes. This mechanism supports a declarative programming model where application logic can respond to high-level events such as ``object rising'' or ``hand approaching surface'' rather than continuously polling low-level signals.
Heuristics may involve:
Heuristics are based on:
\begin{itemize}
\item \textbf{Discrete variables}, derived from Boolean conditions (e.g., \texttt{distance(agent, surface) < 0.3}), that encode binary spatial or contextual predicates.
\item \textbf{Continuous variables}, representing scalar or vector quantities (e.g., position, velocity, acceleration, light intensity), whose values evolve over time.
\item \textbf{Continuous variables}, representing scalar or vector quantities (e.g., position, velocity, acceleration, light intensity), whose values evolve over time and are sampled by the runtime.
\end{itemize}
Each heuristic groups a set of \textit{states}, defined by logical \textit{invariants} over one or more variables, and an optional \textit{transition model} that describes expected state progressions. While transitions are not enforced by the system, they may guide prediction, smoothing, or projection logic in the runtime.
A heuristic consists of named \textit{states}, each defined by an \textit{invariant} --- a logical condition that must hold for the state to be considered active. Applications may optionally define a \textit{transition model} to describe likely or expected state progressions. While not enforced by the system, these transitions can support runtime optimizations such as smoothing, prediction, or performance improvements in tracking and projection. Developers should also specify an optional \textit{activation condition} --- a separate Boolean expression used to indicate when monitoring a state should become more frequent or prioritized.
The following example defines a \texttt{juggling ball} entity that transitions through states based on the vertical component of its velocity ($V.z$), derived from a component attached to the ball's surface:
The following example defines a \texttt{juggling ball} entity that transitions through states based on the vertical component of its velocity ($V.z$):
\begin{lstlisting}[language=json, numbers=none, caption={Juggling Ball Entity's Heuristic-Driven State Model.}]
"continuous_variables": {
"V": "ball_surface.velocity"
"V": {"exp": "ball_surface.velocity", "sample_period_ms": 20}
}, "discrete_variables": {
"T": "room_sensor.temperature > 30",
"H": "room_sensor.humidity > 10"
"T": {"exp": "room_sensor.temperature > 30", "sample_period_ms": 1000},
"H": {"exp": "room_sensor.humidity > 10"}
}, "heuristics": {
"ball_surface": {
"states": {
"up": { "invariant": "V.z > 0", "variant": "distance(agent, ball_surface) < 0.5" },
"down": { "invariant": "V.z < 0", "variant": "distance(agent, ball_surface) < 0.5" },
"idle": { "invariant": "V.z = 0", "variant": "distance(agent, ball_surface) < 0.5" },
"miss": { "invariant": "V.z is unknown", "variant": "distance(agent, ball_surface) < 0.5" }
"up": { "invariant": "V.z > 0", "activation": "distance(agent, ball_surface) < 0.5" },
"down": { "invariant": "V.z < 0", "activation": "distance(agent, ball_surface) < 0.5" },
"idle": { "invariant": "V.z = 0", "activation": "distance(agent, ball_surface) < 0.5" },
"miss": { "invariant": "V.z is unknown", "activation": "distance(agent, ball_surface) < 0.5" }
},
"transitions": {
"idle": "up",
"up": "idle",
"idle": "down"
} } }
"transitions": { "idle": "up", "up": "idle", "idle": "down" }
} }
\end{lstlisting}
In the \texttt{idle} state definition, the system may apply a tolerance threshold (e.g., \texttt{abs(V.z) < 0.01}) to mitigate false positives caused by sensor noise or floating-point imprecision, thereby enhancing robustness. In future work, we aim to investigate adaptive strategies for threshold tuning, clarifying the boundary between system-level inference and application-specified semantics --- particularly in cases where precise control over state transitions is essential to application behaviour.
\paragraph{System Role and Guarantees}
Although heuristics are defined by the application, the system is responsible for continuously evaluating these invariants based on current component data and notifying the application when state changes occur. The variant conditions serve for the system to know when it should increase monitoring of a certain component. The system does not interpret or generalize the semantics of these states --- its role is limited to detecting when invariant conditions are satisfied or violated. This preserves encapsulation and ensures that state meaning remains application-specific.
Although heuristics are authored by the application, the system is responsible for evaluating them at runtime and reporting state transitions. It does not interpret their meaning or enforce transitions --- it merely tracks when invariants hold or are violated. This ensures that the semantics of the states remain under application control, preserving modularity and encapsulation.
Because multiple applications may observe the same component within a shared world, heuristic evaluation serves as a standardized mechanism for interpreting continuous signals without tight coupling. However, only the application that defines a heuristic receives its evaluated states --- other applications may define different heuristics on the same component, or none at all.
Multiple applications may observe the same component and define different heuristics for it. Each application receives only the evaluated results of its own heuristics, even if they are in the same world, allowing for individualized interpretations without conflict. The activation mechanism helps AURA prioritize state monitoring dynamically, based on spatial or contextual relevance, without requiring global synchronization or explicit inter-application coordination.
\subsection{Semantic Interoperability and Ontology-Driven Reasoning}\label{sec:semantic-interop}
@ -449,11 +446,11 @@ Semantic metadata can also assist the runtime in resolving conflicts (e.g., prio
\section{AURA Manifests: Example and Usage}
We illustrate AURA’s architecture through an AR photo gallery application inspired by Microsoft LightSpace~\cite{wilson2010combining}, showcasing how photos can be explored and manipulated across physical surfaces in a multi-user setting. We describe how AURA defines system structure and application behaviour through two complementary JSON-LD-based manifests: the \textit{application manifest} and the \textit{system manifest}. Together, they define the logical, spatial, and operational interface between AR applications and their runtime environment.
We illustrate AURA's architecture through an AR photo gallery application inspired by Microsoft LightSpace~\cite{wilson2010combining}, showcasing how photos can be explored and manipulated across physical surfaces in a multi-user setting. We describe how AURA defines system structure and application behaviour through two complementary JSON-LD-based manifests: the \textit{application manifest} and the \textit{system manifest}. Together, they define the logical, spatial, and operational interface between AR applications and their runtime environment.
\subsection{Application Manifest: Declaring Behaviour and Requirements}
The \textbf{application manifest} is authored by its developer and provides a static declaration of an application's structure and requirements. It includes metadata, agent roles, abstract components, entity definitions, and interaction triggers.
The application manifest is authored by its developer and provides a static declaration of an application's structure and requirements. It includes metadata, agent roles, components, entity definitions, and interaction triggers.
\paragraph{Application Metadata}
Each application begins with basic metadata and spatial constraints:
@ -494,7 +491,7 @@ The application defines that it requires at least one agent (\texttt{instances})
With the \texttt{services} field, the application requires access to hand tracking on a pooling basis, agent coordinates for usage in trigger conditions, and the ability of showing visual output in the agent's hand. Therefore, the application may use these resources directly (system calls) or indirectly (system watchers that require this data).
\paragraph{Components}
Components specify required physical characteristics and access modes. Both entities will use a flat surface that provides video feed and visual output capabilities:
Components specify required physical characteristics and Input/Output capabilities. Both entities will use a flat surface that provides video feed and visual output capabilities:
\begin{lstlisting}[language=json, numbers=none, caption={Application Manifest's Component for Flat Surface.}]
"components": [{
"id": "flat_surface",
@ -503,18 +500,16 @@ Components specify required physical characteristics and access modes. Both enti
"minimum_width": 0.3,
"minimum_height": 0.3
},
"requirements": {
"input": [{
"type": "bw_video_feed",
"minimum_width": 640,
"minimum_height": 480
}],
"output": [{
"type": "visual_output",
"minimum_width": 1280,
"minimum_height": 720
}]
}
"input": [{
"type": "bw_video_feed",
"minimum_width": 640,
"minimum_height": 480
}],
"output": [{
"type": "visual_output",
"minimum_width": 1280,
"minimum_height": 720
}]
}]
\end{lstlisting}
@ -523,12 +518,12 @@ Components specify required physical characteristics and access modes. Both enti
\paragraph{Entities}
The application defines two entities, both referring to the same application manifest component \texttt{flat\_surface}. This does not imply that both entities will have the exact same system-provided component. Application Manifest components are abstract definitions of what is required, but they only assume form inside an entity, as it is based in an entity's \texttt{instances} that the system learns how many components with certain characteristics it has to provide the application with.
The entity \texttt{photo\_gallery} consists of a single interactive visualiser for browsing photos.
The entity \texttt{photo\_gallery} consists of a single interactive visualiser for browsing photos. If it was defined as "1", then if the component that represents this entity was moved off of the ambient's physical space, the application could be halted by the system, as one instance of this entity would be required for it to run.
\begin{lstlisting}[language=json, numbers=none, caption={Photo Gallery Entity Definition.}]
"entities": [{
"id": "photo_gallery",
"components": ["flat_surface"],
"instances": 1,
"instances": "[0-1]",
"discrete_variables": [
"active": "distance(agent, flat_surface) < 0.3"
] },
@ -546,7 +541,7 @@ The entity \texttt{photo\_viewer} can have zero or more instances and consists o
}]
\end{lstlisting}
In both entities, we request the system to notify of valuation changes of the discrete variable "active". When a user approaches within 0.3m of a gallery or photo viewer surface, the system emits an event. This allows the application to react accordingly by requesting access to the associated video feed and/or visual output.
In both entities, we request the system to notify of valuation changes of the discrete variable ``active''. When a user approaches within 0.3m of a gallery or photo viewer surface, the system emits an event. This allows the application to react accordingly by requesting access to the associated video feed and/or visual output.
\subsection{System: Allocation and Coordination}
@ -587,27 +582,15 @@ Ambients can be grouped into worlds for shared observation and data exchange:
}]
\end{lstlisting}
After deployment, the system proposes candidate components matching the application’s requirements, which the application may confirm or remap as part of the bootstrap procedure (see \cref{sec:aura-bootstrap}).
After deployment, the system proposes candidate components matching the application's requirements, which the application may confirm or remap as part of the bootstrap procedure (see \cref{sec:aura-bootstrap}).
\section{Limitations and Future Work}
\section{Conclusion and Future Work}
AURA provides a declarative interface for structuring AR application requirements and managing shared spatial resources. However, it currently assumes a centralised runtime. It also provides no guarantees on global synchronisation, which may lead to diverging histories when multiple applications observe shared component state independently.
This paper introduced AURA, a declarative specification and runtime model for spatially structured, multi-application AR systems. By separating application logic from physical resource allocation and introducing structured data sharing, heuristic-based state modeling, and ambient scoping, AURA enables modular development, scalable coordination, and conflict-free coexistence across applications.
Heuristics in AURA are application-defined and rely on observable variables; the proposal does not yet cover probabilistic reasoning or shared semantic arbitration of state transitions.
Currently, AURA assumes a centralized runtime and lacks global synchronization guarantees, which may lead to diverging histories when applications observe shared state independently. Heuristics are application-defined and deterministic, and semantic arbitration is not yet formalized.
Future directions include:
\begin{itemize}
\item Implementing a reference runtime with arbitration and event routing;
\item Formal specification and validation of manifests and heuristic models;
\item Supporting distributed orchestration across multiple AR devices;
\item Supporting probabilistic reasoning in heuristic evaluation, allowing the system to infer likely state transitions under uncertainty rather than relying solely on deterministic invariants.
\end{itemize}
\section{Conclusion}
This paper introduced AURA, a declarative specification and runtime model for spatially structured, multi-application AR systems. By separating application logic from physical resource allocation, and introducing structured data sharing, heuristic-based state modeling, and ambient scoping, AURA enables modular development and conflict-free coexistence.
The proposed architecture supports safe interaction patterns, semantic extensibility, and runtime adaptability across applications. AURA lays the groundwork for scalable, semantically aware AR systems, and opens promising directions for distributed execution, formal reasoning, probabilistic reasoning, and dynamic spatial orchestration.
Future work includes implementing a reference runtime with arbitration and event routing, formal validation of manifests and ambient compatibility constraints, and distributed orchestration across heterogeneous AR platforms. We are particularly interested in probabilistic extensions to heuristic evaluation for improving tracking and rendering on moving objects, as well as developing an ambient calculus for structured data exchange between components.
%\bibliographystyle{abbrv}
\bibliographystyle{abbrv-doi}