jtasso's final review

This commit is contained in:
Diogo Peralta Cordeiro 2025-04-11 21:42:30 +01:00
parent 69f7a225a6
commit 687c1305db
Signed by: diogo
GPG Key ID: 18D2D35001FBFAB0

View File

@ -164,7 +164,7 @@ AURA is a different approach to structuring AR applications and coordinating the
Importantly, AURA does not enforce a single global spatial model. Instead, each application is bound to an ambient-scoped view, enabling coexistence through a layered abstraction of space. The \textit{``unified''} nature of AURA refers to its consistent runtime contract: all applications interface with the system using a common schema, while the system tracks general interaction and coordinates access to components without requiring direct inter-application communication.
To ground the architecture in a tangible scenario, \cref{fig:teaser} illustrates how AURA supports runtime coordination across multiple independent AR applications. In (a), two applications submit their manifests to an AURA-based system, which parses their spatial and behavioural requirements. In (b), the system creates an ambient for each application, which maps them within the physical space --- ensuring non-conflicting use of shared surfaces --- while enabling the user to intervene in the mapping process. The user can then interact with both applications. AURA enables:
To ground the architecture in a tangible scenario, \cref{fig:teaser} illustrates how AURA supports runtime coordination across multiple independent AR applications. In (a), two applications submit their manifests to an AURA-based system, which parses their spatial and behavioural requirements. In (b), the system creates an ambient for each application, mapping them onto the physical space while preventing conflicts over shared surfaces. The mapping process remains user-adjustable, and interaction occurs via an HMD-based system --- though AURA abstracts away the underlying platform, as discussed later. AURA enables:
\begin{itemize}
\item Standardized definition of application structure, including components, entities, interaction logic, devices, and performance requirements;
\item Concurrent execution of multiple AR applications in shared physical environments without conflict, enabling true coexistence;
@ -221,7 +221,7 @@ AURA's manifest draws upon these lessons: like SSIML/AR and ARML, it seeks a cro
Supporting multiple concurrently running AR applications in a shared physical environment presents challenges related to visual consistency, interaction integrity, and resource management. Although commercial platforms have made strides in spatial persistence and immersive UI, robust multi-application AR remains largely unsupported. Prior work such as Huynh et al.'s ``Layerable Apps'' \cite{huynh2022layerable} and Lebeck et al.'s early analysis of AR runtime conflicts \cite{lebeck2019multiple} highlights user interest in multi-app usage and the architectural shortcomings that impede it. In this section, we compare the architectural approaches of leading platforms --- including Microsoft HoloLens, Apple Vision Pro and ARKit, Magic Leap, Meta Quest, and Android ARCore --- across three dimensions: (1) spatial anchoring and persistence, (2) multi-application coexistence and spatial partitioning, and (3) runtime architecture and resource arbitration. We also contrast these designs with AURA's manifest-based runtime coordination model.
\paragraph{Spatial Anchoring and Persistence.} Modern AR platforms support application-level persistence through spatial anchors. Microsoft's HoloLens enables persistent and shareable anchors\cite{MicrosoftSpatialAnchors} via Azure Spatial Anchors\cite{MicrosoftLearnAzureSpatialAnchors}, though anchor management remains scoped per application. Apple's ARKit supports ARWorldMaps for persistence \cite{ARWorldMap}, while visionOS extends this with automatic anchor restoration \cite{AppleVisionOS}. Magic Leap introduces environmental ``Spaces'' to persist and share spatial mappings \cite{MagicLeapSpacesApp}. Meta's Mixed Reality Utility Kit supports anchor persistence \cite{MetaSpatialAnchorsPersist}, while ARCore offers similar functionality with Cloud Anchors \cite{ARCoreCloudAnchors}. Despite this, all platforms treat anchors as per-app constructs without system-level coordination. In contrast, AURA introduces declarative anchor management via manifests, enabling coordinated anchor access, deduplication, and shared reference across applications.
\paragraph{Spatial Anchoring and Persistence.} Modern AR platforms support application-level persistence through spatial anchors. Microsoft's HoloLens enables persistent and shareable anchors\cite{MicrosoftSpatialAnchors} via Azure Spatial Anchors\cite{MicrosoftLearnAzureSpatialAnchors}, though anchor management remains scoped per application. Apple's ARKit supports ARWorldMaps for persistence \cite{ARWorldMap}, while visionOS extends this with automatic anchor restoration \cite{AppleVisionOS}. Magic Leap introduces environmental ``Spaces'' to persist and share spatial mappings \cite{MagicLeapSpacesApp}. Meta's Mixed Reality Utility Kit supports anchor persistence \cite{MetaSpatialAnchorsPersist}, while ARCore offers similar functionality with Cloud Anchors \cite{ARCoreCloudAnchors}. Despite this, all platforms treat anchors as per-app constructs without system-level coordination. In contrast, AURA introduces declarative anchor management via manifests, enabling coordinated anchor access, no duplication, and shared reference across applications.
\paragraph{Multi-Application Coexistence and Spatial Partitioning.} HoloLens enforces single-immersive-app execution, allowing only auxiliary 2D apps to coexist spatially \cite{lebeck2019multiple}. Magic Leap One enabled limited concurrency via ``prisms'' that spatially confine app content\cite{MagicLeapPrisms}, though overlap is not strictly prevented\cite{lebeck2019multiple}. VisionOS adopts a ``Shared Space'' model \cite{AppleVisionOS}, allowing multiple windowed apps to coexist within a common coordinate frame while preserving visual separation via UI depth cues. AURA diverges by allowing multiple applications to share or isolate space via \textit{ambients} and \textit{worlds}. Conflict resolution is handled proactively through spatial declarations, allowing flexible partitioning or cooperative content merging.
@ -266,15 +266,15 @@ AURA provides a structured and extensible runtime model for AR systems by decoup
\subsection{Fundamental Concepts and Vocabulary}\label{sec:core-concepts}
AURA establishes a declarative and runtime-coordinated interface between applications and spatial environments, separating responsibilities between the application and the system. Each application must provide a static \textbf{application manifest} describing its structure, requirements, and expected behaviour. While the runtime does not strictly require it, we recommend that implementations of AURA maintain a dynamically generated and persistable \textbf{system manifest}, which reflects the physical environment and current component configuration. This supports introspection, debugging, and reproducibility across sessions. \Cref{tab:manifest_comparison} outlines the key differences between both manifest types.
AURA establishes a declarative and runtime-coordinated interface between applications and spatial environments, separating responsibilities between the application and the system running it. Each application must provide a static \textbf{application manifest} describing its structure, requirements, and expected behaviour. While the runtime does not strictly require it, we recommend that implementations of AURA maintain a dynamically generated and persistent \textbf{system manifest}, which reflects the physical environment and current component configuration. This supports introspection, debugging, and reproducibility across sessions. \Cref{tab:manifest_comparison} outlines the key differences between both manifest types.
AURA introduces a shared vocabulary to describe the relationships between abstract application logic and concrete spatial and system resources. These concepts enable multi-application environments to remain spatially coherent, interoperable, and conflict-free.
\paragraph{Component}
A system-defined unit representing a trackable physical structure (e.g., table top, wall), associated device interface (e.g., projector output), or spatial regions. Components are declared in the \textit{system manifest}, including their spatial attributes, capabilities (e.g., input/output modalities). In the \textit{application manifest}, developers declare component \textit{requirements} via abstract descriptions (e.g., minimum dimensions, modalities), which the system maps to real-world instances at runtime. Components serve as the physical anchor for interaction and memory exchange. Elements created and managed internally by the application --- such as photos, UI widgets, or 3D models --- are not declared as components in AURA. However, they may be spatially anchored to or interact with components at runtime, for example by displaying content on a surface component or updating that component's shared memory.
A system-defined unit representing a trackable physical structure (e.g., table top, wall), associated device interface (e.g., projector output), or spatial regions. Components are declared in the \textit{system manifest}, including their spatial attributes, capabilities (e.g., input/output modalities). In the \textit{application manifest}, developers declare component \textit{requirements} via abstract descriptions (e.g., minimum dimensions, modalities), which the system maps to real-world instances at runtime. Components serve as the physical anchor for interaction and information exchange. Elements created and managed internally by the application --- such as photos, UI widgets, or 3D models --- are not declared as components in AURA. However, they may be spatially anchored to or interact with components at runtime, as discussed in \cref{sec:virtual-elements}.
\paragraph{Entity}
An application-defined abstraction representing a logical unit of interaction or visualization within the AR environment. Entities are declared in the \textit{application manifest} and are mapped at runtime to one or more components, which provide their physical presence. An entity can request the system to provide variables --- both discrete (e.g., proximity triggers) and continuous (e.g., the velocity of a component) --- that it uses to drive its behaviour. It may also include heuristics that describe commonly observed patterns through invariant-based state definitions over components (See \cref{sec:heuristics}). While entities are not responsible for rendering or tracking directly, they define how application logic attaches to the environment through the system-managed spatial structure.
An application-defined abstraction representing a logical unit of interaction or visualization within the AR environment. Entities are declared in the \textit{application manifest} and are mapped at runtime to one or more components, which provide their physical presence. An entity may observe discrete (e.g., proximity triggers) and continuous (e.g., component velocity) variables derived from the system's observation of components, which it uses to drive its behaviour. It may also include heuristics that describe commonly observed patterns through invariant-based state definitions over components (See \cref{sec:heuristics}). While entities are not responsible for rendering or tracking directly, they define how application logic attaches to the environment through the system-managed spatial structure.
\paragraph{Agent}
An agent represents an active participant in the AR environment. Depending on the system, Agents may provide services such as position tracking, gesture detection, or visual output --- along with the access mode (e.g., event-driven or polling). In the \textit{application manifest}, the application describes desired and required services. During runtime, application agents are dynamically matched to available system-level agents. Multiple instances of a given agent type may coexist depending on application constraints.
@ -283,7 +283,7 @@ An agent represents an active participant in the AR environment. Depending on th
An ambient defines a spatially and logically bounded context in which a single AR application is executed. Declared in the \textit{system manifest}, an ambient groups a set of components and agents under a coherent coordinate system and enforces application-level isolation. Each ambient is assigned exactly one application at runtime, ensuring exclusive access to its associated components for rendering and interaction. Ambients serve as the primary unit of spatial scoping in AURA: they restrict where an application can observe, modify, or produce output, and they mediate data flow between applications and the system. Although ambients are disjoint in terms of assigned components, they may participate in shared coordination via higher-level \textit{world}s.
\paragraph{World}
A world is a higher-level construct that groups multiple ambients into a shared spatial and semantic context. Declared in the \textit{system manifest}, a world defines global coordination rules --- such as data visibility, and access policies --- that apply across its ambients. Worlds enable interoperability between applications by allowing scoped observation of shared component storage and indirect event communication. While ambients enforce application isolation, worlds provide a mechanism for structured coexistence and communication between applications that occupy distinct but related regions of space.
A world is a higher-level construct that groups multiple ambients into a shared spatial and semantic context. Declared in the \textit{system manifest}, a world defines global coordination rules --- such as data visibility, and access policies --- that apply across its ambients. Worlds enable interoperability between applications by allowing scoped observation of shared component storage and event communication. While ambients enforce application isolation, worlds provide a mechanism for structured coexistence and communication between applications that occupy distinct but related regions of space.
\paragraph{Manifest}
The pair of declarations that form AURA's contract: the application manifest defines what an app wants to do, while the system manifest declares what is possible in the environment. Together, they define a binding between abstract logic and physical reality. An application manifest is static while the system manifest may be updated during runtime.
@ -343,13 +343,13 @@ Components in AURA expose structured data stores that act as shared memory space
AURA supports data exchange between applications by attaching shared memory to components. This enables coordination without direct dependencies between applications. In \cref{fig:shared_memory} we show how worlds and ambients influence data exchange ensuring proper interoperability.
\paragraph{Component Storage}
Each component in the system manifest include a structured data store. Applications can read or write to this memory under defined constraints, enabling indirect coordination. For example, placing a virtual photo on a surface may involve storing its metadata in the corresponding component's data structure.
Each component in the system manifest includes a structured data store. Applications can read or write to this memory under defined constraints, enabling indirect coordination. For example, placing a virtual photo on a surface may involve storing its metadata in the corresponding component's data structure.
\paragraph{Data Scoping}
When components are in an ambient that is contained in a world, their memory is shared with all the other ambients that refer to the same component in the same world. Writes, however, are ambient-scoped and conflict-free. See \cref{fig:shared_memory}.
\paragraph{Event-Driven Access}
Applications may subscribe to updates on component storage, such as when another application modifies a shared component. For example, an application monitoring a movable surface could receive events when another application moves or drops a virtual object on it in a different ambient on the same world.
Applications may subscribe to updates on component storage, such as when another application modifies a shared component. For example, an application monitoring a movable surface could receive events when another application moves or drops a virtual photo on it in a different ambient on the same world.
\paragraph{Request-Based Access}
Applications can query current component storage or historical logs of changes, enabling snapshot-based reasoning.
@ -390,7 +390,7 @@ This hybrid model bridges the gap between AURA's tightly scoped, manifest-driven
\subsection{Heuristics and Observational State Modelling}\label{sec:heuristics}
AURA allows applications to define \textbf{heuristics} --- semantic abstractions over component-level variables --- to express observable states and their transitions. These definitions are included in the \textit{application manifest} and enable the system to assist applications by evaluating logical conditions on tracked variables and notifying them of relevant state changes. This mechanism supports a declarative programming model where application logic can respond to high-level events such as ``object rising'' or ``hand approaching surface'' rather than continuously polling low-level signals.
AURA allows applications to define \textbf{heuristics} --- semantic abstractions over component-level variables --- to express observable states and their transitions. These definitions are included in the \textit{application manifest} and enable the system to assist applications by evaluating logical conditions on tracked variables and notifying them of relevant state changes. This mechanism supports a declarative programming model where application logic can respond to high-level events such as ``object rising'' or ``hand approaching surface'' rather than continuously polling system's sensors signals.
Heuristics are based on:
@ -431,7 +431,7 @@ Multiple applications may observe the same component and define different heuris
\subsection{Semantic Interoperability and Ontology-Driven Reasoning}\label{sec:semantic-interop}
The AURA vocabulary is designed with extensibility in mind, leveraging JSON-LD for semantic richness. To ensure compatibility with emerging AR technologies and applications, Developers can embed domain-specific vocabularies, link to external ontologies, and declare new types of services, components, or agents.
The AURA vocabulary is designed with extensibility in mind, leveraging JSON-LD for semantic richness. To ensure compatibility with emerging AR technologies and applications, developers can embed domain-specific vocabularies, link to external ontologies, and declare new types of services, components, or agents.
\textbf{Example Extensions:}
\begin{itemize}
@ -442,7 +442,7 @@ The AURA vocabulary is designed with extensibility in mind, leveraging JSON-LD f
\item Enabling system reasoning over application intent (e.g., prioritizing alerts over background info).
\end{itemize}
Semantic metadata can also assist the runtime in resolving conflicts (e.g., prioritizing critical overlays), adapting interaction logic, or composing behaviours from multiple applications within a shared world.
Semantic metadata can also assist the runtime in resolving conflicts (e.g., prioritizing critical overlays), adapting interaction logic, or composing behaviours from multiple applications within a world.
\section{AURA Manifests: Example and Usage}