1,720,977 research outputs found
Keeping intelligence under control
Full text linkModern software systems, such as smart systems, are based on a continuous interaction with the dynamic and partially unknown environment in which they are deployed. Classical development techniques, based on a complete description of how the system must behave in different environmental conditions, are no longer effective. On the contrary, modern techniques should be able to produce systems that autonomously learn how to behave in different environmental conditions.Machine learning techniques allow creating systems that learn how to execute a set of actions to achieve a desired goal. When a change occurs, machine learning techniques allow the system to autonomously learn new policies and strategies for actions execution. This flexibility comes at a cost: the developer has no longer full control on the system behaviour. Thus, there is no way to guarantee that the system will not violate important properties, such as safety-critical properties.To overcome this issue, we believe that machine learning techniques should be combined with suitable reasoning mechanisms aimed at assuring that the decisions taken by the machine learning algorithm do not violate safety-critical requirements. This paper proposes an approach that combines machine learning with run-time monitoring to detect violations of system invariants in the actions execution policies
MoVEMo - A structured approach for engineering reward functions
No full textReinforcement learning (RL) is a machine learning technique that has been increasingly used in robotic systems. In reinforcement learning, instead of manually pre-program what action to take at each step, we convey the goal of a software agent in terms of reward functions. The agent tries different actions in order to maximize a numerical value, i.e. the reward. A misspecified reward function can cause problems such as reward hacking, where the agent finds out ways that maximize the reward without achieving the intended goal.As RL agents become more general and autonomous, the design of reward functions that elicit the desired behaviour in the agent becomes more important and cumbersome. In this paper, we present a technique to formally express reward functions in a structured way; this stimulates a proper reward function design and as well enables the formal verification of it. We start by defining the reward function using state machines. In this way, we can statically check that the reward function satisfies certain properties, e.g., high-level requirements of the function to learn. Later we automatically generate a runtime monitor which runs in parallel with the learning agent-that provides the rewards according to the definition of the state machine and based on the behaviour of the agent.We use the Uppaal model checker to design the reward model and verify the TCTL properties that model high-level requirements of the reward function and Larva to monitor and enforce the reward model to the RL agent at runtime
Automotive Architecture Framework: The experience of Volvo Cars
No full textThe automotive domain is living an extremely challenging historical moment shocked by many emerging business and technological needs. Electrification, autonomous driving, and connected cars are some of the driving needs in this changing world. Increasingly, vehicles are becoming software-intensive complex systems and most of the innovation within the automotive industry is based on electronics and software. Modern vehicles can have over 100 Electronic Control Units (ECUs), which are small computers, together executing gigabytes of software. ECUs are connected to each other through several networks within the car, and the car is increasingly connected with the outside world. These novelties ask for a change on how the software is engineered and produced and for a disruptive renovation of the electrical and software architecture of the car. In this paper we describe the current investigation of Volvo Cars to create an architecture framework able to cope with the complexity and needs of present and future vehicles. Specifically, we present scenarios that describe demands for the architectural framework and introduce three new viewpoints that need to be taken into account for future architectural decisions: Continuous Integration and Deployment, Ecosystem and Transparency, and car as a constituent of a System of Systems. Our results are based on a series of focus groups with experts in automotive engineering and architecture from different companies and universities. (C) 2017 Elsevier B.V. All rights reserved
Combining machine-learning with invariants assurance techniques for autonomous systems
No full textAutonomous Systems are systems situated in some environment and are able of taking decision autonomously. The environment is not precisely known at design-time and it might be full of unforeseeable events that the autonomous system has to deal with at run-time. This brings two main problems to be addressed. One is that the uncertainty of the environment makes it difficult to model all the behaviours that the autonomous system might have at the design-time. A second problem is that, especially for safety-critical systems, maintaining the safety requirements is fundamental despite the system's adaptations. We address such problems by shifting some of the assurance tasks at run-time. We propose a method for delegating part of the decision making to agent-based algorithms using machine learning techniques. We then monitor at run-time that the decisions do not violate the autonomous system's safety-critical requirements and by doing so we also send feedback to the decision-making process so that it can learn. We plan to implement this approach using reinforcement learning for decision making and predictive monitoring for checking at run-time the preservation and/or violation of invariant properties of the system. We also plan to validate it using ROS as software middleware and miniaturized vehicles and real vehicles as hardware
Deploying ZKP Frameworks with Real-World Data: Challenges and Proposed Solutions
Full text linkZero-knowledge proof (ZKP) frameworks have the potential to revolutionize the
handling of sensitive data in various domains. However, deploying ZKP
frameworks with real-world data presents several challenges, including
scalability, usability, and interoperability. In this project, we present Fact
Fortress, an end-to-end framework for designing and deploying zero-knowledge
proofs of general statements. Our solution leverages proofs of data provenance
and auditable data access policies to ensure the trustworthiness of how
sensitive data is handled and provide assurance of the computations that have
been performed on it. ZKP is mostly associated with blockchain technology,
where it enhances transaction privacy and scalability through rollups,
addressing the data inherent to the blockchain. Our approach focuses on
safeguarding the privacy of data external to the blockchain, with the
blockchain serving as publicly auditable infrastructure to verify the validity
of ZK proofs and track how data access has been granted without revealing the
data itself. Additionally, our framework provides high-level abstractions that
enable developers to express complex computations without worrying about the
underlying arithmetic circuits and facilitates the deployment of on-chain
verifiers. Although our approach demonstrated fair scalability for large
datasets, there is still room for improvement, and further work is needed to
enhance its scalability. By enabling on-chain verification of computation and
data provenance without revealing any information about the data itself, our
solution ensures the integrity of the computations on the data while preserving
its privacy
Designing Trustworthy Autonomous Systems
Full text linkThe design of autonomous systems is challenging and ensuring their trustworthiness can have different meanings, such as i) ensuring consistency and completeness of the requirements by a correct elicitation and formalization process; ii) ensuring that requirements are correctly mapped to system implementations so that any system behaviors never violate its requirements; iii) maximizing the reuse of available components and subsystems in order to cope with the design complexity; and iv) ensuring correct coordination of the system with its environment.Several techniques have been proposed over the years to cope with specific problems. However, a holistic design framework that, leveraging on existing tools and methodologies, practically helps the analysis and design of autonomous systems is still missing. This thesis explores the problem of building trustworthy autonomous systems from different angles. We have analyzed how current approaches of formal verification can provide assurances: 1) to the requirement corpora itself by formalizing requirements with assume/guarantee contracts to detect incompleteness and conflicts; 2) to the reward function used to then train the system so that the requirements do not get misinterpreted; 3) to the execution of the system by run-time monitoring and enforcing certain invariants; 4) to the coordination of the system with other external entities in a system of system scenario and 5) to system behaviors by automatically synthesize a policy which is correct
Designing Trustworthy Autonomous Systems [Elektronisk resurs]
No full textThe design of autonomous systems is challenging and ensuring their trustworthiness can have different meanings, such as i) ensuring consistency and completeness of the requirements by a correct elicitation and formalization process; ii) ensuring that requirements are correctly mapped to system implementations so that any system behaviors never violate its requirements; iii) maximizing the reuse of available components and subsystems in order to cope with the design complexity; and iv) ensuring correct coordination of the system with its environment. Several techniques have been proposed over the years to cope with specific problems. However, a holistic design framework that, leveraging on existing tools and methodologies, practically helps the analysis and design of autonomous systems is still missing. This thesis explores the problem of building trustworthy autonomous systems from different angles. We have analyzed how current approaches of formal verification can provide assurances: 1) to the requirement corpora itself by formalizing requirements with assume/guarantee contracts to detect incompleteness and conflicts; 2) to the reward function used to then train the system so that the requirements do not get misinterpreted; 3) to the execution of the system by run-time monitoring and enforcing certain invariants; 4) to the coordination of the system with other external entities in a system of system scenario and 5) to system behaviors by automatically synthesize a policy which is correct
Engineering Trustworthy Self-Adaptive Autonomous Systems
Full text linkAutonomous Systems (AS) are becoming ubiquitous in our society. Some examples are autonomous vehicles, unmanned aerial vehicles (UAV), autonomous trading systems, self-managing Telecom networks and smart factories. Autonomous Systems are based on a continuous interaction with the environment in which they are deployed, and more often than not this environment can be dynamic and partially unknown. AS must be able to take decisions autonomously at run-time also in presence of uncertainty. Software is the main enabler of AS and it allows the AS to self-adapt in response to changes in the environment and to evolve, via the deployment of new features.Traditionally, software development techniques are based on a complete description at design time of how the system must behave in different environmental conditions. This is no longer effective since the system has to be able to explore and learn from the environment in which it is operating also after its deployment. Reinforcement learning (RL) algorithms discover policies that can lead AS to achieve their goals in a dynamic and unknown environment. The developer does not specify anymore how the system should act in each possible situation but rather the RL algorithm can achieve an optimal behaviour by trial and error. Once trained, the AS will be capable of taking decisions and performing actions autonomously while still learning from the environment. These systems are becoming increasingly powerful, yet this flexibility comes at a cost: the learned policy does not necessarily guarantee safety or the achievement of the goals.This thesis explores the problem of building trustworthy autonomous systems from different angles. Firstly, we have identified the state of the art and challenges of building autonomous systems, with a particular focus on autonomous vehicles. Then, we have analysed how current approaches of formal verification can provide assurances in a System of Systems scenario. Finally, we have proposed methods that combine formal verification with reinforcement learning agents to address two major challenges: how to trust that an autonomous system will be able to achieve its goals and how to ensure that the behaviour of AS is safe
- …
