References of "Fonteneau, Raphaël"
     in
Bookmark and Share    
Full Text
Peer Reviewed
See detailReinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives
Glavic, Mevludin ULg; Fonteneau, Raphaël ULg; Ernst, Damien ULg

in The 20th World Congress of the International Federation of Automatic Control, Toulouse 9-14 July 2017 (2017, July)

In this paper, we review past (including very recent) research considerations in using reinforcement learning (RL) to solve electric power system decision and control problems. The RL considerations are ... [more ▼]

In this paper, we review past (including very recent) research considerations in using reinforcement learning (RL) to solve electric power system decision and control problems. The RL considerations are reviewed in terms of speci c electric power system problems, type of control and RL method used. We also provide observations about past considerations based on a comprehensive review of available publications. The review reveals the RL is considered as viable solutions to many decision and control problems across di erent time scales and electric power system states. Furthermore, we analyse the perspectives of RL approaches in light of the emergence of new-generation, communications, and instrumentation technologies currently in use, or available for future use, in power systems. The perspectives are also analysed in terms of recent breakthroughs in RL algorithms (Safe RL, Deep RL and path integral control for RL) and other, not previously considered, problems for RL considerations (most notably restorative, emergency controls together with so-called system integrity protection schemes, fusion with existing robust controls, and combining preventive and emergency control). [less ▲]

Detailed reference viewed: 481 (19 ULg)
Full Text
Peer Reviewed
See detailAutomatic phase identification of smart meter measurement data
Olivier, Frédéric ULg; Ernst, Damien ULg; Fonteneau, Raphaël ULg

in Proc. of CIRED 2017 (2017, June)

This paper highlights the importance of the knowledge of the phase identification for the different measurement points inside a low-voltage distribution network. Besides considering existing solutions, we ... [more ▼]

This paper highlights the importance of the knowledge of the phase identification for the different measurement points inside a low-voltage distribution network. Besides considering existing solutions, we propose a novel method for identifying the phases of the measurement devices, based exclusively on voltage measurement correlation. It relies on graph theory and the notion of maximum spanning tree. It has been tested on a real Belgian LV network, first with simulated unbalanced voltage for which it managed to correctly identify the phases of all measurement points, second, on preliminary data from a real measurement campaign for which it shows encouraging results. [less ▲]

Detailed reference viewed: 328 (26 ULg)
Full Text
Peer Reviewed
See detailApproximate Bayes Optimal Policy Search using Neural Networks
Castronovo, Michaël ULg; François-Lavet, Vincent ULg; Fonteneau, Raphaël ULg et al

in Proceedings of the 9th International Conference on Agents and Artificial Intelligence (ICAART 2017) (2017, February)

Bayesian Reinforcement Learning (BRL) agents aim to maximise the expected collected rewards obtained when interacting with an unknown Markov Decision Process (MDP) while using some prior knowledge. State ... [more ▼]

Bayesian Reinforcement Learning (BRL) agents aim to maximise the expected collected rewards obtained when interacting with an unknown Markov Decision Process (MDP) while using some prior knowledge. State-of-the-art BRL agents rely on frequent updates of the belief on the MDP, as new observations of the environment are made. This offers theoretical guarantees to converge to an optimum, but is computationally intractable, even on small-scale problems. In this paper, we present a method that circumvents this issue by training a parametric policy able to recommend an action directly from raw observations. Artificial Neural Networks (ANNs) are used to represent this policy, and are trained on the trajectories sampled from the prior. The trained model is then used online, and is able to act on the real MDP at a very low computational cost. Our new algorithm shows strong empirical performance, on a wide range of test problems, and is robust to inaccuracies of the prior distribution. [less ▲]

Detailed reference viewed: 481 (28 ULg)
Full Text
Peer Reviewed
See detailAn App-based Algorithmic Approach for Harvesting Local and Renewable Energy Using Electric Vehicles
Dubois, Antoine; Wehenkel, Antoine; Fonteneau, Raphaël ULg et al

in Proceedings of the 9th International Conference on Agents and Artificial Intelligence (ICAART 2017) (2017, February)

The emergence of electric vehicles (EVs), combined with the rise of renewable energy production capacities, will strongly impact the way electricity is produced, distributed and consumed in the very near ... [more ▼]

The emergence of electric vehicles (EVs), combined with the rise of renewable energy production capacities, will strongly impact the way electricity is produced, distributed and consumed in the very near future. This position paper focuses on the problem of optimizing charging strategies for a fleet of EVs in the context where a significant amount of electricity is generated by (distributed) renewable energy. It exposes how a mobile application may offer an efficient solution for addressing this problem. This app can play two main roles. Firstly, it would incite and help people to play a more active role in the energy sector by allowing photovoltaic (PV) panel owners to sell their electrical production directly to consumers, here the EVs’ agents. Secondly, it would help distribution system operators (DSOs) or transmission system operators (TSOs) to modulate more efficiently the load by allowing them to influence EV charging behaviour in real time. Finally, the present paper advocates for the introduction of a two-sided market-type model between EV drivers and electricity producers. [less ▲]

Detailed reference viewed: 732 (93 ULg)
Full Text
Peer Reviewed
See detailOn the Dynamics of the Deployment of Renewable Energy Production Capacities
Fonteneau, Raphaël ULg; Ernst, Damien ULg

in Furze, James N.; Swing, Kelly; Gupta, Anil K. (Eds.) et al Mathematical Advances Towards Sustainable Environmental Systems (2017)

This chapter falls within the context of modeling the deployment of renewable en-ergy production capacities in the scope of the energy transition. This problem is addressed from an energy point of view, i ... [more ▼]

This chapter falls within the context of modeling the deployment of renewable en-ergy production capacities in the scope of the energy transition. This problem is addressed from an energy point of view, i.e. the deployment of technologies is seen as an energy investment under the constraint that an initial budget of non-renewable energy is provided. Using the Energy Return on Energy Investment (ERoEI) characteristics of technologies, we propose MODERN, a discrete-time formalization of the deployment of renewable energy production capacities. Be-sides showing the influence of the ERoEI parameter, the model also underlines the potential benefits of designing control strategies for optimizing the deployment of production capacities, and the necessity to increase energy efficiency. [less ▲]

Detailed reference viewed: 231 (15 ULg)
Full Text
Peer Reviewed
See detailDeep Reinforcement Learning Solutions for Energy Microgrids Management
François-Lavet, Vincent ULg; Taralla, David; Ernst, Damien ULg et al

in European Workshop on Reinforcement Learning (EWRL 2016) (2016, December)

This paper addresses the problem of efficiently operating the storage devices in an electricity microgrid featuring photovoltaic (PV) panels with both short- and long-term storage capacities. The problem ... [more ▼]

This paper addresses the problem of efficiently operating the storage devices in an electricity microgrid featuring photovoltaic (PV) panels with both short- and long-term storage capacities. The problem of optimally activating the storage devices is formulated as a sequential decision making problem under uncertainty where, at every time-step, the uncertainty comes from the lack of knowledge about future electricity consumption and weather dependent PV production. This paper proposes to address this problem using deep reinforcement learning. To this purpose, a specific deep learning architecture has been designed in order to extract knowledge from past consumption and production time series as well as any available forecasts. The approach is empirically illustrated in the case of a residential customer located in Belgium. [less ▲]

Detailed reference viewed: 165 (17 ULg)
Full Text
Peer Reviewed
See detailA Gaussian mixture approach to model stochastic processes in power systems
Gemine, Quentin ULg; Cornélusse, Bertrand ULg; Glavic, Mevludin ULg et al

in Proceedings of the 19th Power Systems Computation Conference (PSCC'16) (2016, June)

Probabilistic methods are emerging for operating electrical networks, driven by the integration of renewable generation. We present an algorithm that models a stochastic process as a Markov process using ... [more ▼]

Probabilistic methods are emerging for operating electrical networks, driven by the integration of renewable generation. We present an algorithm that models a stochastic process as a Markov process using a multivariate Gaussian Mixture Model, as well as a model selection technique to search for the adequate Markov order and number of components. The main motivation is to sample future trajectories of these processes from their last available observations (i.e. measurements). An accurate model that can generate these synthetic trajectories is critical for applications such as security analysis or decision making based on lookahead models. The proposed approach is evaluated in a lookahead security analysis framework, i.e. by estimating the probability of future system states to respect operational constraints. The evaluation is performed using a 33-bus distribution test system, for power consumption and wind speed processes. Empirical results show that the GMM approach slightly outperforms an ARMA approach. [less ▲]

Detailed reference viewed: 170 (45 ULg)
Full Text
Peer Reviewed
See detailBenchmarking for Bayesian Reinforcement Learning
Castronovo, Michaël ULg; Ernst, Damien ULg; Couëtoux, Adrien ULg et al

in PLoS ONE (2016)

In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the col- lected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand ... [more ▼]

In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the col- lected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand. Many BRL algorithms have already been proposed, but even though a few toy examples exist in the literature, there are still no extensive or rigorous benchmarks to compare them. The paper addresses this problem, and provides a new BRL comparison methodology along with the corresponding open source library. In this methodology, a comparison criterion that measures the performance of algorithms on large sets of Markov Decision Processes (MDPs) drawn from some probability distributions is defined. In order to enable the comparison of non-anytime algorithms, our methodology also includes a detailed analysis of the computation time requirement of each algorithm. Our library is released with all source code and documentation: it includes three test prob- lems, each of which has two different prior distributions, and seven state-of-the-art RL algorithms. Finally, our library is illustrated by comparing all the available algorithms and the results are discussed. [less ▲]

Detailed reference viewed: 557 (47 ULg)
Full Text
Peer Reviewed
See detailTowards the Minimization of the Levelized Energy Costs of Microgrids using both Long-term and Short-term Storage Devices
François-Lavet, Vincent ULg; Gemine, Quentin ULg; Ernst, Damien ULg et al

in Smart Grid: Networking, Data Management, and Business Models (2016)

This chapter falls within the context of the optimization of the levelized energy cost (LEC) of microgrids featuring photovoltaic panels (PV) associated with both long-term (hydrogen) and short-term ... [more ▼]

This chapter falls within the context of the optimization of the levelized energy cost (LEC) of microgrids featuring photovoltaic panels (PV) associated with both long-term (hydrogen) and short-term (batteries) storage devices. First, we propose a novel formalization of the problem of building and operating microgrids interacting with their surrounding environment. Then we show how to optimally operate a microgrid using linear programming techniques in the context where the consumption and the production are known. It appears that this optimization technique can also be used to address the problem of optimal sizing of the microgrid, for which we propose a robust approach. These contributions are illustrated in two different settings corresponding to Belgian and Spanish data. [less ▲]

Detailed reference viewed: 198 (57 ULg)
Full Text
See detailArtificial Intelligence and Energy
Cornélusse, Bertrand ULg; Fonteneau, Raphaël ULg

Conference (2016, February 02)

Detailed reference viewed: 140 (9 ULg)
Full Text
Peer Reviewed
See detailDecision Making from Confidence Measurement on the Reward Growth using Supervised Learning: A Study Intended for Large-Scale Video Games
Taralla, David ULg; Qiu, Zixiao ULg; Sutera, Antonio ULg et al

in Proceedings of the 8th International Conference on Agents and Artificial Intelligence (ICAART 2016) - Volume 2 (2016, February)

Video games have become more and more complex over the past decades. Today, players wander in visually and option- rich environments, and each choice they make, at any given time, can have a combinatorial ... [more ▼]

Video games have become more and more complex over the past decades. Today, players wander in visually and option- rich environments, and each choice they make, at any given time, can have a combinatorial number of consequences. However, modern artificial intelligence is still usually hard-coded, and as the game environments become increasingly complex, this hard-coding becomes exponentially difficult. Recent research works started to let video game autonomous agents learn instead of being taught, which makes them more intelligent. This contribution falls under this very perspective, as it aims to develop a framework for the generic design of autonomous agents for large-scale video games. We consider a class of games for which expert knowledge is available to define a state quality function that gives how close an agent is from its objective. The decision making policy is based on a confidence measurement on the growth of the state quality function, computed by a supervised learning classification model. Additionally, no stratagems aiming to reduce the action space are used. As a proof of concept, we tested this simple approach on the collectible card game Hearthstone and obtained encouraging results. [less ▲]

Detailed reference viewed: 490 (27 ULg)
Full Text
Peer Reviewed
See detailImitative Learning for Online Planning in Microgrids
Aittahar, Samy ULg; François-Lavet, Vincent ULg; Lodeweyckx, Stefan et al

in Woon, Wei Lee; Zeyar, Aung; Stuart, Madnick (Eds.) Data Analytics for Renewable Energy Integration (2015, December 15)

This paper aims to design an algorithm dedicated to operational planning for microgrids in the challenging case where the scenarios of production and consumption are not known in advance. Using expert ... [more ▼]

This paper aims to design an algorithm dedicated to operational planning for microgrids in the challenging case where the scenarios of production and consumption are not known in advance. Using expert knowledge obtained from solving a family of linear programs, we build a learning set for training a decision-making agent. The empirical performances in terms of Levelized Energy Cost (LEC) of the obtained agent are compared to the expert performances obtained in the case where the scenarios are known in advance. Preliminary results are promising. [less ▲]

Detailed reference viewed: 141 (53 ULg)
Full Text
Peer Reviewed
See detailHow to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
François-Lavet, Vincent ULg; Fonteneau, Raphaël ULg; Ernst, Damien ULg

in NIPS 2015 Workshop on Deep Reinforcement Learning (2015, December)

Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as ... [more ▼]

Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its final value, we empirically show that it is possible to significantly reduce the number of learning steps. When used in conjunction with a varying learning rate, we empirically show that it outperforms original DQN on several experiments. We relate this phenomenon with the instabilities of neural networks when they are used in an approximate Dynamic Programming setting. We also describe the possibility to fall within a local optimum during the learning process, thus connecting our discussion with the exploration/exploitation dilemma. [less ▲]

Detailed reference viewed: 151 (41 ULg)
Full Text
See detailUne histoire d'énergie: équations et transition
Fonteneau, Raphaël ULg

Speech/Talk (2015)

Detailed reference viewed: 26 (2 ULg)
Full Text
Peer Reviewed
See detailArtificial Intelligence in Video Games: Towards a Unified Framework
Safadi, Firas ULg; Fonteneau, Raphaël ULg; Ernst, Damien ULg

in International Journal of Computer Games Technology (2015), 2015

With modern video games frequently featuring sophisticated and realistic environments, the need for smart and comprehensive agents that understand the various aspects of complex environments is pressing ... [more ▼]

With modern video games frequently featuring sophisticated and realistic environments, the need for smart and comprehensive agents that understand the various aspects of complex environments is pressing. Since video game AI is often specifically designed for each game, video game AI tools currently focus on allowing video game developers to quickly and efficiently create specific AI. One issue with this approach is that it does not efficiently exploit the numerous similarities that exist between video games not only of the same genre, but of different genres too, resulting in a difficulty to handle the many aspects of a complex environment independently for each video game. Inspired by the human ability to detect analogies between games and apply similar behavior on a conceptual level, this paper suggests an approach based on the use of a unified conceptual framework to enable the development of conceptual AI which relies on conceptual views and actions to define basic yet reasonable and robust behavior. The approach is illustrated using two video games, Raven and StarCraft: Brood War. [less ▲]

Detailed reference viewed: 502 (39 ULg)
Full Text
See detailFrom Bad Models to Good Policies: an Intertwined Story about Energy and Reinforcement Learning
Fonteneau, Raphaël ULg

Speech/Talk (2014)

Batch mode reinforcement learning is a subclass of reinforcement learning for which the decision making problem has to be addressed without model, using trajectories only (no model, nor simulator nor ... [more ▼]

Batch mode reinforcement learning is a subclass of reinforcement learning for which the decision making problem has to be addressed without model, using trajectories only (no model, nor simulator nor additional interactions with the actual system). In this setting, we propose a discussion about a minmax approach to generalization for deterministic problems with continuous state space. This approach aims at computing robust policies considering the fact that the sample of trajectories may be arbitrarily bad. This discussion will be intertwined with the description of a fascinating batch mode reinforcement learning-type problem with trajectories of societies as input, and for which crucial good decisions have to be taken: the energy transition. [less ▲]

Detailed reference viewed: 39 (1 ULg)
Full Text
Peer Reviewed
See detailUsing approximate dynamic programming for estimating the revenues of a hydrogen-based high-capacity storage device
François-Lavet, Vincent ULg; Fonteneau, Raphaël ULg; Ernst, Damien ULg

in IEEE Symposium Series on Computational Intelligence (2014)

This paper proposes a methodology to estimate the maximum revenue that can be generated by a company that operates a high-capacity storage device to buy or sell electricity on the day-ahead electricity ... [more ▼]

This paper proposes a methodology to estimate the maximum revenue that can be generated by a company that operates a high-capacity storage device to buy or sell electricity on the day-ahead electricity market. The methodology exploits the Dynamic Programming (DP) principle and is specified for hydrogen-based storage devices that use electrolysis to produce hydrogen and fuel cells to generate electricity from hydrogen. Experimental results are generated using historical data of energy prices on the Belgian market. They show how the storage capacity and other parameters of the storage device influence the optimal revenue. The main conclusion drawn from the experiments is that it may be advisable to invest in large storage tanks to exploit the inter-seasonal price fluctuations of electricity. [less ▲]

Detailed reference viewed: 95 (30 ULg)
Full Text
See detailUne Histoire d'Energie : Equations et Transition - Energy Stories, Equations and Transition
Fonteneau, Raphaël ULg

Speech/Talk (2014)

Having access to abundant energy is a key component of our societies' lifestyle. The energy transition amounts in abolishing the dependency of our societies to finite and non-renewable energy resources ... [more ▼]

Having access to abundant energy is a key component of our societies' lifestyle. The energy transition amounts in abolishing the dependency of our societies to finite and non-renewable energy resources. Is this manageable? Applied mathematics may help us imagine the future... [less ▲]

Detailed reference viewed: 18 (2 ULg)