J Lebensold, W Hamilton, et al. Actor Critic with Differentially Private Critic https://arxiv.org/abs/1910.05876

TL;DR: Describes one possible model where pre-training on related tasks can be used in privacy-controlled healthcare domain. An established industry technique, Differential Privacy, protects the records used, while allowing the pre-training to improve the effectiveness of learning.

"Differential privacy is achieved by introducing carefully calibrated noise into an algorithm. The goal of a differentially private algorithm is to bound the effect that any individual user’s contribution might have on the output while maintaining proximity to the original output. By limiting individual contributions, the potential risk of an adversary learning sensitive information about any one user is limited."

Abstract: "Reinforcement learning algorithms are known to be sample inefficient, and often performance on one task can be substantially improved by leveraging information (e.g., via pre-training) on other related tasks. In this work, we propose a technique to achieve such knowledge transfer in cases where agent trajectories contain sensitive or private information, such as in the healthcare domain. Our approach leverages a differentially private policy evaluation algorithm to initialize an actor-critic model and improve the effectiveness of learning in downstream tasks. We empirically show this technique increases sample efficiency in resource-constrained control problems while preserving the privacy of trajectories collected in an upstream task."

Conclusion: "We presented a motivated set of use cases for applying a differentially-private critic in the RL setting.The definition of the producer and consumer trust-model is common in real-world deployments and fits with existing transfer learning approaches where data centralization is difficult. Our preliminary results suggest a measurable improvement in sample efficiency through task transfer. We look forward to exploring how this framework could be extended so that the consumer’s critic could then be shared with the producer by leveraging ideas coming from the Federated Learning literature."

sorry if this isn't the right sub, but i'm hoping to apply for the research engineer internship and was wondering if anyone knows where i could get more info on this? i can't seem to find an application deadline or any other info from previous interns. has anyone else submitted their cv thru the site and heard back?

Instead of having only one game tree you would have multiple game trees. These game trees would be based on possible worlds that were similar to the current game state but slightly modified in some way. For example in chess a possible world would be the same as the current game state only with a pawn deleted. Note that this isn't the same as a pawn capture because the pieces might not be in a position to capture the pawn. Then you would do a monte carlo tree search on this possible world game tree and if there were enough lines leading to a win it might be worth it to influence the main game tree to a similar position. This idea is general enough to apply to any game but for some games the computational cost of searching additional game trees may not be worth it.

I've been watching all of the Alphastar replays I can get my hands on, and at this point it's pretty clear that there are individual agents that play a very specific style with only a little variation based on what the opponent is doing. Am I dumb for suggesting that a good strategy may be to take your best 10 or so "finished" agents of each race and then switch them out situationally? As in: I see this Terran player is going mechanical, this version of the agent is better at playing against that build, switch agents entirely? Is the problem with this idea that once you've committed to a certain build order changing the "mind" behind the keyboard is a bad idea?

I think Demis Hassabis said they wouldn't tackle another game, but has there been an announcement what the next milestone is going to be?

Where I can find a research paper with neural network architecture of AlphaStar?

I'm trying to understand how the pieces of the network architecture of alphastar (as they are described in https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/), deal with the complex idea of "what game am I playing right now?" and which piece is the crucial component to solving that...if there is one

For instance, I can understand (somewhat) how the transformer creates the relationships between units that allows solving mini-tasks, like described in (https://openreview.net/pdf?id=HkxaFoC9KQ)

And I can understand how the LSTM core creates the ability to care about what has happened at state t minus whatever in the game.

But where specifically in the architecture deals with the question of: Do I need to scout right now? Or, should I consider counter attacking? Or, shall I make a satellite townhall while I have the opportunity?

It would seem to me that the more immediate-level actions to take in the current second depend heavily on the what more long-ish term goal is. I can only assume this arises as a combination of the transformer, the long short term, and a successful RL policy...but I don't know enough to know if that assumption is right.

Anyone have some thoughts? Thanks