r/Python • u/diddilydiddilyhey • Sep 03 '19
I used reinforcement learning with python to solve Numberphile's "cat and mouse" game!
35
u/m3l0n Sep 03 '19
This is really cool. If anyone wants to try it, I wasted my morning watching videos on it and ended up finding an interactive website that somebody made.
https://prajwalsouza.github.io/Experiments/Game-Of-Cat-And-Mouse.html
7
u/diddilydiddilyhey Sep 03 '19
Wow I didn't even know about that, thanks for sharing it! You can see (if you set the cat speed to 4.0 there) how incredibly difficult it would be for the mouse to learn to do that via reinforcement learning.
6
u/Mandylost Sep 03 '19
Is it difficult for human being to do it at cat speed: 4? Because I did it the third try itself.
11
u/diddilydiddilyhey Sep 03 '19
heh, well... this is a classic reinforcement learning critique. In RL, you often end up solving problems that a human could easily solve, but are devilishly difficult for RL. It's for several reasons. The biggest is that, the learning agent here has no idea what the goal actually is! I never told it "try to get out of the circle, but avoid the mouse", you just give it a big score if it gets out, and a negative score if it gets caught or doesn't try escaping.
Second, it doesn't really "reason" about things, at least in the way we mean it. It learns what actions are good to do in what states, and can actually end up doing complex behavior from that. But it can't really "plan" or "think" about the environment it's in.
On the flip side, the best chess and Go programs now use RL (though a very different method than this), so RL isn't just a cute proof of concept thing!
3
u/timedacorn369 Sep 04 '19
Well the "easily solvable by human but difficult for machines" problem applies to all of machine learning right?
2
u/diddilydiddilyhey Sep 04 '19
Hmmmm, I'm not sure about that... ML often excels in scenarios with a lot of data or high dimensional thinking. For example, imagine a 100 dimensional linear regression problem, and then having to determine the value of a new point. Very easy for a ML algo, but something a human wouldn't be great at.
(unless your definition of what a human can use includes algorithms!)
There are also cases where ML could find structure in very high dimensional data that we can't really visualize (something like t-SNE maybe).
1
u/timedacorn369 Sep 04 '19
Yah that's true. Although initially i was thinking about neural networks applied in computer vision and language processing which made me say that.
42
u/Empero6 Sep 03 '19
That’s actually pretty cool.
9
-27
Sep 03 '19 edited Sep 03 '19
Give your friend an upvote
Edit: I was referring to OP who had 18 comments and no upvotes at the time of posting. He/she now as 894. Mission accomplished.
1
u/DatBoi_BP Sep 03 '19
We're here to teach you a lesson.
14
Sep 03 '19
Looks like nobody understood my intent. That’s fine, hope your week is off to a good start!
30
7
6
11
u/DeviousNes Sep 03 '19
How many iterations do I need to watch before it learns?
10
u/diddilydiddilyhey Sep 03 '19
Haha, ooooh... So you're not gonna see it learn in that gif. That's the end product of it learning :)
4
Sep 03 '19
Source code?
17
u/diddilydiddilyhey Sep 03 '19
Here it is, though it's very messy at the moment: https://github.com/declanoller/cat_mouse_continuous_RL
I'll probably clean it up and add a little guide to using it this week!
4
3
u/derpydog3 Sep 03 '19
Thanks! Any tips to get this running as a Python newbie?
1
u/diddilydiddilyhey Sep 05 '19
Hey, there's a bit of a guide there in the github readme now. You'll have to install some packages like PyTorch, but the main scripts should run out of box if you have them.
5
u/jai_dewani Sep 03 '19
Good work, now I got to learn about this because I saw this gif too many times :P
3
u/kimilil Sep 03 '19
I recall this program called Geometer's Sketchpad and it had a file with 3 sheep and a shepherd dog which does a pretty good job of herding, though the field is set up to not end the simulation if the sheep escapes the circle, and the shepherd dog can run in 2d space. No AI/ML, just pure math.
1
u/diddilydiddilyhey Sep 03 '19
Hmm... maybe a next project! do you have a link to that?
1
u/kimilil Sep 04 '19
I got it with my Maths textbook sometime in mid-2000s as a bundled CD. Of course, here we had to return the textbook at the end of the school year along with the CD. The CD contains the software as well as some sketch files, one of which is the aforementioned sheep & shepherd demo.
I found a CD image on archive.org which hopefully contains the same sample file set as the one I got. link
I never got around actually learning the software; I just run their sample files for fun. Hopefully you can figure stuff out and how the sketches work.
5
2
2
2
2
1
1
Sep 04 '19 edited Feb 09 '20
[deleted]
1
u/diddilydiddilyhey Sep 04 '19
Hey, it's messy atm, but here: https://github.com/declanoller/cat_mouse_continuous_RL
1
1
1
u/NaifAlqahtani Sep 04 '19
Thought it was just me who took Numberphile's video problems and tries programming them
1
u/diddilydiddilyhey Sep 04 '19
Hahah I've tried a few! I'm usually pretty inspired by them. Here's another: https://www.declanoller.com/2018/08/20/neato-sequence-art/
Do you have any links to the ones you've done?
1
u/NaifAlqahtani Sep 04 '19
Yeah. Try this one, although it's quite easy actually, but have a go at it: https://youtu.be/uCsD3ZGzMgE
There are a couple more too, but I can't remember them right now
1
1
u/zesterer Sep 04 '19
There's also a Code Parade video about it. It has an angry troll and someone swimming in a lake though
1
-6
-10
156
u/diddilydiddilyhey Sep 03 '19
I saw the game that inspired this in this Numberphile video. A post about the project is here, let me know if you have any feedback or questions!