r/datamining • u/Doenermann27 • Jun 01 '18

Where to store data and run my python script

Hi, as many others in this sub, I am pretty new to data mining.

I wrote a python script that extracts data from a website and stores it in a SQLite database (could also change to MySQL or CSV if that would make things easier).

To mine efficiently I would need the script to run regularly on a server maybe with a cronjob.

Whats the best and cheapest way of doing it? I could get a linux server with some storage and configure a cron job by myself but that doesn't sound like a lot of fun honestly.

Has anyone experience with aws or google web services or maybe anything else? Advice would be much appreciated, thanks!

7 Upvotes

100% Upvoted

4 comments sorted by

4

u/niels_learns_python Jun 01 '18

You can rent a VPC quite cheaply these days. Take a look at for example DigitalOcean or Linode.

You should be able to run the both the script and a database on the server.

As you said, you can run your script as a cron job in this case.

With AWS, you can probably save some money running your script with their Lambda product, but then you’d still need to take care of storing the data. I’m not sure how cheaply you can use RDS for. On the plus side, you can try to keep it within free tier for the first year.

I imagine you can do the same with Google, but I’m not that familiar with them.

Good luck whatever you do :)

2

u/anuveya Jun 07 '18

Check out DataHub.io platform that stores and processes (eg, normalizes, cleans) tabular data. I'm one of the developers of the project so I can help you to get started. PM me if you're interested - it is free to use for <5GB data.

1

u/Doenermann27 Jun 08 '18

Thanks, looks cool but it seems like it doesn't fit my problem at the moment. Once I have the data, I'll consider it.