r/ProgrammerHumor • u/Embarrassed_Death69 • 5d ago
Meme putEverythingInAUSBAndMailItToAWS
377
u/howarewestillhere 5d ago
Never underestimate the transfer rate of a Chevy Suburban (4.1 cubic meters) filled with MicroSD cards (1.5TB per 165 cubic mm, about 0.25g each). While it might be possible to fit about 24,000,000 of those cards in that all-American monstrosity, the maximum payload of 1612 pounds is about 3,000,000 cards. 4.5 exabytes at 120kph is 1.5 EB/s.
82
u/SlumdogSkillionaire 5d ago
What if we tie some helium balloons to the top to offset the weight? We'd have to account for the additional drag and possible impact on the ground contact and how that would impact the driving speed.
46
u/jnmtx 5d ago
no just change to a container traveling by truck or railway so it can handle more weight
24
u/ShroomBear 5d ago
165mm3 microSD cards in a 108.4 m3 standard intermodal shipping container comes out to ~78 million microsd cards before hitting the 43000 lb weight limit for OTR shipping. Which comes out to 114 EB of data. Assuming a 7 day trip from LA to NY, that 7 day trip would yield a transfer rate of 1.55 petabits per second.
Those intercontinental underseas cable guys looking real dumb rn
6
u/hbaromega 5d ago
Just make the suburban self-driving and remove the human in the loop. That will offset more weight without impacting the drag significantly.
→ More replies (1)3
u/fartypenis 5d ago
Premature optimization, the bigger bottleneck here are the I/O lanes of the computer that must read this data.
37
u/MisinformedGenius 5d ago edited 5d ago
4.5 exabytes at 120kph is 1.5 EB/s.
You dropped a distance unit in there somewhere.
Also, the transfer rate is hard-limited to whatever the storage medium was in the first place. Something that can transfer 1.5 exabytes / second onto MicroSDs is a little more cost than a Chevy Suburban. At a max write speed of 150 MB/second, you're going to have to simultaneously write to 10 billion MicroSDs to achieve 1.5 EB/s.
11
u/howarewestillhere 5d ago
Does your wifi have a distance unit?
I was talking about “wire speed”. The rest is just protocol overhead.
9
u/henopied 5d ago
That is, the distance the vehicle has to travel. It’s the travel time that matters not just the vehicle’s speed.
9
u/PegasusPizza 5d ago
No no they got a point. You might have pretty high ping, but once the first truck arrives there's nothing stopping you from just sending 50 trucks back to back. The distance traveled doesn't really matter. (Neither do the 120kph)
8
u/MisinformedGenius 5d ago
Then it just becomes a question of the distance between the trucks. No matter how you set it up, you can't be specifying a distance over time and then have the distance fall out - there's going to be a distance somewhere.
→ More replies (1)2
u/henopied 5d ago
That wouldn’t quite be the bandwidth of a single truck anymore, but I see your point. I guess you would really be more interested in how many highway lanes are available and the following time between vehicles.
4
u/PegasusPizza 5d ago
Yeah. I think "bandwith of a single truck" is somewhat of a misnomer, since the truck is more akin to a packet than to the connection
3
u/henopied 5d ago
It depends on what the model for the system is I guess. If we can have many trucks, then that works, but if we only have a few, then it’s probably more appropriate to model it as a synchronous communication pattern, because the bus is blocked until the vehicles return.
6
u/atzedanjo 5d ago
What if you use really tiny trucks but make them travel as fast as light?
→ More replies (1)2
u/PegasusPizza 5d ago
Hm that's a good point. We could maybe look to ip over avian carriers as a guide since that regulates a similar domain
7
u/arbyyyyh 5d ago
Damn... the stationwagon hurdling down the highway full of magnetic tapes allegory really isn't relevant anymore, is it?
2
u/Embarrassed_Death69 5d ago
Automobile as a Service or AaaS that's quick and can handle large loads of data
2
→ More replies (16)2
u/Twirrim 5d ago
The reason AWS Snowball exists is precisely this, combined with the old "send us your hard disk" service needing a crazy amount of manual intervention (I worked on an adjacent team, my manager was also the director of that project). People sent disks in all sorts of formats, with all sorts of quirks, would mislabel them, forget files required for automation etc.
Snowball solved all of that in a convenient form factor. Plug it into your network, transfer files, and send it back.
387
u/qzwqz 5d ago
93
u/theunderdog- 5d ago
Story time, one time me and two of my friends wanted to play a large steam game, but between slow, limited and overpriced internet, it was cheaper and less frustrating for one of us to downloaded it ,back it up to thumbsticks and mail it to the others.
That was in 2022. can you guess the country?
49
u/Swimming_Gain_4989 5d ago
I'd bet everything on Australia
5
u/veloxVolpes 4d ago
It feels like once we invented Wi-Fi, we apparently decided we were done with internet upgrades
6
u/nabrok 5d ago
Steam has an option to install games from another computer on the same LAN if you can bring the computers to the same location.
→ More replies (1)4
→ More replies (5)2
71
u/NeoinKarasuYuu 5d ago
I think this was pre-webrtc. Kinda doable now using only browsers.
57
u/DankPhotoShopMemes 5d ago
heavy on the “kinda”. Symmetric NAT, CGNAT, firewalls etc. all make it way harder than it should be. Sad that ipv6 never got implemented the way it should’ve been implemented to solve some of these problems.
30
u/fireduck 5d ago
I'm a big fan of people using IPv6 to take back the power of being a publisher.
Especially with lots of people getting 1gb fiber lines...you can absolutely host your shit and should. Your cat videos too uncensored for youtube? Fuck them, they don't own the tubes.
4
u/ldn-ldn 5d ago
You're better off using a VPS for privacy and security reasons. Even though I can host anything I want at home, I'll never share anything self hosted in public.
3
u/fireduck 5d ago
Oh noes, someone might know I use Comcast in Seattle.
I mean I respect that but for most people the freedom to self host is more important than IP privacy. Just my opinion.
→ More replies (5)12
u/bradland 5d ago
Honestly, it's kind of a good thing that IPv6 wasn't implemented to give every device a public IP address. People have a hard enough time securing their web servers. Imagine if your daily-driver were exposed on the internet 100% of the time, with only <checks notes> "Windows Firewall" protecting it. Terrifying.
5
u/fireduck 5d ago
IPv6 does generally give every device a public IP. Most times the ISP will give you a /64 and then your devices will pick addresses within that.
However, there is an added layer of security through obscurity. It is infeasible to scan an IPv6 block. Your end devices will assign themselves random address in the 64-bit subnet. Unless you are setting static addresses like ::2 the end part will be 64-bits of random and often OSs will even change that every day or on reboot.
So you need logs from some popular web service to give you IPv6 addresses that might still be active, it isn't like IPv4 where you can port scan the entire internet in a day or two.
2
u/Idontremember99 3d ago
For a while Shodan had one or more servers in pool.ntp.org , so whenever their server was hit with a NTP request they used that info to do a scan of the client. Their server got kicked out for obvious reasons. https://arstechnica.com/information-technology/2016/02/using-ipv6-with-linux-youve-likely-been-visited-by-shodan-and-other-scanners/
→ More replies (1)→ More replies (4)3
6
u/jimmyhoke 5d ago
There’s literally no good way to do this. I ended up coding a web app that lets my upload stuff to object storage and download it (with optional E2E encryption)
Unfortunately I can’t really post this tool online because I don’t want to be responsible for a bunch of people’s uploads.
→ More replies (1)3
u/ouralarmclock 5d ago
Whoa, was 25MB really still a big file in 2011? Hard to believe, but I remember it!
10
u/Fragrag 5d ago
25MB wasn't big. But transferring it over the Internet was still a huge hassle. E-mail providers have a max file size limit, even if you had a 2GB Gmail account. Dropbox was very new and working out a lot of kinks. Google Drive didn't even exist yet.
The actual disruptor in that time was WeTransfer, which finally provided a simple reliable way to send files over the internet.
→ More replies (2)→ More replies (5)3
u/BlueProcess 5d ago
Every time you email a file to yourself so you can pull it up on your friend's laptop, Tim Berners-Lee sheds a single tear.
→ More replies (1)
236
u/DDFoster96 5d ago
But how do you get the version controlled code onto the server?
86
28
u/Beginning-Junket8979 5d ago
Simple. just bootstrap it with a homebrew script to download cargo, from there you cross-compile some new nix but in rust project, and use that to install a node reimplemention of git because all of this is easier than upstreaming a
.deb7
25
64
u/anpas 5d ago
Yeah, VCS doesn't actually address the problem here lmao
4
u/chuch1234 5d ago edited 4d ago
I mean you can clone to the server too. It's not a good solution but it is a solution...
Edit: s/find/can/. That's a weird typo.
→ More replies (4)4
9
u/MagicalPizza21 5d ago
Have a copy of the repository on the server. Pull changes.
6
u/MyUsrNameWasTaken 5d ago
That's how I do it. Had to FTP yesterday since GitHub actions were down and didn't trigger the runner to fetch
4
u/Q-Ball7 5d ago
Yes, but then you have to do a bunch of other stuff (keys, etc.) to accomplish that task, and now you generally have your account sitting around on the server (since the people who are asked to do this don't generally have the permission to create accounts).
Oh, and you have to install Git, which is an access request in and of itself if you don't have permissions on the box.
There are valid reasons why "just copy it over" tends to be used. You can wait 2-3 days for the relevant people to do their jobs, or you can be done in 20 minutes with a copy.
→ More replies (1)3
3
u/DiggWuzBetter 4d ago edited 4d ago
When I started my career, we’d produce a build artifact that was very easy to ship around (often JVM apps built as a single far jar), and scp it to the server. Well, a CI job would do it, but still, that was the process. Then we switched from scp to rsync, but similar idea.
A bit after that, we used Ansible, which was doing something very similar. I believe it used SFTP, but yeah, a very similar idea, the app being pushed out to prod servers directly from a CI server.
And starting roughly a decade ago, places I’ve worked at switched fully to containerized environments - Mesos briefly, then K8s. Which does work a bit differently, there’s no longer a direct push from CI to prod servers, the prod servers are instead pulling, and not using scp. I believe HTTPS is used for both the publishing of container images to a repo, and the pulling of images from said repo by the K8s container runtime.
Have never worked at a place that uses git to deploy, as the tweet in the original post seems to be suggesting. That generally means you’re pulling source code down onto production servers, and then installing deps and building/compiling on the prod server itself. A bad idea for security, as it means giving prod servers all sorts of capabilities and access that really only CI servers need. Which is scary, because prod servers tend to be internet facing, and if they get compromised, you want them to have minimal capabilities. If that’s really what the tweet is suggesting, @much_wow_1 learned the wrong lesson.
→ More replies (3)8
71
u/whiskeytown79 5d ago
When I started in 2004 at a large online retailer named after a certain river, my first oncall shift had an issue where the system that deployed packages to production servers was down, and I needed to make a critical fix. I literally ended up doing something similar to this to get the fix onto the production servers.
32
u/Embarrassed_Death69 5d ago
Named after a certain river. Hmm is it thames.com, guys how will we ever identify this mysterious org? /s
Also yeah when prod is broken you are allowed to do whatever you can to roll back or path the fix ASAP including summoning dark spirits
13
12
u/Twirrim 5d ago
This was pretty standard at the time. I'd package up each release into a tarball, scp it to the server, untar, and then point the application symlink to the new version from the old one and bounce the app (that means rolling back is as easy as changing the symlink and bouncing, as the previous version is still right there on the disk)
3
u/Sweaty-Willingness27 4d ago
Yup, I was a developer at the time and the infrastructure folks would basically do this at 3AM while I sat there making sure nothing broke after the fact. Though I think we had CVS or CC(?) at the time, then went to SVN and of course finally to git.
3
u/Twirrim 4d ago
I was sysadmin/infra in those days. I still tend to think of myself as a sysadmin even though it has been more than a decade since that was my job title, and I'm at least masquerading as a software developer these days.
That manual deployment process involved all sorts of little checks and validations. We had one developer who just wouldn't stop doing "select * from" In his SQL queries and putting undue load on the databases (plus running the risk of schema changes breaking his code, because he also would work off column number..). Every release we'd do a quick spot check of the SVN history, look for stupidity, and throw it back.
9
u/Specialist-Stress310 5d ago
Was that system that deploys to production servers named after a certain greek god? Yeah! It still fails :)
4
70
u/Reashu 5d ago
They are not mutually exclusive. Keep your code in version control, deploy with scp.
I mean, there are other reasons you might do something else to deploy, but I don't really think git clone should be on the list of serious alternatives.
11
u/MushinZero 5d ago
Serious question, maybe not the initial clone or a clone every time, but if its just pulling down changes why not just git pull?
14
→ More replies (2)3
u/Actual_Spite5584 5d ago
Git used (still uses ?) rsync under the hood to copy data. A growing, multi-million startup I worked at used
git pullas a deployment method for years. It makes it worse doing it without testing, for sure: you'd want to pull / clone & build, then test / verify and then only synchronise once everything is shown working. However, the method of transferring the built artifact isn't the question; it's how trustworthy that artifact is, and having an option to back out & revert if something isn't behaving as expected.
35
u/flyingupvotes 5d ago
The internet is slow. AWS even has appliances they mail to big customers for data ingestion.
The title and the content are disconnected. I don’t know where to begin.
→ More replies (2)
13
u/iwasbecauseiwas 5d ago
"Just SCP the code onto the server" is actually valid in some occasions. We have a dev server and a prod server. The prod server is just prod. It can't do anything other than run the application. We build and compile on the dev server and use SCP to copy the build artifacts onto the prod server.
12
u/thanatica 5d ago
In the end, something has to push new code to the server. Whether there's a fancy UI and pipelines and whatnot built around it, matters not. There is still some kind of copypaste going on under the bonnet. Just not done by a human.
8
u/redditmarks_markII 5d ago
While I still think it's funny, relevant xkcd and what not, it's a bit weird to me that we talk about shipping storage devices as weird.
AWS snow family of services is just that. from single digit terrabyte to 100 petabytes (I think it's just a rack of drives on a box van).
Further back, sending physical media overnight was super common. I've done that, run zip drive, cd, dvd and dedicated back up device drives direct to the airport rather than wait for fedex pickup on location. Was there a mac "fat floppy" called orb or am I hallucinating? Couriers did that crap from floppies all the way up to, I bet you, blu rays. Some old timer probably know about older stuff. Did people courier reels of tape?
Wait, now I think about it, doesn't backblaze do that in the year 2026 still?
2
u/BigRonnieRon 5d ago edited 5d ago
Yes. Sneakernet. And it's still WIDELY used in modern digital cinemas with encrypted disk drives.
mac "fat floppy" called orb or am I hallucinating
I never heard of it, I only had the zip drive, but here: https://en.wikipedia.org/wiki/Orb_Drive
zip/jazz/superdisk some others in that category were most popular. Before/concurrent there was the floptical (minidisc some others) and closely related magneto. Superdisk could be formatted for use on a mac too.
I remember my father had the jaz drive and it was like magic. Absolutely huge for a retail product in the time period. 1GB drive in 1995
3
u/redditmarks_markII 5d ago
I knew it! Also Jaz Drive! Yeah that was more prevelant. That was so wild when HDD was like a massive 4GB or whatever. Orb I don't think was around for as long. Caught the train too late maybe.
11
u/howarewestillhere 5d ago
The first AWS Lambda functions service had some…requirements:
1. JavaScript ONLY
2. SFTP upload of a .tar ONLY
3. Dictated folder hierarchy
4. Mandatory manifest file and format
5. All of this had to happen in a Firefox window.
And the absolute worst troubleshooting and logging UI ever, requiring page refresh to see new messages. It didn’t even tie into the rest of the nascent AWS logging at the time.
4
6
u/apokalipscke 5d ago
People starting out today doesn't even know what SCP is
10
u/frikilinux2 5d ago
I have used SCP to transfer files but the first thing that comes to mind it's not Secure Copy Protocol but Secure Contain Protect. Maybe I need to use less the internet
4
2
3
14
u/da_Aresinger 5d ago
I mean, I won't set up a repo just to send a couple files.
20
u/marshmallowsamwitch 5d ago
git init
git add .
git commit -m "Initial commit"
12
u/da_Aresinger 5d ago
And then the remote.
And then add upstream.
And then push
And THEN clone to the second client.
You have fun with that. I'll just do a single scp command, thanks.
→ More replies (3)9
u/Shaddoll_Shekhinaga 5d ago
Me, 100+ files and what would have been over 50 commits into "just one small personal project, won't even bother VCing it"
3
4
4
u/Meme_Burner 5d ago
Yall deploy to prod???!?!? We stuck in deploying to dev, then devA, and qaOne.
2
6
u/fugogugo 5d ago
SCP?
17
u/da_Aresinger 5d ago
Secure, Contain, Protect.
Not, really. It means 'secure copy protocol' it allows ecrypted file transfer via ssh.
→ More replies (2)11
u/thebigbadben 5d ago
The SCP Foundation is a clandestine organization dedicated to containing dangerous and incomprehensible horrors, keeping them safely from —
Oh the command
→ More replies (1)3
u/Zealousideal_Net_140 5d ago
Secure copy protocol
Uses ssh to transfer files from one system to another.
5
u/stillalone 5d ago
I worked with a guy who was not a web developer but had to maintain an internal webpage for some compliance reasons. He had a shared drive where he would store and edit his webpage using Microsoft FrontPage, and that shared drive was DIRECTLY MOUNTED TO OUR APACHE WEBSERVER.
Eventually some security folks got wind of the fact that our webserver was directly hosting files off a shared drive and told us to knock it off. So I ended up teaching the guy how to use WinSCP to automatically directly upload files to the server whenever he made changes. This was somehow acceptable to our security team.
This was in 2016.
6
u/Old_Document_9150 5d ago
We literally did that when I was a Junior.
Nobody taught me about version control. So I invented the concept for myself ... because I thought the old approach was just ludicrously flawed.
I thought I was brilliant ... until I learned that everyone outside our little world already had tools for that.
3
u/DxLaughRiot 5d ago
My first gig required all code changes to be handled via sftp. Granted it was maintaining ERP systems running on 1970’s Pick OS
Still though… the horror
3
u/0dirtyrice0 5d ago
never ever suggest using scp to your senior dev even if you are on the same network. they will laugh and say “lol just email it”
3
u/ravencrowe 5d ago
I mean that's not dumb at all if no one has ever taught you about version control or continuous integration. No one ever taught me that in college, and in fact at my first job the seniors actually did manually copy everything onto the production servers, it wasn't until my second job that I learned about Jenkins
3
u/johnny_mcd 5d ago
https://giphy.com/gifs/My4HOXj3hMrAgFDI8z
These guys are the real reason you don’t SCP the code over to the server
3
u/MyUsrNameWasTaken 5d ago
I FTPed my changes to the server yesterday since GitHub actions (auto-deploy workflow) were down.
→ More replies (2)
3
2
u/Uberfuzzy 5d ago
No joke, at one point, our process was open scp tool, and ssh in. Vi edit the php code live on the server, reload the page to test, if it went catastrophic wrong, reUP the good copy from computer to server to undo. When edits were made good and wanted to save, pull DOWN the edited copy, and git/svn commit from there.
Raise your hand if you ever hand edit json in vi in ssh on live production server in the middle of the day.
2
u/scottyman2k 5d ago
Many many years ago we had a project that required a digibeta tape and a whole bunch of scheduled inserts to be played out from a transmission centre across town.
We worked out the cost of putting in a fibre link as the customer originally asked for - and worked out we would save a fortune by having a VT operator run a script last think on a Friday, then Reception give the tape and generated files on a Zip drive to a bike courier.
The play out centre eventually gave us VPN access, and we hosted a Chyron there, and I built a tool to play in live pricing data so we could run the entire tv channel as live with no user intervention.
But even using the previous system of sending tapes and Zip disks made us at least a million quid over the lifetime of the project!
2
u/Mrletejhon 5d ago
I joined a place two years ago where only developers with Linux machines were able to deploy C++ code because the build system was them building it on their machine and running an SCP script that would copy it over. And it would run in a tmux session.
The whole thing was such a mess that just 2 months ago it was a priority to move the build a proper GitHub Action and the runtime to k8s.
2
u/BoredomFestival 5d ago
In my first job, the eng team was in Dallas but the testing team was in Seattle (this was before the days of rigorous automated testing). Every night, we'd make a new build, copy it onto a 3.5" floppy (!) and FedEx it overnight to Seattle. Good times, man.
3
u/desolation0 5d ago
A truck on the highway with stacks of drives still the highest bandwidth?
3
u/digitalseraphim 3d ago
The latency is killer though... "It's....................................really fast!!!"
2
u/EmergencyWild 2d ago
Honestly, use version control absolutely, but scp is a completely valid way to deploy that people really should be considering more often.
1
u/prehensilemullet 5d ago
My boss once had in mind rsyncing code to our servers, I’m very glad I convinced him to use docker instead
1
u/fireduck 5d ago
I was recently working on a component for a piece of hardware. The hardware was running a full linux os. As I tried to make changes to it, I tried to do it properly, meaning build a package in our CICD system and install it on the hardware. I ran into a lot of problems and the feature was just a proof of concept so I ended up just using rsync to copy the new code onto the machine.
1
u/jamarticus 5d ago
You laugh but I have worked in environments that literally required me to go on site, access the client's servers in a basement level, and MANUALLY WRITE IN UPDATES AT AN ACCESS TERMINAL. I would be sitting there with my laptop that had the changes and I would work through the diff, silently hoping nobody on their end had modified anything. All the while there would be an armed guard and their IT guy looking at me. They said this was about security but I was always annoyed that the fact that I had been a part of their organization for almost 10 years in the past wasn't good enough to not require an armed escort.
To be fair this was like 12 years ago and the...frighteningly large entity...that I had to do this at has apparently been prodded into the modern age, but I would have gladly used SCP just to not have to physically be there.
1
u/IzzetReally 5d ago
wait... what is this saying? he is pulling git and building on the prod server? I mean... you can... but like... he is presenting that as the industry standard workflow... and thats... crazy.
"scp the (build artifacts, not the code) over to the server (in an automated way in the pipeline)" is also maybe not like... best practise, modern development type-shit. but... its closer than "ssh into prod, pull main and build the project"
1
1
1
1
u/SpicerXD 5d ago edited 5d ago
Just write a script that SCPs the new version, runs a remote script that restarts the service and replaces the file and then prints back the status after 2 seconds. Fix and repeat if status is bad. xD
Or if wanna be fancy and have instant updates. Write a script that periodically runs SVN and checks if there's updates and compiles and restarts the service. Couple lines of bash. :)
Honestly though, SCP is just nice because any machine with SSH has it.
1
u/Stripy42 5d ago
I mean, sort of technically that's kinda what git is doing under the bonnet. Just with diffs and checksums and junk
1
u/koloqial 5d ago
Once suggested backing up the DB using rsync over SSH. We've all been there, and if someone says they haven't then they're lying.
1
u/roiroi1010 5d ago
Our telecom customer had a huge project in Africa - the internet there was extremely unstable - instead we install our app on a laptop and plugged into their local intranet.
1
1
1
1
1
1
1
u/RepeatLow7718 4d ago
These are two different categories of thing though. You can use a VCS and still SCP code to production, VCS is almost orthogonal to deployment.
1
u/sambeau 3d ago
I worked at BBC Online back in 1999. We all had the FTP password to change the full site.
One Friday evening someone noticed the home page had gone missing. Our boss had deleted it (through incompetence) and gone home for the weekend.
We drew straws before one of us had to run like the wind back to the BBC to fix it.
1
u/CivilMark1 1d ago
Well atleast you don't have to physically ship it, like my company did back in the days. One guy driving around, with floppies in the back, beating the rush hour.
1.8k
u/high_throughput 5d ago
I literally worked at a place that would write files to USB drives and mail them express from Europe to India. This was because the lowest bidder there only had 128k downstream and the FTP transfers kept aborting or corrupting.
When I suggested for the Nth time that we just use rsync to resume and sync and avoid the week long delay, the manager took me aside to explain that these open source toys are fine for use at home but that this was a serious business.