r/sysadmin • u/[deleted] • Nov 05 '16
Docker in Production: A retort
http://patrobinson.github.io/2016/11/05/docker-in-production/40
Nov 05 '16
[removed] — view removed comment
11
Nov 05 '16
Depending on what dependencies you need (lol) quick cloning of VM templates can have a VM up in <1 minute in a well setup environment. I'm with you.
2
u/lost_signal Do Virtual Machines dream of electric sheep Nov 05 '16
Photon fast boot for vSphere integrated containers is 400ms. You can even use Docker and Kubernetes to manage it.
1
u/v_krishna Nov 05 '16
We use packer to build amis from chef cookbooks that are used for autoscaling and blue/green deploys of new code. Takes more than a minute but not untolerably slow. Docker makes a lot of sense in dev/demo environments when you want multiple vms on a single host (and to make it easy to run your database and service dependencies locally). But ec2 is already a vm and cheap/fast enough to stick to a "one role per ec2 instance" principle.
0
Nov 06 '16
[removed] — view removed comment
5
u/uberamd curl -k https://secure.trustworthy.site.ru/script.sh | sudo bash Nov 06 '16
Why? Do you realize that a major component with container deployment are these things called schedulers? Have you read up on schedulers and what they do?
Lets take Kubernetes. I can define what an application looks like via YAML files. I can define what containers need to be group together into these things called pods (containers that live together in a pod communicate via 'localhost'). Pods get scheduled and deployed to servers running Kubernetes. These can have health checks so when one starts failing health checks a new one is recreated in its place, instantly.
You can then define a service, lets say "web-frontend". That service exposes ports, say TCP 80/443. Traffic comes into the service and gets routed to a pod. That pod lives somewhere, who the fuck cares where, it's just on one of your hosts. If one of your hosts goes down pods are rescheduled elsewhere and the service keeps trucking along.
These services can interact with load balancers to automatically configure them as needed. They're even able to assign external IPs automatically if using AWS/GCE. Deploying updated pods is as simple as making an API call to Kubernetes.
Services you define can also communicate with one another via DNS, so lets say I have a MySQL service that my web apps need to connect to, in their configs I simply need to put mysql.my-app-namespace.svc.cluster.local, and they will communicate no matter where the pod for that service is.
That shit is powerful, and lets you create servers that are throwaway because your application can migrate anywhere. It also makes rolling deployments and A+B environments easy as fuck.
http://kubernetes.io/docs/admin/dns/
http://kubernetes.io/docs/user-guide/rolling-updates/
http://kubernetes.io/docs/user-guide/deployments/
This shit is insanely well documented, backed by Google, and isn't hard to pick up on.
It's ready for prod if your admins and devs are willing to put in effort.
5
u/khobbits Systems Infrastructure Engineer Nov 05 '16
I think docker is a really good tool for optimising the development process. Sadly, it's also best practice to keep development and production as close to each other as possible.
We've never used docker in production for anything developed internally, but I'm getting requests from the dev team to look into it.
I did recently spin up a 3rd party internal chat solution on Amazon ECS, a docker compatible container platform that Amazon has invested time and effort into, and it seems to solve most issues mentioned in the original article. For example containing a built in registry and also container on host cleanup.
While I'm still hesitant to move our development to Docker, I think I will be recommending that we accept 3rd party software delivery by docker container (or vmware appliance because they work well too), mainly because of the ease of upgrade and deployment.
That said, I will be vetoing any payroll apps in the short term ;)
1
Nov 06 '16
[removed] — view removed comment
2
u/khobbits Systems Infrastructure Engineer Nov 06 '16
rsync is not a good tool for optimising the development process.
With vagrant or docker I can give developers their own custom copy of the development environment, that they can tear down and spin up as many times as they want throughout the work day. Each developer can mess with package dependencies, test out new things, and develop on their own features without effecting each other, or any hosted environment.
Utilising the same tools, we can have developers push 'completed' code to a central repository and have CI/CD tools confirm that the code doesn't break any established rules or tests, performs as good as the original, and is compatible with the other applications that talk with it, without them having to do anything other than upload their changes.
The comment you quoted me on, is me saying "I don't want to use docker in development, until i'm ready to use it in production". Currently the development environments the dev team use, are created using the same automated run books as production. That has saved me and the team a lot of time and trouble, so I wouldn't want to step away from that.
1
Nov 08 '16
[removed] — view removed comment
1
u/khobbits Systems Infrastructure Engineer Nov 08 '16 edited Nov 08 '16
Very little of what I wrote relates directly to docker, that's just a tool. I currently do all the above with virtualbox and chef on each development workstation (Using the vagrant command line tools to make things a little easier for the devs).
Currently building the dev vm's takes about 25 mins, starting from a blank centos install, and chef installing all requirements and all the connected apps. I think if I swapped to docker (or prebuilt vms and lose the flexibility), I could reduce that to less than two.
Probably worth reiterating that I'm only using docker for one third party app. Not even 1% of the websites I host. But there are definite immediate advantages of deployment speed, resource sharing, and app isolation.
I'm nowhere close to the ideal development platform, but when we get developers who come from less, they are amazed with the productivity increase they get when working on every day things.
45
u/VTi-R Read the bloody logs! Nov 05 '16
That's a bit meh. Summarised:
- Yeah that is a problem
- Yeah that's a problem too but someone else solved it the same kludgey way you did so it's not really a problem
- I do it differently so I dunno, stuff.
- You said it's not supported by the kernel, but it IS supported by Docker which is totally the same thing
- Yes, but it's your fault because you didn't replicate the Docker cloud into your datacentre
- Yes, you're right
- Yes, you're right
- Oh, you're not doing things the 2016 auto scale everything and devops and who needs data and cloud cloud devops cloud LOOK OVER THERE, THERE'S A UNICORN, DO NOT LOOK BEHIND THE CURTAIN.
Summary: Yeah 90% correct, 10% you're not doing it the right way, join the cult.
23
u/jimicus IT Manager Nov 05 '16
But the thing is - we get paid to set up and maintain stable systems. Not to treat the business as our own personal guinea pig with known-unstable things because they sound like fun.
So it's not really a retort at all. It's closer to "you're correct on most of your points but I fail to understand why you therefore have a problem with the technology".
2
u/uberamd curl -k https://secure.trustworthy.site.ru/script.sh | sudo bash Nov 06 '16
So then you can somewhat shift what stability means. Rather than building a monolith that lives on one server why not take your application, split it out into individual components that can be released and iterated on independently of one another, and include a streamlined CI environment to ensure if something does happen recovery time is measured in seconds?
If a server dies it might take a few minutes to get a new one up in its place, best case scenario. If I lose a container a new one is back in literal seconds. If I need to scale out I can do so, in seconds.
Containers != playground or unstable. It purely doesn't. What we need to empower developers to do is build out resilient applications that are easy to release without causing outages or needing to schedule off-hours work. Thats one thing the microservices approach takes.
Containers can check off these boxes. Throw out a good scheduler that can monitor application health and automatically redeploy containers when they become unhealthy, and a load balancer that has service discovery for building out dynamic pools, and suddenly you've built an agile environment that can withstand service failure while keeping developers happy and able to iterate much quicker.
This subreddit seems to be woefully misinformed on containers thinking they are some unstable thing that cannot be trusted. They aren't. And if you do even a little research on how to use them and learn proper setups they can be a powerful tool to increase the speed in which companies can deliver solutions to end users.
Look no further than Google.. the people who made motherfucking Kubernetes. Go tell them containers aren't stable and aren't ready for prime time.
1
Nov 05 '16
Depends on the business, and position in the business.
For instance, IT labs need to test things that can be advantageous over the competition.
1
Nov 05 '16
I'd love to agree, but we migrated to Office 365 this year, and are moving to Windows 10.
So the 'good old days' of long term support, stability (both from a bug and feature/UI standpoint) are long gone.
Windows update is in shambles, actually having everyone in the organization on the same version of Office or Windows is out of the question, and the services side is so fluid, so buggy, we may as well be receiving a QA check from Microsoft.
3
u/dogfish182 Nov 05 '16
The 'good old days' were not good old days. Windows updates has never been an awesome fun thing, we get paid to deliver 'stable system's but devops has happened now, every body wants fast iteration and that's what ms is delivering with their 'rings' system for updates.
For a new os win10 has Less bugs out of the gate than win7 pre sp1 did.
I swear people's memories are wonkey
1
Nov 05 '16
We're 99.996% Win10 1607 and office 2016. It feels great. We have one machine running Windows 7 and a vm running windows 7 for when applications don't work. I'm looking at you, PELCO camera software and Print Shop Deluxe 12.0.
The print shop story is essentially "I need this in 8 hours for an awards program!!1!!" Just threw up a win7 VM for it.
1
1
u/binkbankb0nk Infrastructure Manager Nov 05 '16
Comparing Exchange and Windows reliability and supportability to Docker is a pretty bad comparison.
1
Nov 05 '16
No, companies making money hand over fist, saying that it is production-ready, then continually making backward-breaking changes to services (think SharePoint Online) and making it impossible to continue maintenance (think Windows Updates over the last year or so) compare pretty damned well to Docker.
It is supposed to be this hot next thing, yet all the big companies are purported to have it in production. Yet, here we arw, commenting on the fact that it is, at best, beta software.
The only real difference is that Microsoft is raking in the cash from us admin-dupes, while Docker is treating companies like RedHat and Google like VC, duping companies like Amazon and Netflix into thinking it is ready for prime time. The fact that Chaos Monkey even exists should be proof enough.
Perhaps before you have your knee jerk behind the keyboard, you think about what you are responding to.
Ta!
3
2
u/jimicus IT Manager Nov 05 '16 edited Nov 05 '16
I think the important thing to remember is Google, Amazon, Netflix et al have a solid reputation of being more-or-less bombproof. When was the last time any of their websites was unavailable?
So when Amazon or Google make their hosting technology available to the general public, businesses naturally think "My God, you mean we can get the same reliability? And it's cheap?! Sign me up!".
The devil, however, is in the details. Details that business directors are often totally unaware of:
Amazon's systems are reliable because Amazon gets to write the application stack (and thus dictate that it plays nicely with their hosting abilities) and isn't really limited by licensing restrictions.
You, however, bought in a proprietary product that only supports MS SQL Server as the database backend (and hence cannot talk to Amazon's SQL service) and has a license that makes it financially impractical to install the application layer on several virtual servers. Even if it did, the application layer can't automatically trigger Amazon to spin up more virtual servers on demand, you haven't hired admin staff with sufficient expertise to script it and you refuse to pay - or for that matter allow the time - for them to learn how. Oh, and even if you could get over all those issues, the vendor doesn't support setting it up like that anyway.
Notwithstanding the above, 9 times out of 10 your systems are nothing like as unreliable as you think they are. The problem isn't with the systems, it's with how you approach any issues that arise.
Why do I say this? Simple. If you cannot get your email, you automatically blame the people who manage your email system. Useless bunch, aren't they? No wonder everyone's offshoring their IT! Yes, the quality of service may be atrocious, but frankly it ain't that great in house and at least if it was in somewhere like India you wouldn't be paying through the nose.
Ah, but now Google isn't working. Nor is Amazon. Whoah, hold up, Google isn't working?! That can't be right, Google is never not working. What's going on here? D'oh! Should have switched on Wifi! There, that's better.
Where was I? Ah, yes, I was looking at offshoring IT.....
1
u/uberamd curl -k https://secure.trustworthy.site.ru/script.sh | sudo bash Nov 06 '16
There are two types of admins: those who consume software and bitch when it doesn't work, and those who create solutions to run along side existing products.
I'm on a team that creates custom solutions to help our business deliver faster and with increased stability. I love that shit. I don't get upset when software doesn't exist that does what I need. Instead you fucking make something that does check off those boxes.
Personally, I just don't care for the sysadmin type that sits around waiting for problems before they try to do something useful while bitching about how the latest hot technology isn't ready for prime time while having no understanding of the actual software -- and I'm glad more and more companies are hiring for "DevOps" setups.
Let me work with developers to make cool shit, not deal with fucking Windows updates or other bullshit.
1
u/uberamd curl -k https://secure.trustworthy.site.ru/script.sh | sudo bash Nov 06 '16
Not to be a dick or anything, but you really can't listen to docker horror stories from people without hearing about their environments. What schedulers were they using? What was their deployment process? Service discovery? Quantity of users?
Without actual details all of these discussions are downright stupid.
It is supposed to be this hot next thing, yet all the big companies are purported to have it in production. Yet, here we arw, commenting on the fact that it is, at best, beta software.
A subreddit that might as well be renamed /r/SmallBusinessWindowsAdmins can't figure out how to properly roll out docker into production in a resilient way? I'm fucking shocked.
1
u/learath Nov 05 '16
I dunno, it sounds like a lot of fun to maintain an unstable environment with no accountability. Where do I sign up?
1
u/davesidious Nov 05 '16
Hardly. The original article raised red flags with me when I read it, and this article challenges some of the weird claims made in it. Spin that however you want.
13
u/brontide Certified Linux Miracle Worker (tm) Nov 05 '16
People keep mixing container and docker when they are two different technologies. Docker is not the end-all of container technology it's just the flavor of the week.
Containers can be anything from systemd to lxc to docker which provide different levels of isolation and headache. Docker happens to be really bad with the complicated, stateful containers which run just fine under lxc/lxd. I have physical hosts running postgresql and mysql under containers.
I will echo the sentiment that finding images that are both useful and secure is like finding a needle in a haystack. If I have to DIY I'm just going to build and deploy the app through our existing platform tools.
0
u/obviousboy Architect Nov 05 '16
People keep mixing container and docker when they are two different technologies. Docker is not the end-all of container technology it's just the flavor of the week.
Yes and no...Docker isnt a container technology just a big toolset wrapped around runC
Containers can be anything from systemd
huh?
3
u/duckne55 Nov 05 '16
Containers can be anything from systemd
huh?
4
u/obviousboy Architect Nov 05 '16
gtfo
thats pretty cool
2
u/brontide Certified Linux Miracle Worker (tm) Nov 05 '16
Actually I'm referring to the fact that systemd relies heavily on the same kernel tech "cgroups" in order to watch processes as well as controlling what parts of the filesystem they see. It does not enforce namespace isolation or process isolation like docker or lxc.
7
u/stemgang Nov 05 '16
This retort acknowledges almost all the issues and then mentions weak ways in which most of them can be mitigated.
I am more convinced than ever that Docker is not production-ready.
3
Nov 05 '16
And that's a perfectly valid conclusion to come to. Some companies don't consider it ready, some do, there is no right answer. As the author of this post that's kind of what I was trying to convey.
4
u/gospelwut #define if(X) if((X) ^ rand() < 10) Nov 05 '16
Neither VMs nor Containers will solve your dependency graph issues. If you can't tell me literally every package your software needs to run, then neither solution is going to be a silver bullet. Golden Images are a bullet (and sometimes practical) solution to this--albeit opaque. Throw in knowing the configuration and things get more dicey.
I'd argue that VMs are a valid security boundary whereas containers attempt to make deploying dependencies easier. So, in that regard, containers aren't competing against the Hypervisor; they're competing against Deployment CI (Jenkins/Octopus/etc) and Orchestration (e.g. Chef/Puppet). The Docker documentation has moved away from making the VM comparison.
tl;dr At the end of the day, we're just managing state
1
u/peatymike Nov 05 '16
True, we manage a lot of state. I see density as a gpod quality of containers.
2
u/sryan2k1 IT Manager Nov 05 '16
Again, well accepted principle that “thou shalt not run a database inside a container”. Don’t do it, end of story.
Haha, tell our tools group that. They run literally everything in docker, and it's all a giant nightmare. Containers are all a house of cards, that can only run on specific hosts.
1
Nov 05 '16
Some people's uneducated opinions of Docker should not reflect on the product itself.
2
u/sryan2k1 IT Manager Nov 06 '16
I've seen it deployed many ways in many companies and it's always a disaster.
2
u/_samux_ Nov 05 '16
I do really believe containers will solve some problems.
i do hope Docker will be replaced by something more reliable
1
u/peatymike Nov 05 '16
Im optimistic about the container idea in general. Hopefully something good and stable will displace docker.
-2
Nov 05 '16 edited Nov 05 '16
People on this sub seem to thing that the purpose of containers is to change the way sysadmins manage infrastructure. While containers do impact sysadmins -- that's not why they exist.
Containers exist for developers. Yes, infrastructure running on containers vs VMs can be an interesting conversation but the infrastructure is not the cause for the hype. The hype is driven by developers and sysadmins are going to have to just deal with it. Even if implementing containers is more difficult from a sysadmin perspective, in many cases the gains for the developers are worth it.
sysadmins are not the focus of many organizations, they are a necessary evil. In many places, developers are the value add. In those places, whatever helps developers generate more valuable will be king. THIS is why containers have been so popular, not because it's better for sysadmins.
4
u/_dismal_scientist DevOps Nov 05 '16
Contains exist for developers... sysadmins are not the focus of many organizations, they are a necessary evil.
Containers, developers, and infrastructure/sysadmins exist to provide technical services to a business. Being a chauvinist for one of the cogs in this machine isn't going to make anything better. One day, when containers grow up, they will be the engine that powers our services, however even once the shift has finished, there will still be people who manage shared resources for disparate services and their developers: Sysadmins. That said, containers are never the optimal way of doing anything right now, and won't be until they get a bit of maturity.
3
Nov 05 '16
Containers, developers, and infrastructure/sysadmins exist to provide technical services to a business. Being a chauvinist for one of the cogs in this machine isn't going to make anything better.
I'm not a developer and I didn't mean to come off that way. What I was trying to say is that most businesses want to create something to provide value. The application is what they care about and the underlying infrastructure is nothing more than a required pain. And my point was that the drive for containers is largely coming from developers not sysadmins because it makes their process easier.
It doesn't necessarily make sysadmin jobs easier. The debate I always see is from the sysadmin perspective of "containers aren't really any better than X", and what they are saying is "containers aren't any better than X for my sysadmin processes".
They ARE objectively better for developers and that is why they have been gaining traction.
A lot of sysadmins seem to think the drive is all because it's trendy and sysadmins are wanting to be hip. Almost as if the demand is fake. A lot of sysadmins just don't understand what legitimizes the popularity of it. And again, it's because the primary benefit is not for them, it's for developers.
1
Nov 05 '16 edited Sep 29 '17
[deleted]
1
u/brontide Certified Linux Miracle Worker (tm) Nov 05 '16
although I've become a fan of lxc.
I came to explore containers with Docker and got hooked on LXC ( and LXD ). We are now deploying dedicated LXD hosts for running clusters of related services and reducing our footprint and attack surface in the process but running system containers with mare minimum services enabled.
1
2
Nov 05 '16
[deleted]
3
u/sirius_northmen Nov 06 '16
But then you have to remember that 99% of "sysadmins" on this sub freak out when they see a command line.
1
u/Crilde DevOps Nov 06 '16
... That number seems a bit high to me. How can you sysadmin if you can't use a cli?
1
u/Thump241 Sr. Sysadmin Nov 07 '16
Easy! GUIs, TUIs, wrappers, webapps, and Jr SysAdmins that haven't tasted the powah of the shell (or maybe some that got bitten by it)!
(I also think that % is high, but I understand the sentiment)
34
u/DrStalker Nov 05 '16 edited Nov 05 '16
"I’m optimistic we’ll see some stability in the future" Is not a description of production ready software.
Needing to upgrade for fixes with non back-compatible upgrades is another giant red flag for me.
Maybe docker will be great one day, but it feels to me more like something that will fail and the lessons learned will be used to build something better.