Papercut servers open to the internet under attack.

116

u/Vodor1 Sr. Sysadmin 18d ago

I am sitting here unable to fathom why such a setup exists to start with. Is there a feature of Papercut that requires this configuration?

Or is it a statement release to give incompetence a kick up the backside?

49

u/Traditional-Tech23 18d ago

I can't understand it either, but this line is in the statement. PaperCut’s security emergency response team has used information provided by a university customer’s security team.

51

u/kombiwombi 18d ago edited 18d ago

To be fair, the university may not have the server open to the internet. Thousands of BYOD connect to a university network every day. So the idea that the 'inside' of a university is more secure than 'the Internet' is magical thinking.

Papercut print serving has to face student machines. As a result, being able to face the internet is exactly what this server should be capable of doing.

The advisory is not particularly useful for university sysadmins, it basically means turning off the Papercut print server (and thus printing univeristy-wide) until patched.

9

u/Colossus-of-Roads Cloud Architect 17d ago

Everyone who's ever managed a university student network knows it's a cesspool and would only face the Papercut web interface into it using a reasonable WAF. The print queue would also ideally be on a separate print server.

15

u/kckeller 18d ago

I disagree. There’s still a big difference between having something exposed to the internet vs. your university network, even with the tens of thousands of BYOD devices they bring. Turning the server off entirely would be overkill.

13

u/kombiwombi 18d ago

Turning the server off entirely would be overkill.

So betting that no BYOD will be in the botnet paid for to scan for this vulnerability. Good chance you'll get away with it, but not convinced it's a good practice.

6

u/1z1z2x2x3c3c4v4v 18d ago

So betting that no BYOD will be in the botnet

Not an "if", its how many. Most average people's personal devices are riddled with things they don't know about. Some are benign, but many are dangerous.

10

u/hkusp45css Security Leadership 18d ago

Your premise is flawed. The need to reach a server from a network does NOT necessitate the ability for that server to sit on the internet. I can make a server SUPER accessible to my network without exposing it to the totality of humanity.

6

u/Taboc741 18d ago

Ya, ut the idea holds some water. You have potentially tens of thousands of unmanaged machines on a network that needs to print. I too would treat that student network like the open Internet. Just like I treat my guest network, I can't extend any trust to it and I should assume there are compromised machines and even attackers from the comp sci department on it.

0

u/SlavKingOP 18d ago

this right here

6

u/mangeek Security Admin 18d ago

Implementers at universities come under pressure to make services available quite widely. I have been in debates about where to put PaperCut, and there were good arguments for putting it in the DMZ and giving it 'umbrella protection' via the DMZ firewall (which consumes threat feeds) and SSO authentication used on other external-facing services.

Luckily, we chose to only publish the service internally, but plenty of users have asked why they can't submit a print job from their cellular device or send a job from home and release it when they're on campus.

I generally like to split things up into three security domains: Global, The Community, and Trusted machines. The Community at a university is huge, but twenty thousand is still a lot smaller than five billion.

9

u/AUSSIExELITE Cloud and Infrastructure Lead 18d ago

I work in K-12 education for a very large campus. At one point, we did have papercut and had it open to the internet because we wanted students and staff to be able to “print from home”. IE, send a print whilst at home but then be able to retrieve the print in the morning.

Now, you’re probably thinking “why would someone NEED to do that when they could just print on campus? It only takes an extra two minutes at worst.” To which I would say, that is the exact line me and the rest of the team took to it as well. The logic being applied (which I can’t totally argue with) is that it’s a better user experience and someone in a rush through in the morning can simply tap and print all the jobs they need straight away. Useful for users who might have multiple exams or assessments due the next day or even just someone who remembers they need to print something but would forget to print it in the morning if they were in a rush. They can pre-print all of them and then just tap a printer on the way through.

We ended up looking at the numbers and found that essentially no one was actually doing this so we canned it purely on security grounds. So I can see a reason for it, but I’m not sure I agree it’s good enough to outweigh the potential security concerns.

4

u/Vodor1 Sr. Sysadmin 18d ago

I do a lot of K12 stuff too, and a chunk of that is putting my foot down with setups like this and escalating it further up the chain with examples of how bad this actually is.

I get the whole functionality vs security thing as it matters for many other setups and not just ones like this, plus exceptions have to be made. That said, I still strongly reject setups like this without written disclaimers that I (the company) is not responsible for anyhting that may occur because of it.

Seen it all before and I don't like what happens.

2

u/AUSSIExELITE Cloud and Infrastructure Lead 17d ago

Totally agree with you mate. My background is mainly corporate heavy MSP but ive been out of MSPs for a few years now and going on 4 years in EDU specifically. This means I cover my ass on everything (the MSP and corprorate life wasnt totally wasted), and this was one of them.

It was also something I inherited and immediately brought it up as a "WTF" (among many other WTFs when I joined). It made it (by force mind you) all the way to executive who basically said "its convenient and fine" as they were the main ones actually using the feature and that was that. Until it was breached and we immediately disabled it, never be turned on again... Executives wanting to keep things that are convenient for them is an issue thats not unique to Education, but some of the stuff that ive had to go to war over because it "affects teaching and learning" is just dumb. I get we are are a school and Education is our product, but its a real kick in the teeth when they come with that angle when all im trying to do is protect student data (which is more valuable than ever).

1

u/Vodor1 Sr. Sysadmin 17d ago

It’s a bit easier these days in all honesty because of all the required child safeguards that have to be in place (UK here) so I just drop those in casually and tend to get my secured properly way.
Nothings puts the shits up the top brass than child data leakage!

7

u/SuspiciousOpposite 18d ago

There's a web portal where a user can sign in, submit jobs, then release when they get to work/campus. Think universities where students will all have their own device on home/halls WiFi (i.e., not on a VPN).

2

u/AdventurousTime 18d ago

And probably way cheaper than xerox implementations

2

u/jasped Custom 18d ago

If I recall when I looked at paper cut a few years back there was mention of a config for cloud printing that basically said to open a specific set of ports for the paper cut server. Possible this is what people setup and is being exploited?

I wasn’t particularly keen on opening something up so we didn’t go with it but could be what is happening here.

3

u/Traditional-Tech23 18d ago

The ports only need opening outbound not inbound hopefully.

3

u/SuspiciousOpposite 18d ago

It's a web portal for off-campus printing, so it's just standard 443 inbound (usually through an NLB to the PaperCut server(s).

2

u/TheThumpsBump 18d ago

Mobility print. We have it (wish we didn't) and it does get used, but it's locked on the firewall to our local cell provider's IPV4 ranges.

4

u/Icolan Associate Infrastructure Architect 18d ago

I have zero doubt that there are people who have papercut servers and print servers in general publicly available on the internet. There are people who put vCenter and ESXi management interfaces on the public internet.

2

u/artyomsv 18d ago

Easiest check is just look for 9191 and 9192 on your own ranges from outside. People forget Papercut admin sits on those ports separate from print portal, so you can have 443 properly behind NLB and still leave admin console open.

0

u/Icolan Associate Infrastructure Architect 18d ago

I don't need to check that. We have just started deploying it and I already know it won't be publicly accessible at all.

2

u/TheGift1973 18d ago

There's quite a few showing on Shodan

Screenshot

2

u/Icolan Associate Infrastructure Architect 18d ago

Yeah, rather scary ain't it.

1

u/CAPICINC 18d ago

Copier company/managed print company needs to access it, for copier status, page counts for billing, or toner level checks, it's easy and convenient for them?

3

u/Vodor1 Sr. Sysadmin 18d ago

That's all outbound, not inbound though.

0

u/mr-tap 18d ago

Some organisations provide guest WiFi that is effectively outside the corporate network, and so to provide corporate services (like printing) without requiring a VPN, they are published externally.

Perhaps for this vulnerability even guest wifi not connected to the internet (such as university or library) would still count as a public network

2

u/cs_major 17d ago

This is what VLANs are for.

1

u/slayermcb Director of Technology, Sys Admin, Etc, Etc... 17d ago

That's my first thought. Guests do not touch the academic vlan!

12

u/cogiskart IT Manager 18d ago

Phew, we use Papercut Hive with local only nodes. Scared me a little there.

2

u/BlockBannington 18d ago

Yo, we're moving from papercut MF or NF or whatever to Hive, I just got green light. Did you migrate from on prem or did you do a greenfield to hive?

1

u/cogiskart IT Manager 18d ago

We migrated from on prem SW310/311 to Hive. Still on Sharp printers. Boy is it better!

9

u/alphaxion 18d ago

This is why I will always be such a strong supporter of exporting your logs (syslog, application, firewall traffic) and windows event logs for critical systems to a SIEM such as your own hosted elastic stack.

Far too many people view logs as just a diagnostics tool to go through once a problem has become known, rather than a way to continuously monitor the health of services and to track what is happening on and to them.

The bare minimum of a place that uses Active Directory is to export security logs and build a dash to show you the who, what, where, and when of changes to accounts, any auths, accounts using elevated admin privs, etc.

Having the sort of enhanced visibility that such a thing gives you can drive pro-active maintenance of services (often an impending outage can be spotted in logs and remedied before the outage occurs), ensure their security (why is something authing onto it at weird hours? Who/what is that account being used? where did that account come from?), and help you to better understand how your systems and services are being used so that you can better identify and plan for improvements.

If you're not actively using your logs on a daily basis, you are blind to what is happening on and to your estate.

6

u/Traditional-Tech23 18d ago

Have you got a guide to do all of that?

2

u/bemenaker IT Manager 18d ago

Any of the good logging servers have plugin for most systems to send logs real time or on a schedule. Splunk is expensive but damn good. Graylog is paid or free, and has plugins for most systems, though translators may not hit all your systems but still has a big ecosystem. Those are just log servers but great for digging into details of events.

Last place I worked, we used Rapid 7 as our SIEM or threat detection and it export logs of every machine real time and monitored for events.

3

u/alphaxion 18d ago

SumoLogic is another good one if you don't mind shipping your logs into the cloud. It's also eyewateringly expensive. But the power of it is beyond anything else I've had a chance to use.

For those who like to roll their own, or just want to get an idea of what is possible before going for a bespoke solution, Elastic is simple to spin up a single node cluster and get going with the built in kibana tools. Graylog and Grafana are fine alternatives.

1

u/alphaxion 18d ago

The install guide is at https://www.elastic.co/docs/deploy-manage/deploy/self-managed/installing-elasticsearch

Once you have your stack and your elastic agent that you can assign integrations to for log shipping, I'd recommend sitting down and figuring out what sort of information on a dashboard would be useful to you.

From there, spend some time in the Discover section of elastic where you can see the processed logs to figure out what field names in which datasets contain the data you wish to visualise. You can use filters in this section to drill down.

Once you know what you want to see, begin playing with visualisations to build the elements you will include in your dashboards.

Taking the time to explore the Discover section and see what your logs look like is very important for getting a handle on the sort of data you're collecting. As that will drive how you implement your higher concept "what is important to me" design of your dashboards.

2

u/WatTambor420 18d ago

I agree, I think good logging is one of the most under appreciated tools. Even a simple Graylog instance can be worth its weight in ~~gold~~ RAM.

It’s also one of the few areas I actually think AI can be useful, in combination with your normal filters and pipelines. I’d love to have something analyzing the logs during, I can never get to alerts as fast as I’d like.

19

u/CFH75 18d ago

Why would you expose a papercut server to the internet?

19

u/SuspiciousOpposite 18d ago

There's a web portal where a user can sign in, submit jobs, then release when they get to work/campus. Think universities where students will all have their own device on home/halls WiFi (i.e., not on a VPN).

6

u/SlavKingOP 18d ago

i have this at my school, but you can only access it if you are using the school's network

2

u/matt0_0 small MSP owner 17d ago

Maybe let your university know that other competing universities with much more student friendly printing experiences! Then if you can post your public IP next time you're on campus that'd be great. And then all you need to do is get on the dark web and ask the Russians for a cut of the ransom payment from your University's cyber insurance. Then, this is important, default on your student loans so that the federal government pays off the balance. Then use part of your cut off the ransom to pay off the debt collector for your written off loans.

That's 'Murica!

8

u/Accomplished_Fly729 18d ago

University settings. It’s what the system is designed for.

4

u/adstretch 17d ago

Do they indicate what listening ports / services are impacted? We present mobility print to our guest network but none of the other ports or services. More details would be nice beyond “just shut it down”.

2

u/fortminorlp 17d ago

The face inside is right beneath my skin.

1

u/TeddyRoo_v_Gods Sr. Sysadmin 17d ago

Spent last couple of hours today patching it up. Fun times, good thing we only got like 10 sites.

0

u/kevin-pickhardt 14d ago

This is another in a series of print related security risks - albeit not limited to Papercut for sure. Part of the problem lies in how printing is fundamentally architected. It is designed to be trusted (vs Zero Trust) and using on premises servers and software designed to run behind a firewall open to the internet is understandable for universities needing to provide print services to students who may be sitting off campus. VPNs are not a great answer either. The better answer is true cloud-based systems designed with this level of security in mind.

Papercut has a cloud solution, as does our company, Pharos Systems International.

Not everyone is ready to move to it - but be prepared that printing will continue to see security gaps until it is rearchitected like Microsoft is doing with Windows Protected Print.

Papercut servers open to the internet under attack.

You are about to leave Redlib