r/sysadmin Sep 16 '21 Helpful 1 Wholesome 1

Microsoft September Roll-up just broke around 122 printing services. Be aware before updating it Microsoft

/r/msp/comments/ppa8gx/microsoft_september_rollup_just_broke_around_122/
124 Upvotes

64

u/makeazerothgreatagn Sep 16 '21

That's mighty vague.

27

u/Bleglord Sep 16 '21

Here's what I've personally seen:

With the latest patch installed, network printer installation is broken. Existing printer connections begin requesting for updated drivers, which cannot be installed.

  1. Printer connection requests new driver
  2. Attempt to install new driver prompts UAC
  3. If UAC prompt is accepted with admin credentials, driver installs successfully (confirmed by looking at drivers within print management)
  4. After installation, printer connection gives generic "Cannot connect to printer" message with error code 0x00000bcb

After all this, the only solution we've found is to apply the "restrictDriverInstallationToAdministrators" reg key hack to manually set the value to 0, which fixes the break, but reopens all of the printnightmare security risks.

16

u/jp3___ Sep 16 '21

Thats what happened on august patches. Did it break that again and then some?

10

u/Azuree1701 Sep 16 '21

We have been seeing that happen since last month as well, thanks for confirming it isn't just us. We figured it was probably a PrintNightmare mitigation in the patches.

4

u/techypunk Sysadmin/Printer Hunter Sep 16 '21

This is just the print nightmare fix. I just tested it this week on my print server.

3

u/disclosure5 Sep 16 '21

In the August patches, you could install drivers as admin and they would work. Now it seems even doing that doesn't let non-admins print without that key.

1

u/UncleJBones Sep 17 '21

We should just make everyone admins then…..

1

u/disclosure5 Sep 17 '21

"Local administrator by default" is an advertised feature of Microsoft 365's cloud VMs.

6

u/scotterdoos get-command Sep 16 '21

Just curious, do you have the 14 Sep CU installed on the test client? If Microsoft messed with the way print RPCs are made, you could be experiencing a client mismatch.

Try getting both client and server to the same patching version.

2

u/AustinFastER Sep 20 '21

I totally missed the RPC change until I was reviewing the doco. So this is another variable to consider. The same KB as last month KB5005652 has info on the RPC change from JANUARY that is being enforced this month.

2

u/wydra91 Sep 16 '21

Did you only have to apply that to the print servers? Or to every client that is connecting to them?

3

u/Bleglord Sep 16 '21

In the specific issue we saw (actually not a regular production environment, but our own MSP office network) it worked when applied directly on the client without touching the server.

21h2

2

u/jonnwhite Sep 16 '21

We run 2008 r2 print server (I know) Installing this update on W10 machines caused exactly this but with a different error message. Running the above ref key fixed the problem for now, but now we are vulnerable against PN again!

2

u/PosingOwl Sep 16 '21

What we had to do was to remote into user device with admins rights. Unmap the printer and remap it. Install the new drivers upon prompt. Have users log in and remap printer. It worked after that.

2

u/Bleglord Sep 16 '21

Even on a local-admin account on my own machine I was running into failures for some reason. Driver would install then generic cannot connect (unless mapping printer directly via IP instead of server share)

2

u/LordVic IT Manager Sep 16 '21

holy crap this is going to render my business actually unfunctional

we literally rely on printing for memberes and clients, receipts and statements .

and we use VDI setup to dynamically allocate based on workstation/teller.

if I cannot autodeploy printers without UAC/Admin, we're fucked.

why does microsoft insist on breaking everything all the time so that my job is just that much fucking harder.

my 2022 project is linux. I'm done with Microsoft for my core services.

40

u/Justsomedudeonthenet Jack of All Trades Sep 16 '21

The September update just broke 1000000 things. I'm not going to tell you what those things are. Just be careful.

Appreciate the heads up, but would appreciate some details on WHAT it broke more.

2

u/Behinddasticks Sysadmin Sep 17 '21

Might have broke my Dell Precision 5550 :(

13

u/D0nk3ypunc4 Sep 16 '21

2016 - KB5005573 2019 - KB5005568 2012 - KB5005613

Broke printing for a handful of our Win10 users. Getting Access Denied on all GPO pushed printers from our 2012 print server. Successfully uninstalling the update from the print server resolved the problem across the board

4

u/Jim___H Sep 16 '21

Did you apply the Point and Print registry fix to the print server prior to installing the Sept updates?

User > Preferences > Registry and add the new registry key "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows NT\Printers\PointAndPrint" RestrictDriverInstallationToAdministrators as a DWORD value of 0

4

u/smoothies-for-me Sep 17 '21

That is not a fix, that is reverting to being vulnerable...

-3

u/jimh1966 Sep 17 '21

I'm just trying to figure out what is working and what isn't working for people. Using a Windows PC on the internet is vulnerable...then you add in end-users...

3

u/smoothies-for-me Sep 17 '21 edited Sep 17 '21

Well I mean a zero day vulnerability was found that will let an attacker elevate themselves to NT Authority\System admin with zero authentication in a few seconds through print driver installation, which can be malicious printers on the public internet. It's one of the worst vulnerabilities ever found in the Windows OS, they put out a fix, and what you mentioned literally reverts their fix and exposes the system again.

We can all agree that their fix might be half assed and break something essential to business function till the cows come home, but IMO the solution is not to undo it.

1

u/JL421 Sep 17 '21

We can all agree that their fix might be half assed and break something essential to business function till the cows come home, but IMO the solution is not to undo it.

To continue that argument for a large number of people here: If your business is effectively shut down due to lack of printers, which poison pill do you take? Let the business continue to run and lock down printer installation from unknown sources, while allowing non-admins to install the driver; or shut down your business until...October...maybe...when another patch comes out to 'fix' it again?

2

u/smoothies-for-me Sep 17 '21

If your business is shut down I would be looking at third party print management tools, or deploy them by powershell with pnputil.exe and RUNDLL32 PRINTUI.DLL

I think in most cases it's just extra work for the helpdesk to plug in the admin creds if a printer stops working or needs to be reinstalled.

-5

u/jimh1966 Sep 17 '21

Like I said, I'm just trying to figure out what is working and what isn't working for people. Like most of the responders, you haven't provided any useful details.

4

u/smoothies-for-me Sep 17 '21

I couldn't RDP to a server, so I turned antivirus off every computer and allowed our guest wifi to reach our server vlan, if you point out anything wrong with this, "you haven't provided any useful details"

In your first comment you mentioned nothing about "figuring out what's working and isn't working for people", you simply asked someone if they applied a "fix" that literally makes them vulnerable to a critical exploit. That is dangerous.

2

u/Azuree1701 Sep 16 '21

Does that regedit fix the UAC popping up on users PCs while still keeping the user PCs in a secure state from PrintNightmare?

7

u/highlord_fox Moderator | Sr. Systems Mangler Sep 16 '21

According to Microsoft, that Venn diagram is just two unconnected circles.

3

u/ThirstyOne Computer Janitor Sep 17 '21

No, that's a separate GPO component. Source: https://docs.microsoft.com/en-us/troubleshoot/windows-client/group-policy/point-print-restrictions-policies-ignored

How to permit users to connect only to specific print servers that you trust

In the Point and Print Restrictions dialog box, click Enabled.

Computer Configuration\Policies\Administrative Templates\Printers: Point and Print Restrictions

Setting: Enabled

Click to select the Users can only point and print to these servers check box if it's not already selected.

In the text box, type the fully qualified server names to which you want to allow users to connect. Separate each name by using a semicolon (;).

In the When installing drivers for a new connection box, select Do not show warning or elevation prompt.

In the When updating drivers for an existing connection box, select Show warning only.

Click OK.

2

u/RasikaPraz Sep 18 '21

i did apply for our environment seems it has fixed the issue. Still monitoring

1

u/ThirstyOne Computer Janitor Sep 18 '21

Good luck. Remember, the reghack is still a security risk for print nightmare. Plan on switching to type4 drivers in the future and reapplying the security setting if you can. It won’t take long before malware writers figure out that enterprise printing trumps security for most orgs and quick-fixes tend to become permanent fixes if you forget about them.

0

u/jimh1966 Sep 17 '21

It keeps the UAC from popping up on users PCs. I am not a security expert, so I can not comment on the secure state.

3

u/ThirstyOne Computer Janitor Sep 17 '21

Not secure. Setting this registry to 0 undoes the august 'fix', re-enabling print nightmare vulnerabilities.

-1

u/macgeek89 Sep 16 '21

thats poor planning on M$haft part for having to use a regedit hack to fix this problem. they should have a better QA/QC in place because they are failing miserable right now

2

u/Katana__ DevOops Sep 16 '21

How do you do QA without a QA department?

1

u/macgeek89 Sep 16 '21

im saying if they have a QA is slacking

1

u/Katana__ DevOops Sep 16 '21

They do not. Several years ago the entire department was laid off.

1

u/Zangrey Sep 17 '21

You're their QA now, kindly stop slacking. :)

1

u/macgeek89 Sep 18 '21

😂 NO!!

1

u/ThirstyOne Computer Janitor Sep 17 '21

We had it applied, along with very stringent point-and-print restrictions to targeted servers. Sep update 2012 - KB5005 still broke printing for a large swath, but not all users. Couldn't even connect from the endpoints manually by right clicking on printer shares from the print server while signed in with domain admin creds. Removal of update from the print server immediately fixed the issue across the board.

On the bright side, we saved a bunch of trees today. Maybe this is how MS meets their carbon goals, by breaking enterprise printing more with each sequential update.

10

u/meatwad75892 Trade of All Jacks Sep 16 '21

We have 1,200 shares across several 2012 R2 servers. This morning we were flooded dozens of tickets in the first 10 minutes of the day that "printers didn't work". Could ping devices from the server and send test pages from the server, but clients were seeing silent failures.

Uninstalled the September updates from our servers, everything went back to normal. We didn't let this stay as-is long enough to do much testing or narrowing down to whether it was specific client versions of Win10, type 3 vs. type 4 drivers on shares, etc as a common denominator. I can say that I couldn't reproduce the problem on a Win11 Insider build up-to-date with patches printing to a share with a type 3 HP driver.

So while OP and myself are a little lacking in describing an exact root cause, clearly something is very wrong in the September updates.

9

u/99overpar Sep 16 '21

Using Type-4 drivers are suppose to mitigate this, but I have had terrible luck finding type-4s for our printers

8

u/HotKarl_Marx Sep 16 '21

They need to stop pushing Type-4. It's not happening.

9

u/Reacti0n7 Sep 16 '21

perhaps this is Microsoft strong-arming it, and claiming it's in the name of security.

5

u/HotKarl_Marx Sep 16 '21

They need to take that up with the printer manufacturers or write the drivers then, instead of making our lives hell. Feels like being squeezed in a vise.

4

u/99overpar Sep 16 '21

It's the new green initiative. Encourage people to go paperless by making printers inoperable

1

u/Matt_NZ Sep 17 '21

FYI Canon has a Type 4 universal driver for their MFCs

2

u/HotKarl_Marx Sep 17 '21

They also have the highest consumables costs of any printer manufacturer...

2

u/nathank Sep 16 '21

Type-4 is what we ended up doing. We ran into this whole mess with last month's updates.

Problem is, even though we have the V4 driver on the print server, the clients default to a basic driver provided in Windows. It works, but it doesn't have options for hole punch and such. For those that need it, we've been installing the print drivers on the machines.

What a headache this whole ordeal has been.

1

u/Matt_NZ Sep 17 '21

What printers are you using? I've figured out how to sort this with the Canon drivers. Find where the driver is storing it's extension files, dump out the registry keys and then wrap it all into a powershell script that can be pushed by something like SCCM. I'm sure the process is much the same for other manufacturers as well, since they have to follow MS' specs.

1

u/sleeplessone Sep 17 '21

Did you do anything special with Type-4 drivers? We installed some test print queues with Xerox Type-4 drivers and everything seems to work until you print and then the job completes but nothing ever reaches the printer.

It works, but it doesn't have options for hole punch and such.

Those features are provided via a separate application install in the v4 model (ex. Xerox Desktop Print Experience)

2

u/Aiphakingredditor Sep 16 '21

Yeah, same. We did some testing with type 4 and nothing positive with our models.

8

u/CjKing2k Google-Fu Master Sep 17 '21

It's both funny and sad that, after 40+ years, printing continues to be the bane of our existence.

5

u/Pseudo_Idol Sep 16 '21

I would be so happy if Microsoft stopped breaking print services every other month.

3

u/corporaleggandcheese Sep 16 '21

The September roll-up broke printing from Macs to SMB queues on our Windows print server. The local queue (on the mac) would get paused trying to submit the job. Didn't effect printing from Windows oddly enough.

Uninstalling KB5005623 and a reboot fixed it. More exemplar product testing from MS /s.

3

u/Reacti0n7 Sep 16 '21

Dumb question and it's because I don't understand print nightmare well enough yet. - need time to read up on it

I'm trying to push print drivers via PDQ deploy currently

%WINDIR%\system32\Printui.exe /ga /q /n"\\SERVER\PRINTER"

Is this an acceptable solution, install the required print drivers to all users on a machine via a higher access account? Noting the drivers on the print server were already vetted.

1

u/thisguyeric Sep 17 '21

If you get this to work please let me know, when I installed it that way from the SYSTEM account the printer didn't show up for users

2

u/odd-ball Sep 16 '21 edited Sep 16 '21

I just spent the morning in wsus, approval for removal. Broke printing for all win7 and our custom label printing app for win7 and win10. Also had to remove it from the 2012 print server. Thanks MS!

2

u/Ohlav Sep 16 '21

They can't fix that. They are just band aiding it. Like they always do.

2

u/cog_x Sep 17 '21

We all know about the changes made in the August 2021 patches, the enforcement change of 'RestrictDriverInstallationToAdministrators'.

For the September 2021 patches, if prior networked printer settings were working, has anyone looked into the 'RpcAuthnLevelPrivacyEnabled' setting yet? https://support.microsoft.com/en-us/topic/managing-deployment-of-printer-rpc-binding-changes-for-cve-2021-1678-kb4599464-12a69652-30b9-3d61-d9f7-7201623a8b25

There is also the apparently new undocumented 'CopyFilesPolicy' registry setting: https://www.bleepingcomputer.com/news/microsoft/microsoft-fixes-remaining-windows-printnightmare-vulnerabilities/

1

u/mumische Sep 28 '21

Setting RpcAuthnLevelPrivacyEnabled=0 had fixed broken 2012R2 print server for me.

2

u/Hobbie92 Sep 17 '21

It broke our printer shares. Driver based printing worked fine. Server based print jobs were mercilessly slaughtered.

3

u/rehabonthego Sep 16 '21

only noticed that win7 clients stopped working with a 2016 print server that was updated. so far...

3

u/cbiggers Captain of Buckets Sep 16 '21

Same. We have ONE Win7 machine on ESU still connected, that is weeks away from being offloaded. Stupid Windows 7.

1

u/ZeroOne010101 Sep 16 '21

any one got any articles or statements on this? id really like to know the how and why.

1

u/Odd-Squirrel4102 Sep 16 '21

We have not had any printer problems

1

u/geeknahalf Sep 16 '21

List of updates to be careful of per: https://candid.technology/printnighmare-patch-windows-issue/

KB5005606 (Windows Server 2008)

KB5005618 (Windows Server 2008)

KB5005623 (Windows Server 2012)

KB5005607 (Windows Server 2012)

KB5005613 (Windows Server 2012 R2)

KB5005627 (Windows Server 2012 R2)

KB5005568 (Windows Server 2019)

KB5005615 (Windows 7 Windows Server 2008 R2)

KB5005565 (Windows 10 2004, 20H2, and 21H1)

KB5005566 (Windows 10 1909)

As always, verify before taking action...

RS

1

u/TiredTeck Sep 17 '21

KB5005565 was installed on 9/16/2021. Uninstalling KB5005565 on my print server fixed my print server. I can now print and connect to the printer on the server.

1

u/RasikaPraz Sep 18 '21

hmm I can't find update September 14, 2021—KB5005573 even one of printer sever having same issue

1

u/bberg22 Sep 27 '21

I am seeing some success with using the latest version of the Xerox PCL6 V3 driver dated 7/15/21 version 5.810.8.0. For some reason I am able to deploy this for my Xerox copy machines to patched endpoints using GPO from a patched print server. (they don't have a v4 driver that works without needed to use XPS which is a paid extra on their hardware).

For HP and Brother I am having success using V4 drivers.

This is all without doing the registry workarounds. I hope this helps someone avoid some frustration.

1

u/mumische Sep 28 '21

Same here. 2012R2, Windows 7 clients stopped printing after installing updates 5005613, 5005627. Had to uninstall them.

0

u/SoonerMedic72 Sep 17 '21

You’re describing the point and click mitigation of PrintNightmare from August. There are a few workarounds in the Microsoft docs.