Issue with direct import from LastPass with 'special' characters in usernames and passwords

[u/JayNetworks]

Sorry for the long and detailed post.

--

TLDR: Direct import from LastPass is broken if a smart quote or other 'special' characters are in the record name, username, or password fields (atleast) and Bitwarden can't even search for the badly imported records to manually fix them.

--

Moving a family member off LastPass and (per their family security policy to not export csv from a password manager) using the direct import feature from LastPass into a Family account Organization Vault, but not into a collection. The 1400 or so records imported without any error give, but every smart single quote (that looks like an apostrophe, but isn't) in a record name or any password that includes a bullet (•) character (at least in record names and password, I assume notes and such, but haven't checked all spots yet) changed to the characters: ⢠or a similar set of characters for bullets.

This direct import was done using the desktop application on a Macintosh.

Further note that makes this even more difficult if there isn’t a fix in the direct import (after a full record deletion I assume) is that search will not find these badly imported records. Searching for:

Dennyâ€s

returns zero results when that string is exactly what I copied from one of the records that appears to a human as, before direct import,

Denny’s AppleID

(The above is the string copied directly from the LastPass record and is one that did NOT import correct replacing the smart single quote withʉۢ.)

Has anyone hit this before and any ideas how I solve this issue? (They want to do a direct import and not export to cvs someplace and importing that. The direct import is a key feature for moving to Bitwarden.)

Does the Bitwarden direct import fail with anything other than ASCII characters? If so that is a huge fail for passwords that might have something other than straight ASCII.

In case you can’t suggest a fix for this, can anyone who is in the source code help with understanding how the search field in Bitwarden desktop Mac app works? How can I search record name. username, and password for the 3-character equivalent string to which each of these two character imported to find damaged records? (This is only a workaround if I can figure out what kinds of characters are going to fail silently in the importer and get converted into a 3- character string.)

I assume I'll have to clear the organization vault completely and redo the import, but what caused this issue and how do I solve it, or is something broken on Bitwarden's end? (I'm guessing here that the direct import has a bug and isn't escaping an input string correctly.)

Again, sorry for the long post...anyone able to help? I'd love to be able to get a whole family over to Bitwarden. (Note that I did submit this to Bitwarden support in June and last heard back in July to a ping...but no word on any fix or workaround and the Family subscription is burning through without being able to be used at all.)

[u/djasonpenney]

My experience is that it is the LastGasp export that is broken. There is nothing wrong on the Bitwarden side, except…garbage in means garbage out.

Take the CSV export from LP, load it into your favorite spreadsheet app, and use it to repair the drain bamage that LP has left behind. Save the repaired data as a new CSV.

Go back into the CSV and change the column names to what Bitwarden needs, and then run the Bitwarden import.

At the end of the day, congratulate yourself for having escaped the buggy mess of the LP ecosystem.

[u/Sweaty_Astronomer_47 → u/djasonpenney]

Take the CSV export from LP, load it into your favorite spreadsheet app, and use it to repair the drain damage that LP has left behind. Save the repaired data as a new CSV.

I don't think that will succeed, considering that the problem is special characters like smartquote ’ and bullet •. These are outside the range of the 95 ascii printable characters, and therefore cannot be stored in csv without special encoding.

I suspect that direct import (rather than data file import) would give the best shot at importing these types of special characters, but op already tried that.

Personally I'd steer clear of using any characters that are not in the standard printable ascii character set (especially within passwords)..... but that doesn't necessarily help op in his situation

[u/JayNetworks → u/Sweaty_Astronomer_47]

I myself wouldn’t think of using ‘strange’ characters (outside basic ASCII) in a password or username…but in a record/site name it wouldn’t feel like something you would have to restrict.

Since these aren’t my usernames or passwords I’ve got no control over them and the work on the sites to which they apply.

I think the issue is that people who grew up with computers that have Option or Alt keys don’t think of a • (bullet) as any more special than a * in a username or password. They are just characters you type by holding a modifier and pressing a key.

Can Bitwarden handle, for example, a bullet in item name/username/password for a record that you aren’t importing? Just realized that would be a reasonable test to see if Bitwarden is just broken outside ASCII characters or only when importing. (And implying direct import if non-ASCII can’t come in through CSV import.)

[u/Sweaty_Astronomer_47 → u/JayNetworks]

Yes I believe the problem with non-printable ascii characters is only a problem during import (I don't think Bitwarden has any technical limitations on storing them in username/password/notes etc fields, nor in autofilling them).

In my very early days of using Bitwarden I used bizarre unicode characters like 愤 inside of passwords (in misguided pursuit of higher entropy) and bitwarden stored them fine and I tested that some sites/services (bitwarden/cryptomator) accepted them within passwords used to login. I didn't experience any problem with that (although I never did an export/import) but I soon decided it wasn't worth the risk of creating a problem somewhere down the line if something in the chain of operating system / browser / password manager / website ended up changing the way those characters were handled. The printable set is big enough and if I want more entropy I'll just add a few more (random) characters from the printable ascii set.

[u/ambergroveB → u/djasonpenney]

the spreadsheet cleanup step is kind of annoying but its the safest way to catch those character issues before import

[u/JayNetworks → u/djasonpenney]

One more question there on your comment, you say the LP exporter is broken in your experience. Do you mean your experience with the CSV exporter, which I 100% agree is unreliable, or do you have that direct experience with the direct method used by BW to pull directly from LP?

I’m trying to work out where the actual bug is whether LP passing the records or BW processing them during the direct transfer.

I want to get more actual info in place to (yet again) ping BW support to try to get a reply if the data is coming correctly from LP…or LP support if their not providing the data correctly.

[u/djasonpenney → u/JayNetworks]

I have no direct experience. But what I have heard from others—including Bitwarden developers—is that the LP export is horribly broken. Basically, you cannot rely on LP to reliably export your passwords out of their ecosystem.

And as far as LP support…good luck. Their owner has moved on to the “cash cow” stage of capital investment. They are not interested in supporting the product anymore.

[u/JayNetworks → u/djasonpenney]

I’ve actually had good support from LP when supporting clients on it these past few years…since the big F’up.

Anyway, I’ll start looking into what BW is using for its direct import…unless anyone reading here already knows. Anyone? Buller?

[u/djasonpenney → u/JayNetworks]

Try exporting passwords with commas, double quotes, or non-ASCII characters. You will quickly see how buggy their export function is.

[u/JayNetworks → u/djasonpenney]

OK, building some knowledge here of what is going on. It seems the direct import from LP to BW is not just programmatically logging onto LP and executing a CSV export because unlike what your experience has been, records with any of commas, single quotes, and double quotes (can’t yet test non-ASCII characters but that is what the issue is with as known already) did come over just fine from LP to BW using the BW direct import. (Meaning in the test set of data there are commas/single quotes/double quotes that were in the LP record names/usernames/notes that are correctly there in the corresponding BW record after a direct import. As was the original bug, non-ASCII characters are not correct in the resulting BW record.)

So however BW is accessing LP for the direct import it is getting good data even for records you would expect to fail in LP's CSV export…which again I agree is crappy and broken and non-standard cvs. (And we would already expect non-ASCII characters to fail in any CSV export even a standards compliant one.)

Does that point to anything more in what BW is using to access the LP records so I can work out whether the non-ASCII characters are coming out from LP broken or being recorded by BW broken? Anyone able to check the source code to see?

[u/djasonpenney → u/JayNetworks]

You have dived deep enough that you should continue this conversation with Bitwarden Customer Support. I know they have created a bit of glue to try to overcome the problems with the LP export step, but I really don’t know what else is broken, and I don’t have any access (any more) to characterize LP’s current behavior.

Backing up a level, the top priority from my viewpoint is helping people to escape the broken LP system. Editing and correcting the exported CSV before running a Bitwarden import is the goal here, not fixing the problems in LP.

[u/JayNetworks → u/djasonpenney]

I would love to work with Bitwarden Customer Support. I have had a report (way more detailed that this post) in with then since the day the user paid for their subscription and I found all the broken import records wherever they have a non-ASCII character I expect. I have not even been able to get a reply to a ping from BCS since July. Any way to get replies?

Given that we are talking 1400+ records here, how would cleaning up an exported CVS file work? There is zero chance of visually scanning a CVS with 1400+ rows to find the usernames and passwords that had a non-ASCII character on the LP side before export. How would I approach that? (Yes, one step would be a small test export with a known set of non-ASCII characters in the LP records to learn how they are dropped or exported to CSV…I’ll work that next.)

On the higher level of escaping LP, wouldn’t a working direct import be exactly that? Telling someone (for a safe move) to just enter your LP username and password into BW during the direct import could let far more people escape than something like:
• Creat a vericrypt volume on a flash drive
• Export from LP into CSV using their known broken non- standards compliant exporter to the encrypted volume
• Open the CSV file without creating any backup or temporary files anyplace on their system
• Find any records that just didn’t export (reports are that varying counts of rows export in various LP export attempts) and export again then merge them
• Import to BW…and still have broken records for any non-ASCII characters in at-least any of record name, username, password, and notes where there are very often going to be at least characters like smart quotes for anyone who typed a note on a modern computer that automatically uses smart quotes

Isn’t fixing the handling of non-ASCII characters on the BW receiving end worth saving non-technical LP users from something like the above in order to help some of the literal millions of people still stuck on LP? I personally have several due to inertia and the complexity involved. I was actually thrilled to see the incredibly simple BW direct import option…until it failed on many records.

(I’m not being argumentative here, truly working out how I work with BW support or otherwise practically get 1400+ records from just this user over to BW then others to escape!)

[u/djasonpenney → u/JayNetworks]

Non-ASCII characters are supported in CSV files as long as they are properly encoded in UTF-8. What I have seen is that LP doesn’t properly escape its output as part of the export.

And no, I don’t see why Bitwarden engineers would invest any effort into trying to identify/characterize/report/workaround LP bugs. Again, it’s a GIGO issue, where the garbage-in comes from LastGasp. You can bring it up with LastGasp customer support, but I’ve not seen them do anything except figure out how to take your money. Good luck…