Lots of Nextcloud issues

guyinpv

So it seems we are having millions of issues. Millions I tell you! The employees are all running around claiming the ghost in the system is reversing changes, undeleting files, not saving changes, reverting changes, etc etc.

I can never duplicate these problems, but we have dozens of them and now the boss fully distrusts the system, says it's sh*t and we can't work like this any more.

So what problems?

Files are cut and pasted to move between folders. The pasted file remains but the cut version also remains in the old location. Even worse, it appears like it really cut (on local machine of course), but later we find the file still exists, was put back, re-uploaded, reverted, whatever. Thus we have duplicate files across different folders.
Because cut/pasting files seems to randomly fail, this happens like 50% of the time, we're constantly having to re-delete files that should not be there.
Changes to files don't sync. We work on a lot of Excel and people are complaining they open the files, make their edits, everything appears to save and what not. Then we find out days later the information is missing. The file either saved but later reverted, or someone else's older copy re-uploaded? Or maybe it never sync'd those local changes to begin with? (I'm not sure how that can be because I would think I'd get a conflict file at some point).
Files are renamed but this seems to interfere with editing them or moving them around. Sometimes editing a file will also come with making a change to the filename. Again these changes appear to work from the perspective of the local user but later we find problems with the sync on everybody else.
I've had complaints that two people will be on the phone with each other. One person is making edits while the other one waits. Person 1 says ok I've saved it and says it synced. Person two opens file but doesn't see changes. This goes on for 30 minutes, supposedly, she waited 30 minutes and could not see those changes.
Files activity and revisions appear incorrect. A file will say it's 20 days old (when looking at web interface) but we know for sure someone has edited the file within the last week. Looking at current and prior revisions, those changes are not there.
Of course, when we discover these problems, we go looking for those missing changes, nobody has them, all files are in sync with same data as the web interface shows, but the person swears up and down they made those changes and saved the file. Thus I get a lot of "my changes just disappeared" complaints and "they were there before!"
I have a laptop that I'm using to train everybody how to use the cloud better. Waiting for files to sync for a longer period of time before making a consecutive change, etc. This laptop is never used to edit files, it's just a spare computer in my office that happens to have NC on it.
Suddenly 20 minutes after I've been giving my presentation and looking at all these problems, I get like 7 files appearing in certain folders with "conflict" in their names.
I happen to know that in NC, conflict files are not synced to the cloud but are just kept locally for the person to figure out.
I have not edited any files on this laptop. For example two of the "conflict" files were over 3 weeks old since their last changes, pretty much archived files. But they appeared as conflicts on my laptop, within a folder where the current copies aren't even located.
I other words, 3 weeks ago file1 and file2 were moved into /archive. But now on my laptop suddenly the two of them appear as conflict files within /whatever/folder where they used to exist in the past.
Why did my laptop suddenly drudge up random old files from weeks ago and call them conflicts? And why only 20 minutes after I've been on the laptop and everything has long since been sync'd up? They literally just popped up out of nowhere, these 5 or 6 conflict files. And again, I don't edit files on this laptop, so what's the conflict? grrr

I'm always telling people to wait for their changes to sync. Looking for the blue dot to appear or the system tray icon to change and come back to green, etc. And of course everybody says they do this. So they make changes, wait for sync. Then days later we find everything is reverted or files duplicated or changes missing, etc.

I'm out of ideas for troubleshooting this. Part of me says user error but come on, using a cloud server should not be this freaking delicate. What more can people do after waiting for change to sync, reopening files to make sure changes are still there, calling someone else to open the file on their computer to verify it sync'd, but then still later finding the changes are gone?
I can believe user error for occasional one-off problems. Two people having file open at same time and getting a conflict or whatever. Or I can believe occasional network issue like maybe wireless had a hiccup or network was down right at time of file save? Who knows. But we are dealing with dozens of problems, everybody is complaining, issues dating back weeks. We now have to fully audit all our files and records against paper copies, ugh! It can't all be user error.

I thought, maybe there is a time issue, like a computer somewhere with timezone set different? I wonder if some random infrequently-used computer is going online and creating all these sync issues when it tries to catch up?
Could computers with different timezones or clocks cause issue? Could it be something with computers that aren't used very often? Mobile devices? Home computers that are only rarely used with NC?

I'm at a loss how to troubleshoot, since I can't actually duplicate any issue on command. Nobody takes personal responsibility, nobody says maybe they didn't really save, or maybe they really did a copy/paste rather than a cut/paste. Everybody blames the cloud, saying it is doing things on its own. I have to hear " 'it' changed my file, 'it' created a copy. 'it' didn't save my changes" etc. Always "it" did it.
I'm glad we don't pay our cloud a wage, since it appears to do more work on our files than our employees do!

I've about had it with cloud systems I tell you. Box was a mess, OneDrive was a mess, Google was a non-starter, no system seems to be robust enough to handle the extremely complicated task of 10 people editing various Excel files and moving them around in folders.
Maybe I need to wait for the next evolution of cloud sync software? A new technology that lets people edit files which then sync changes between computers. If anybody knows of such software that can do this, let me know!

For now, what the heck am I supposed to do about NC? Nobody trusts it, nobody thinks they are doing anything wrong. It's the magical cloud that just does things all on its own to mess with us.

I have to believe there is some kind of condition where a computer is mistakenly thinking its copy of a file is a newer one and overwriting the real file. If a file backup tool or antivirus were somehow updating the time stamp and causing old files to get uploaded, I would have a serious system-wide effect of every file being editing by the same person, which is not happening. Nor is NC recording the latest activity or revision as the file being overwritten. The revision history doesn't contain the supposed changes the person is so sure they made.
That tells me those original changes never sync'd in the first place. But even if that is true, why isn't there conflict files?

Nothing makes sense and I can't think of how to troubleshoot. If only there were a tall bridge nearby.

black3dynamite

I'm sorry but I'm too impatient to read all of that. Are all these issues related to files being shared amongst two or more users?

momurda

This is likely due to people doing things with files shared among many people, when the NC sync is still happening(blue whirly thing) rather than green checkmark.

Emad R

@guyinpv

Are you sharing accounts ? or each machine/person has his own accounts ?

How are you managing shared folders, I am doing user account called shares and that user has folders that are shared like LOG to log group, and that share can not be reshared again.

Maybe cause it is cloud based you have latency issues, maybe, is the system self hosted ?

It is interesting read as I am using NC 13 in prod recently, but I am paring it with saltstack, and I am using salt to push files in NC folder. Not the best approach but its working well, and I am backing up user files.

I noticed you say Excel alot, let me tell you one thing. Excel was not meant for sharing PERIOD. no way around it, you want to share excel consider web based spread sheet like:

Office 365 Excel = https://office.live.com/start/Excel.aspx
Can be for free

NextCloud + Only Office

Other than that forget, 2 users in nextcloud or any other software using the same file will create conflicts, and I noticed Nextcloud handling them very well, It creates the conflict file and keeps the original. The idea with NC, try using the share system of user called share and making folders in this user and sharing it to groups and see if that helps instead of individual sharing files . Making a default shared folder for users in the same dept usually helps.

And it seems you have latency issues, my NC is self hosted and everything is zippy and responsive.

All in all take a deep breath and check your investment and everything and management and if you want to stop and simply opt for file server SMB, get Windows File server latest and expensive stuff and that might work better with Excel files. But think about the issue your solving, its not NC its the fact that they are using Excel, also what version of Excel on all machines and put the x32 version is possible. I am using 15.0.5031.1000 so please ensure you have the latest. You will now if you have the latest setup files cause it takes 25-30 mins to install.

Oh also disable all the NC apps and addons you dont use, I have only Gallery/PDF viewers/Sharing and THemeing and 2 more.

ALso how did you set it up, Check setting it up with PHP-FPM version 7 that might make things faster.

scottalanmiller

@guyinpv said in Lots of Nextcloud issues:

Changes to files don't sync. We work on a lot of Excel and people are complaining they open the files, make their edits, everything appears to save and what not. Then we find out days later the information is missing. The file either saved but later reverted, or someone else's older copy re-uploaded? Or maybe it never sync'd those local changes to begin with? (I'm not sure how that can be because I would think I'd get a conflict file at some point).

So no known issue. This is how a normal file system behaves if people overwrite each other.

scottalanmiller

@guyinpv said in Lots of Nextcloud issues:

Files are cut and pasted to move between folders. The pasted file remains but the cut version also remains in the old location. Even worse, it appears like it really cut (on local machine of course), but later we find the file still exists, was put back, re-uploaded, reverted, whatever. Thus we have duplicate files across different folders.
Because cut/pasting files seems to randomly fail, this happens like 50% of the time, we're constantly having to re-delete files that should not be there.

This is most likely from one person removing and another person replacing it.

scottalanmiller

@guyinpv said in Lots of Nextcloud issues:

I've had complaints that two people will be on the phone with each other. One person is making edits while the other one waits. Person 1 says ok I've saved it and says it synced. Person two opens file but doesn't see changes. This goes on for 30 minutes, supposedly, she waited 30 minutes and could not see those changes.

Are they awaiting BOTH syncs, or just the one? No single person will know if the sync is done. They should be using a non-sync method, like the web, if they want to do it instantly like this. Just like how SMB works.

scottalanmiller

@guyinpv said in Lots of Nextcloud issues:

I've about had it with cloud systems I tell you.

First, your issue is with users, not cloud systems.

Second, the technical thing you are seeing here is sync, not cloud.

You can use NextCloud differently to not have these possible issues.

stacksofplates

Git.

stacksofplates

Sorry, kind of trolling. But if they are having this many problems with just editing files they shouldn't be able to edit the version of the file that everyone sees. You're almost to the point where you have to do some kind of approval process to have changes made. That sounds ridiculous.

scottalanmiller

@stacksofplates said in Lots of Nextcloud issues:

Sorry, kind of trolling. But if they are having this many problems with just editing files they shouldn't be able to edit the version of the file that everyone sees. You're almost to the point where you have to do some kind of approval process to have changes made. That sounds ridiculous.

Sharepoint allows for that.

guyinpv

@emad-r said in Lots of Nextcloud issues:

Are you sharing accounts ? or each machine/person has his own accounts ?

Each has own account.

@emad-r said in Lots of Nextcloud issues:

How are you managing shared folders, I am doing user account called shares and that user has folders that are shared like LOG to log group, and that share can not be reshared again.

I have a general "admin" user that nobody logs in to. This user is the one with the master shared folder that is shared to everybody else. We don't have a re-shared type issue. It's just a single folder shared to all accounts by this main user.

@emad-r said in Lots of Nextcloud issues:

Maybe cause it is cloud based you have latency issues, maybe, is the system self hosted ?

I couldn't say, but when I test making changes to a typical Excel file, it only takes a few seconds to sync and then turn green again. So maybe 5 to 10 seconds to sync back to everybody else.

@emad-r said in Lots of Nextcloud issues:

I noticed you say Excel alot, let me tell you one thing. Excel was not meant for sharing PERIOD. no way around it, you want to share excel consider web based spread sheet like:

They aren't sharing like editing at the same time or anything. The people here are too primitive to make them use a web-based browser spreadsheet. Never gunna happen here. We have thousands of spreadsheets.

@emad-r said in Lots of Nextcloud issues:

Other than that forget, 2 users in nextcloud or any other software using the same file will create conflicts, and I noticed Nextcloud handling them very well, It creates the conflict file and keeps the original. The idea with NC, try using the share system of user called share and making folders in this user and sharing it to groups and see if that helps instead of individual sharing files . Making a default shared folder for users in the same dept usually helps.

We are aware of how conflicts work. This is very visual to people when they see a conflict file appear, they know what is going on and what to do about it. But in all the various problems I listed, there is no conflict file created.

@emad-r said in Lots of Nextcloud issues:

But think about the issue your solving, its not NC its the fact that they are using Excel, also what version of Excel on all machines and put the x32 version is possible. I am using 15.0.5031.1000 so please ensure you have the latest. You will now if you have the latest setup files cause it takes 25-30 mins to install.

We have Office365 so everybody is on the latest.

@emad-r said in Lots of Nextcloud issues:

Oh also disable all the NC apps and addons you dont use, I have only Gallery/PDF viewers/Sharing and THemeing and 2 more.
ALso how did you set it up, Check setting it up with PHP-FPM version 7 that might make things faster.

I have no extra addons enabled, pretty raw/fresh install on a new VULTR instance. Everything is up to date. PHP is on 7.

guyinpv

@scottalanmiller said in Lots of Nextcloud issues:

@guyinpv said in Lots of Nextcloud issues:

Changes to files don't sync. We work on a lot of Excel and people are complaining they open the files, make their edits, everything appears to save and what not. Then we find out days later the information is missing. The file either saved but later reverted, or someone else's older copy re-uploaded? Or maybe it never sync'd those local changes to begin with? (I'm not sure how that can be because I would think I'd get a conflict file at some point).

So no known issue. This is how a normal file system behaves if people overwrite each other.

Don't see how that's possible. It would not be users overwriting each other, it would have to be the NC client overwriting. But why would it do that? Why would someone else's older file overwrite the newer changes made by the person?

guyinpv

@scottalanmiller said in Lots of Nextcloud issues:

@guyinpv said in Lots of Nextcloud issues:

Files are cut and pasted to move between folders. The pasted file remains but the cut version also remains in the old location. Even worse, it appears like it really cut (on local machine of course), but later we find the file still exists, was put back, re-uploaded, reverted, whatever. Thus we have duplicate files across different folders.
Because cut/pasting files seems to randomly fail, this happens like 50% of the time, we're constantly having to re-delete files that should not be there.

This is most likely from one person removing and another person replacing it.

Nobody would replace it, how can they? If I cut a file and paste it somewhere else, NC should be treating this as me deleting the file, which should then propagate to the rest of our users. After all a "normal" file delete propagates and works relatively well. But why would it fail a lot by using cut/paste? What's the difference?

guyinpv

@scottalanmiller said in Lots of Nextcloud issues:

@guyinpv said in Lots of Nextcloud issues:

I've had complaints that two people will be on the phone with each other. One person is making edits while the other one waits. Person 1 says ok I've saved it and says it synced. Person two opens file but doesn't see changes. This goes on for 30 minutes, supposedly, she waited 30 minutes and could not see those changes.

Are they awaiting BOTH syncs, or just the one? No single person will know if the sync is done. They should be using a non-sync method, like the web, if they want to do it instantly like this. Just like how SMB works.

Right, but when person A says they are done making changes and the icon is green, the other person shouldn't have to wait 30+ minutes for it to sync. I've tested sync speeds and for these files it only takes 5 seconds so it's really weird. One or the other computer is simply not making it to the cloud, or blocked by firewall or I don't know what. The NC client certainly isn't throwing any errors.

guyinpv

@scottalanmiller said in Lots of Nextcloud issues:

@guyinpv said in Lots of Nextcloud issues:

I've about had it with cloud systems I tell you.

First, your issue is with users, not cloud systems.

Second, the technical thing you are seeing here is sync, not cloud.

You can use NextCloud differently to not have these possible issues.

There are far too many "random" problems for me to just blame users for everything. Especially when we are all aware of these problems are literally verifying every edit.
Open file, edit, close file, wait for sync, open file, check edits are still there, go to another computer, check if file synced, open file, check for edit, looks good......2 days later, changes are gone, nobody has those previous changes that were verified. You can't tell me this is still a user error.

I don't know how else to use NC differently to make all these problems go away.
We update files. Rename files. Move files around. Create new files.

That's it. Very rarely would any two people need to change the same files at the same time, otherwise this would simply end up with a conflict file which is normal and we expect those when this happens.

What we don't expect is for verified changes to files that multiple people saw were updated, to be randomly reverted. We don't expect newer files to be overwritten by older files, I can't fathom how that kind of event can happen. Some kind of weird timestamp issue in Windows?

We don't expect a very normal Windows operation like cut/paste of a file to magically make the cut file reappear at a later time.

These are weird errors, which leave no backtrace of activity or file revisions in the web interface.

guyinpv

@stacksofplates said in Lots of Nextcloud issues:

Sorry, kind of trolling. But if they are having this many problems with just editing files they shouldn't be able to edit the version of the file that everyone sees. You're almost to the point where you have to do some kind of approval process to have changes made. That sounds ridiculous.

Heck I'd rather do a lot of things.

These Excel files are mainly used for tracking physical inventories, parts on order, incoming/outgoing/received/awaiting/etc.

Much better to use dedicated software for this kind of tracking, but we can't always get what we want. They are used to spreadsheet files, been using them for a decade, so a custom database just isn't in the cards.

Files based on vendor, which also contain lists of our own orders which are awaiting product for the incoming parts, etc. It's been evolved over the years and it's just the way it is. How they manage the files, how they are named, what folders they go in, and how data is entered, all very precise and people trained to use it.

And up until a few years ago these files would have been on a folder share on some random computer in the office. At least with the LAN share if someone opens a file, it won't let another person open it too.

hobbit666

Not used NC but are there any logs? Or could you enable extensive logging and a share you know gets the most complaints? To see who is checking a file out and saving changes, then the next.

Could it be Excel "caching" the file on people's machines so when they click save it only saving their changes to a "old" copy? (Just stabbing in the dark)

guyinpv

@hobbit666 said in Lots of Nextcloud issues:

Not used NC but are there any logs? Or could you enable extensive logging and a share you know gets the most complaints? To see who is checking a file out and saving changes, then the next.

I don't know about any logs beyond just the activity stream and revisions system. The desktop client also has an activity log that I watch.

What makes this difficult is that nobody seems to know the problem happened right at the time. It's always days if not weeks later that people suddenly "are sure" they updated the file but now the changes are gone. The activity stream will simply lack any edits by them.

Case-in-point. A file they say had about 12 rows of data in it. The activity stream has 5 changes by users and there are 4 files revisions. None of the revisions contain the 12 rows, none of the activity shows an edit by the person who claims to have edited it. But they are SURE they did it, saved the file, it synced, they saw the data there in the file after that, etc. But a week or two later those 12 rows can't be found and there is no history of the changes.

My thinking here is that maybe they did update the spreadsheet, but somehow those changes only remained locally, it just didn't sync. But if this is true, eventually they would get a conflict cause the system would try to sync when it comes back online. Either it would sync the new changes (assuming web version never changed), or the web version would be newer and they would get a conflict file created, which also didn't happen.

I'm feeling like Dr. House right now, I think everybody is lying. NC the desktop client, the end user, Excel. Everybody is lying and I can't find the truth!

@hobbit666 said in Lots of Nextcloud issues:

Could it be Excel "caching" the file on people's machines so when they click save it only saving their changes to a "old" copy? (Just stabbing in the dark)

I don't think so. When Excel opens a file it creates a temp copy, a hidden file with a tilde in the file name, these are excluded by NC anyway. Then when the file is saved, the tmp file goes away. But I don't believe there is any kind of 2nd full copy/cache.

Now, Excel does have an autosave feature. The location is not within NC though, it's down in the user's AppData folder somewhere. So this is likely not an issue. It's not autosaving to the original file, in other words, autosave creates another file somewhere else.

JaredBusch

Your users are doing something wrong. Simple as that.

No sync service does anything like this without a reason

Your issue is not Nextcloud. It is far from perfect, but it is not the problem here.

Your users are almost certainly not saving things where they should be. No matter what they try to say.

Also, if you have the preview pane enabled in Windows explorer, that will lock files. Been down that disaster with ODfB in the past.