At my wit's end with UNRAID in general

[btrcp2000]

Started this journey in the spring, bought the hardware in my signature in July. 8 months later, I still cannot find a way to make this thing stable, nor can anyone on the limetech forums find the issue.

The original issue was that the whole server would hard crash (no GUI, no shares, no putty) when rebooting VMs, requiring a power cycle/parity check to recover. Otherwise it was stable, and I was just avoiding rebooting VMs at all costs. Along the way I tried:

1. swapping out sata cables/ports
2. replaced potentially bad hard drives
3. swapped out ram
4. upgraded to 6.4
5. posted diagnostics on limetech forums (always useless becuase they are wiped after a reboot)
6. Used FCP troublsehooting mode to post diags, johnnie.black on limetech can't find anything
7. posted screenshots of the occasional errors that rarely show up twice on the unraid terminal screen
8. rebuilt the docker image

Woke up this morning to another frozen server, second time this week. no error on the terminal screen, no rhyme or reason. Family is starting to move beyond sagetv since it has been such a mess for so long. I'm getting there myself, it has become a colossal time sink.

I don't even know why I'm posting this as I am out of questions to ask. I have probably $1k in this hardware, willing to replace something if I know it will help, but no one can tell me whether it might be RAM, motherboard, PSU, or something else entirely. Maybe it's the colossus cards, maybe its the passed through Nvidia cards that unraid refuses to work with and no one else can explain that either. Each tiny little tweak takes days to show up in the form of another crash, but they all end that way.

I need Linux due to the java heap, but I am very close to declaring unraid a sizeable waste of money in my case.

[BobPhoenix]

If your still at all interested in this I would run unRAID as a NAS only no SageTV, no VMs and no Dockers for at least twice the length of time of the longest you've ever gone between problems. So if it has run for a week between problems then try running unRAID alone for two weeks. If that doesn't work then it is most likely a hardware problem. Since you have tried memory I'm assuming you've run memtest on said memory for 24+hours. Another possible problem is power supply. Flaky power supplies can give a wide range of problems not all of them obvious.

If it does work then you can try adding things. Maybe try the SageTV docker and run it for a few weeks with no VMs or other dockers (I only have SageTV docker running on my unRAID servers). Then add a VM you might try Windows 7 before going to Windows 10 (that's what I have for VMs on my dual processor unRAID using Seabios) use Seabios not OVMF first then switch if you really want to use OVMF for the VM.

All the suggestions I've got if you have any interest in proceeding.

[KryptoNyte]

1) If the motherboard allows running on just (1) CPU, I'd probably try that, one at a time. Is all the hardware you purchased brand new?
2) Because I'm unfamiliar with the way that operating system acts when things go bad, I'd pop in an old hard drive and install Windows, as I'm more familiar with hardware troubleshooting in Windows.
*3) Have you swapped out the power supply yet, and put the system on a UPS?
4) I have one system that can't get stable with just one Colossus cards in it, so I'd pull those at some point, at least for testing. Hopefully you have some other tuners to keep Sage running through the removal.
5) Have you torn the entire system down and put it back together to make sure everything is seated properly (including heat syncs on the processors), extreme I know, but sometimes necessary steps.
6) Have you found some software to stress test the system, CPU, RAM, etc.. for a period of time? Basically trying to find a way to force the problem so troubleshooting becomes more effective.

*If you haven't tried another power supply, I would do this first.

[btrcp2000]

i'm considering putting the server back on the old box for now, then I could try letting it run without disrupting the household, but that's a project in itself.

The mobo, cpus, and ram were all purchased used from mrrackables/unixsupply, the backplane from ebay as part of a supermicro case specific for that board.

Sig is outdated, the VM is actually now Win7 32 bit, which was part of an experiment to see if it was something about the VM.

Whole thing is on a brand new Cyberpower 1325VA UPS complete with whatever sine wave does. The case came with dual PSUs but only using one, will try swapping out.

I did run memtest for one pass, no errors found. RAM is ECC, and since I first booted it up, correctable errors have been found 6 times in the Supermicro IPMI config, all on the same stick. They don;t really ever match up with the freezes, and limetech forums didn't seem too concerned about this. I have pulled the one DIMM, so its now running on 24mb.

I have HDHRs as well, so we can live without colossus. Even when its not freezing those sometimes give issues But if that winds up being the issue I may as well dump sage.

@kryptonyte- what generation are your colossus cards, 1 or 2?

[KarylFStein]

+1 for swapping out the power supply. My desktop was acting really weird recently: freezing / crashing / reboot loops, but also going days without issues. After a long time of trying to figure things out I finally plugged in another power supply and that seems to have fixed things.

[KryptoNyte]

Colossus 1. Purchased used on Ebay.

That RAM chip that you say has correctable errors - I have almost no experience with ECC RAM, but any time I've personally had RAM that throws any kind of error at all, I get exactly the problems that you describe - strange system crashes at times that don't appear to align with any other traceable event.

[EnterNoEscape]

Originally Posted by KryptoNyte

That RAM chip that you say has correctable errors - I have almost no experience with ECC RAM, but any time I've personally had RAM that throws any kind of error at all, I get exactly the problems that you describe - strange system crashes at times that don't appear to align with any other traceable event.

I second this suggestion. I've been through this myself with a Linux server with all new hardware.

[btrcp2000]

I third it. Pulled the faulty stick today, if I make it to next weekend without it soiling itself, I will feel pretty good. For what it's worth, the limetech forum people never thought much of the ram errors since its ECC. From what I have read about it today, it seems like if errors are actually getting reported, that means the ECC doohickeys weren't even able to correct it so as soon as you see one it's pretty serious. But for the record I had never heard of ECC until this build. No real idea what I'm talking about.

[btrcp2000]

Well, that didn't last. Rebooted the windows VM as part of a different video card install, hard freeze once again. This is without the RAM that was generating errors.

[egeller]

I had my first freeze on Unraid last night, in the middle of the Super Bowl (and I'm 8 miles from the Eagles' home stadium, so you can guess how that made me feel).

A quick and lucky search of the Lime forum showed the following actions to take, when the server freezes, and the web GUI is unresponsive. I tried it and it worked to unfreeze it.

1. Telnet into the server, using Putty ( I am not in the habit of using the Unraid terminal).
2. Issue the command ps -ef
3. According to the guidance, the last PID in the list will be the one causing the freeze. Issue the command kill <PID>

I had issued a command to reboot the server from the GUI when Unraid had stopped responding. The GUI did not respond. Once I killed the process (it was a Plex task), the server immediately went about the business of rebooting

YMMV,

Erv

[btrcp2000]

Congrats on an exciting win last night after a long wait. I'm in the Cincinnati area, where results are not really a requirement and mediocrity counts as a result anyway. Who Dey and Go Reds!

I will make a note of this, but I had gotten out of the habit of even checking putty as telnet would go away too. My freezes tend to take everything, no shares, no GUI, no putty, etc. Would love to catch the last thing it was doing before it crapped out though, so I will sticky note this to the monitor.

[Wrx4me]

Sorry to hear this as it is very frustrating and difficult to track down. I had a freezing problem last year but mine turned out to be an incompatibility with the amount of RAM my Motherboard was willing to tolerate.

I was running stable for months with two 8GB sticks of RAM until I decided to add 2 additional 4 GB sticks totaling 24GB memory. After I added the extra RAM, my server would freeze every 5th or 6th day of operation. I ended up removing the extra modules as I found out that no combination other than 2 sticks was stable on my MOBO. If I used either the two 4 GB or 8 GB modules the server was OK. However, any attempt to use more than 2 sticks of RAM resulted in a server freeze in less than a week. Now my server runs for months on end without a hiccup, in fact no freezes or crashes at all.

My point is that you should go back and look for anything that you may have added to this build that is quite different from your other builds, including the used hardware and you may just figure it out. My extra RAM modules also had memtest errors but ran stable when only 2 banks of memory were in use on the MOBO. Good luck!

[btrcp2000]

Originally Posted by egeller

I had my first freeze on Unraid last night, in the middle of the Super Bowl (and I'm 8 miles from the Eagles' home stadium, so you can guess how that made me feel).

A quick and lucky search of the Lime forum showed the following actions to take, when the server freezes, and the web GUI is unresponsive. I tried it and it worked to unfreeze it.

1. Telnet into the server, using Putty ( I am not in the habit of using the Unraid terminal).
2. Issue the command ps -ef
3. According to the guidance, the last PID in the list will be the one causing the freeze. Issue the command kill <PID>

Went to reboot the VM, which usually results in a hard freeze but not this time. UNRAID is currently in a half-crashed state, meaning the VM tab won;t load or refresh, but seems to be otherwise functional. Here's the output. Based on your comment, plex is the last entry. Seems strange to me as plex just seems to run fine in the bakground as long as the server itself is running. It's only VM activity that seems to kill it. Mean anything to you?


nobody 35806 34492 0 04:17 ? 00:00:00 /usr/bin/openbox
122 35830 35795 0 04:17 ? 00:00:00 nginx: worker process
122 35831 35795 0 04:17 ? 00:00:00 nginx: worker process
root 35832 35786 0 04:17 ? 00:00:03 /bin/sh /usr/bin/tailstatusfile /config/log/app.log
root 36084 2 0 04:17 ? 00:00:00 [kworker/0:4]
root 37025 2 0 01:00 ? 00:01:40 [kworker/0:1]
root 38268 2 0 11:53 ? 00:00:00 [kworker/u81:1]
root 39259 2 0 11:21 ? 00:00:00 [kworker/u80:1]
root 39775 34493 0 04:17 ? 00:00:00 forstdin -d ? -- LINE fdmove 0 3 importas -u LINE LINE pipeline
root 39792 34485 0 04:17 ? 00:00:16 /usr/bin/x11vnc -display :0 -rfbport 5900 -rfbportv6 -1 -no6 -noi
nobody 39809 34491 0 04:17 ? 00:02:40 /usr/local/crashplan/electron/crashplan
root 39872 39775 0 04:17 ? 00:00:01 tail -n0 -F /config/log/service.log.0
root 40111 2 0 11:37 ? 00:00:00 [kworker/u81:3]
root 40738 4916 0 11:54 ? 00:00:00 php-fpm: pool www
nobody 40867 35563 0 04:17 ? 00:00:01 /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaser
root@UNRAID:~#

[egeller]

Originally Posted by btrcp2000

Mean anything to you?

In short, yes. In my case I had a Plex task listed as the last task. I do not remember if it was the same task you are seeing.

I had issued the reboot command from the GUI when it froze. After I issued the kill command against the last-listed Plex task, the reboot proceeded normally and all has been well ever since. So my recommendation is to kill the Plex task and see what happens.

I had been using Plex heavily earlier in the day of the freeze, so I had thought this was a one-off thing, related to my activity (fixing metadata and such). If you are also seeing a problem whose root cause is Plex, maybe there's a bug in Plex. Either that or I need to sacrifice a goat to appease a deity.

Erv

[wayner]

Originally Posted by egeller

Either that or I need to sacrifice a goat to appease a deity.

That's never a bad idea

[stuckless]

Originally Posted by egeller

In short, yes. In my case I had a Plex task listed as the last task. I do not remember if it was the same task you are seeing.

I had issued the reboot command from the GUI when it froze. After I issued the kill command against the last-listed Plex task, the reboot proceeded normally and all has been well ever since. So my recommendation is to kill the Plex task and see what happens.

I had been using Plex heavily earlier in the day of the freeze, so I had thought this was a one-off thing, related to my activity (fixing metadata and such). If you are also seeing a problem whose root cause is Plex, maybe there's a bug in Plex. Either that or I need to sacrifice a goat to appease a deity.

Erv

I used to have issues, some time ago, with plex on linux, but since moving to unRAID, I have never had an issue. But if recall, there was something related to "indexer" in plex that used to kill machine every time. But again, I've never run into this on Plex. Are you using the linuxserver.io plex container? I tend to use linuxserver.io containers where possible.

[rerooks]

I have the same motherboard with 2x2630 CPUs and 64 Gb of RAM. It was overall a pretty cheap ebay purchase.

I have been running quite stable for some time, with about 8 dockers and 2 VMs.

Could there be a processor bug that is related to VMs?

If you have two of them, maybe pull one and see how it does? You can then put all of the memory on one processor bus. Make sure to adjust memory and PCI slots for the one processor. You could also buy a cheap 2630 as a test.

[btrcp2000]

Interesting. What other hardware do you have in there? I really want to blame Hauppauge.

It has been suggested to me to strip it down to just a NAS, then add back dockers and VMs one by one until it soils itself again, which makes logical sense to rule things in/out. Before I do that though, my gut tells me that the Hauppauge Colossus v1 tuners aren't helping because they have always required a lot of TLC such as the nightly reboots. I'm going to pull them and let it run with everything else intact for a while. If it makes it 3 days, that will be uncharted territory, but not a long term solution because i still have to watch my sh!tty baseball team . I also have a Colossus 2 on the way, might also try an HDPVR2.

I am currently running with only 16gb. One stick occasionally throws up correctable error, but I pulled it and its mate. Hard freeze again last night, so I guess I will be putting those back in

[wayner]

The BM-3000 type of encoders might be the best way to go in the future - the only downside is that they only do stereo for audio. But they are network devices just like the HDHR so then you don't have to worry about stuff like USB ports or PCIe slots.

[rerooks]

Originally Posted by btrcp2000

Interesting. What other hardware do you have in there? I really want to blame Hauppauge.

I purchased 32Gb modules on eBay for about $110, ECC RDIMM 1600, so a total of 8, with 4 per CPU. They seem to be brand new.

The hardware is 3 HD Homerun Primes, with cable cards, so the connection is purely through ethernet, which is nice since it eliminates other hardware issues.