ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Fedora 26 No Space Left on Device with Plenty of Space Available

    Scheduled Pinned Locked Moved IT Discussion
    linuxext4fedorafedora 26
    37 Posts 8 Posters 6.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • wirestyle22W
      wirestyle22
      last edited by wirestyle22

      So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently

      stacksofplatesS scottalanmillerS 2 Replies Last reply Reply Quote 1
      • ObsolesceO
        Obsolesce
        last edited by

        Thanks for the explanations. It helped me search better, and I eventually found this: https://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html

        Shows how to check available inode space and how to fix it as well.

        As for Scott's problem, I'm guessing something creating a lot of temp files that are cleared during a reboot.

        wirestyle22W 1 Reply Last reply Reply Quote 0
        • wirestyle22W
          wirestyle22 @Obsolesce
          last edited by

          @tim_g If it's the same file system wouldn't that keep the number of temporary files down to a minimum? Legitimately asking, not being snarky

          1 Reply Last reply Reply Quote 0
          • stacksofplatesS
            stacksofplates @wirestyle22
            last edited by

            @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

            So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently

            Usually comes from orphaned INodes. Can happen if the process halts or server is shutdown incorrectly. This is what the lost+found directory is for.

            1 Reply Last reply Reply Quote 1
            • scottalanmillerS
              scottalanmiller @wirestyle22
              last edited by

              @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

              So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently

              It did not. Deleting millions of unneeded files did.

              wirestyle22W 1 Reply Last reply Reply Quote 0
              • scottalanmillerS
                scottalanmiller @Obsolesce
                last edited by

                @tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.

                Maybe I need a good explanation, if someone doesn't mind.

                It's one of those weird things that on tests and in classrooms people talk about constantly as if it is an every day thing. But in the real world, with thousands of servers over decades of use, I've actually had this happen now... once. I am an idiot for not checking that straight away, but it is so insanely rare in the real world that it never occurred to me that it was a real possibility. Especially when the system had no reason to be making so many files.

                1 Reply Last reply Reply Quote 2
                • wirestyle22W
                  wirestyle22 @scottalanmiller
                  last edited by

                  @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                  scottalanmillerS 1 Reply Last reply Reply Quote 0
                  • scottalanmillerS
                    scottalanmiller @Obsolesce
                    last edited by

                    @tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                    Seems like a major Linux turn-off if this happens for no reason.

                    It's not Linux related. It's filesystem related. And it is so insanely rare that it's not a thing to worry about in the real world. Windows has inodes too, they call them File ID.

                    1 Reply Last reply Reply Quote 6
                    • scottalanmillerS
                      scottalanmiller @wirestyle22
                      last edited by

                      @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                      @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                      Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                      travisdh1T 1 Reply Last reply Reply Quote 2
                      • travisdh1T
                        travisdh1 @scottalanmiller
                        last edited by

                        @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                        @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                        @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                        Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                        Tiny files to, right?

                        JaredBuschJ scottalanmillerS 2 Replies Last reply Reply Quote 0
                        • JaredBuschJ
                          JaredBusch @travisdh1
                          last edited by JaredBusch

                          @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                          @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                          @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                          @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                          Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                          Tiny files to, right?

                          That is not relevant except for that fact that large files would have filled drive space and likely been noticed.

                          wirestyle22W 1 Reply Last reply Reply Quote 0
                          • scottalanmillerS
                            scottalanmiller @travisdh1
                            last edited by

                            @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                            @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                            @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                            @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                            Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                            Tiny files to, right?

                            Empty I think. Tons were just folders.

                            1 Reply Last reply Reply Quote 1
                            • wirestyle22W
                              wirestyle22 @JaredBusch
                              last edited by wirestyle22

                              @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                              @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                              @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                              @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                              @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                              Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                              Tiny files to, right?

                              That is not relevant except for that fact that large files would have filled drive space and likely been noticed.

                              How is that not relevant? More files = more inodes being used

                              JaredBuschJ 1 Reply Last reply Reply Quote 0
                              • JaredBuschJ
                                JaredBusch @wirestyle22
                                last edited by JaredBusch

                                @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                                Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                                Tiny files to, right?

                                That is not relevant except for that fact that large files would have filled drive space and likely been noticed.

                                How is that not relevant? more files = more inodes being used

                                Size of the files is not relevant. Pay attention to what is said.

                                travisdh1T 1 Reply Last reply Reply Quote 0
                                • travisdh1T
                                  travisdh1 @JaredBusch
                                  last edited by

                                  @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                  @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                  @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                  @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                  @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                  @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                  @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                                  Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                                  Tiny files to, right?

                                  That is not relevant except for that fact that large files would have filled drive space and likely been noticed.

                                  How is that not relevant? more files = more inodes being used

                                  Size of the files is not relevant.

                                  It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.

                                  Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.

                                  scottalanmillerS 1 Reply Last reply Reply Quote 0
                                  • stacksofplatesS
                                    stacksofplates
                                    last edited by

                                    Ya I've never run into it in the real world only on trick questions

                                    scottalanmillerS 1 Reply Last reply Reply Quote 1
                                    • scottalanmillerS
                                      scottalanmiller @stacksofplates
                                      last edited by

                                      @stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                      Ya I've never run into it in the real world only on trick questions

                                      That's why I was so surprised. This never happens!

                                      ObsolesceO 1 Reply Last reply Reply Quote 2
                                      • scottalanmillerS
                                        scottalanmiller @travisdh1
                                        last edited by

                                        @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                        @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                                        Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                                        Tiny files to, right?

                                        That is not relevant except for that fact that large files would have filled drive space and likely been noticed.

                                        How is that not relevant? more files = more inodes being used

                                        Size of the files is not relevant.

                                        It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.

                                        Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.

                                        The one major exception is marker files. "touch thishappened" as a file automatically and never clean up and you are using inodes without using any space. That's who you can easily learn about inode depletion. But who does that?

                                        stacksofplatesS 1 Reply Last reply Reply Quote 0
                                        • stacksofplatesS
                                          stacksofplates @scottalanmiller
                                          last edited by stacksofplates

                                          @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                          @scottalanmiller I could've sworn there was a post here related to a reboot solving it

                                          Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.

                                          Tiny files to, right?

                                          That is not relevant except for that fact that large files would have filled drive space and likely been noticed.

                                          How is that not relevant? more files = more inodes being used

                                          Size of the files is not relevant.

                                          It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.

                                          Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.

                                          The one major exception is marker files. "touch thishappened" as a file automatically and never clean up and you are using inodes without using any space. That's who you can easily learn about inode depletion. But who does that?

                                          People writing bad scripts that use lock files and forget to remove them.

                                          That wouldn't create enough to do this though.

                                          RamblingBipedR 1 Reply Last reply Reply Quote 0
                                          • ObsolesceO
                                            Obsolesce @scottalanmiller
                                            last edited by

                                            @scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                            @stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

                                            Ya I've never run into it in the real world only on trick questions

                                            That's why I was so surprised. This never happens!

                                            I understand. I've done a lot on Linux over the years and I've never run into anything having to do with inodes. That's why I was clueless and had to ask.

                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 2 / 2
                                            • First post
                                              Last post