What is the difference between Dead and Failed for service status?

IRJ

I am doing some testing to monitor services via command. I need an expected response when a service is considered down.

During my testing I got two different results after stopping a service.

################

First service - Suricata

#################

sudo systemctl show -p SubState --value suricata

the state returned from that command is running as we would expect.

Now we run sudo systemctl stop suricata

Then run sudo systemctl show -p SubState --value suricata which returns failed.

################

Second service - Wazuh-Agent

#################

sudo systemctl show -p SubState --value wazuh-agent

The state returned from that command is running as we would expect.

Now we run sudo systemctl stop wazuh-agent

Then run sudo systemctl show -p SubState --value wazuh-agent which returns dead.

DustinB3403

Did you look at man systemctl to get more details yet? I just checked, and while I don't see the specific status listed in the manual, there are a few links to additional resources.

DustinB3403

From the docs I can find, the service isn't set to start on boot, and also isn't running. So it would be inactive (dead) in that case.

IRJ

It looks like dead is the default state of all other services after being stopped.

DustinB3403

But this is a child process of Wazuh, no? So would the forking flag need to be set to accurately report the status of the service?

Not about wazuh, but may pertain.

IRJ

@DustinB3403 said in What is the difference between Dead and Failed for service status?:

From the docs I can find, the service isn't set to start on boot, and also isn't running. So it would be inactive (dead) in that case.

Yeah the parameter ActiveState would return inactive

SubState returns dead

IRJ

@DustinB3403 said in What is the difference between Dead and Failed for service status?:

But this is a child process of Wazuh, no? So would the forking flag need to be set to accurately report the status of the service?

Not about wazuh, but may pertain.

No these ones are the children

DustinB3403

@IRJ said in What is the difference between Dead and Failed for service status?:

sudo systemctl show -p SubState --value wazuh-agent

Oh I see what you have going on (a bit tired cranky kid last night).

From the top

Active: active (running) means the service is setup to start on boot and is running.

Active: failed means the service crashed for some reason. Check journalctl for more details

Active: inactive (dead) means not set to start on boot, and not running.

DustinB3403

Also you could setup your monitoring to report/alert for "Anything but active (running)". It would have the same effect, because in any case of the services not running this system function stops working.

DustinB3403

Did you go through the systemctl enable wazuh-agent so the service is started at boot?

DustinB3403

There are also other SUB status as show here

 systemctl list-units --type service --all
  UNIT                                                  LOAD      ACTIVE   SUB     DESCRIPTION
  auditd.service                                        loaded    active   running Security Auditing Service
  brandbot.service                                      loaded    inactive dead    Flexible Branding Service
  chronyd.service                                       loaded    active   running NTP client/server
  cpupower.service                                      loaded    inactive dead    Configure CPU power related settings
  crond.service                                         loaded    active   running Command Scheduler
  dbus.service                                          loaded    active   running D-Bus System Message Bus
● display-manager.service                               not-found inactive dead    display-manager.service
  dm-event.service                                      loaded    inactive dead    Device-mapper event daemon
  dracut-shutdown.service                               loaded    inactive dead    Restore /run/initramfs
  ebtables.service                                      loaded    inactive dead    Ethernet Bridge Filtering tables
  emergency.service                                     loaded    inactive dead    Emergency Shell
● exim.service                                          not-found inactive dead    exim.service
  firewalld.service                                     loaded    active   running firewalld - dynamic firewall daemon
  [email protected]                                    loaded    active   running Getty on tty1
  httpd.service                                         loaded    active   running The Apache HTTP Server
● ip6tables.service                                     not-found inactive dead    ip6tables.service
● ipset.service                                         not-found inactive dead    ipset.service
● iptables.service                                      not-found inactive dead    iptables.service
  irqbalance.service                                    loaded    inactive dead    irqbalance daemon
  kdump.service                                         loaded    active   exited  Crash recovery kernel arming
  kmod-static-nodes.service                             loaded    active   exited  Create list of required static device nodes for the current kernel
● lvm2-activation.service                               not-found inactive dead    lvm2-activation.service
  lvm2-lvmetad.service                                  loaded    active   running LVM2 metadata daemon
  lvm2-lvmpolld.service                                 loaded    inactive dead    LVM2 poll daemon
  lvm2-monitor.service                                  loaded    active   exited  Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
  lvm2-pvscan@8:2.service                               loaded    active   exited  LVM2 PV scan on device 8:2
  mariadb.service                                       loaded    active   running MariaDB database server
  microcode.service                                     loaded    inactive dead    Load CPU microcode update
  network.service                                       loaded    active   exited  LSB: Bring up/down networking
  NetworkManager-wait-online.service                    loaded    active   exited  Network Manager Wait Online
  NetworkManager.service                                loaded    active   running Network Manager
● ntpd.service                                          not-found inactive dead    ntpd.service
● ntpdate.service                                       not-found inactive dead    ntpdate.service
  plymouth-quit-wait.service                            loaded    inactive dead    Wait for Plymouth Boot Screen to Quit
  plymouth-quit.service                                 loaded    inactive dead    Terminate Plymouth Boot Screen
  plymouth-read-write.service                           loaded    inactive dead    Tell Plymouth To Write Out Runtime Data
  plymouth-start.service                                loaded    inactive dead    Show Plymouth Boot Screen
  polkit.service                                        loaded    active   running Authorization Manager
  postfix.service                                       loaded    active   running Postfix Mail Transport Agent
  rc-local.service                                      loaded    inactive dead    /etc/rc.d/rc.local Compatibility
  rescue.service                                        loaded    inactive dead    Rescue Shell
  rhel-autorelabel-mark.service                         loaded    inactive dead    Mark the need to relabel after reboot
  rhel-autorelabel.service                              loaded    inactive dead    Relabel all filesystems, if necessary
  rhel-configure.service                                loaded    inactive dead    Reconfigure the system on administrator request
  rhel-dmesg.service                                    loaded    active   exited  Dump dmesg to /var/log/dmesg
  rhel-domainname.service                               loaded    active   exited  Read and set NIS domainname from /etc/sysconfig/network
  rhel-import-state.service                             loaded    active   exited  Import network configuration from initramfs

IRJ

@DustinB3403 said in What is the difference between Dead and Failed for service status?:

Did you go through the systemctl enable wazuh-agent so the service is started at boot?

Yeah that has been done. I was manually stopping services to force changing state so I could see the output.

IRJ

@DustinB3403 said in What is the difference between Dead and Failed for service status?:

Also you could setup your monitoring to report/alert for "Anything but active (running)". It would have the same effect, because in any case of the services not running this system function stops working.

I found another anomaly

I get this when I check app armor status via apparmor_status

IRJ

@DustinB3403 said in What is the difference between Dead and Failed for service status?:

There are also other SUB status as show here

systemctl list-units --type service --all
UNIT LOAD ACTIVE SUB DESCRIPTION
auditd.service loaded active running Security Auditing Service
brandbot.service loaded inactive dead Flexible Branding Service
chronyd.service loaded active running NTP client/server
cpupower.service loaded inactive dead Configure CPU power related settings
crond.service loaded active running Command Scheduler
dbus.service loaded active running D-Bus System Message Bus
● display-manager.service not-found inactive dead display-manager.service
dm-event.service loaded inactive dead Device-mapper event daemon
dracut-shutdown.service loaded inactive dead Restore /run/initramfs
ebtables.service loaded inactive dead Ethernet Bridge Filtering tables
emergency.service loaded inactive dead Emergency Shell
● exim.service not-found inactive dead exim.service
firewalld.service loaded active running firewalld - dynamic firewall daemon
[email protected] loaded active running Getty on tty1
httpd.service loaded active running The Apache HTTP Server
● ip6tables.service not-found inactive dead ip6tables.service
● ipset.service not-found inactive dead ipset.service
● iptables.service not-found inactive dead iptables.service
irqbalance.service loaded inactive dead irqbalance daemon
kdump.service loaded active exited Crash recovery kernel arming
kmod-static-nodes.service loaded active exited Create list of required static device nodes for the current kernel
● lvm2-activation.service not-found inactive dead lvm2-activation.service
lvm2-lvmetad.service loaded active running LVM2 metadata daemon
lvm2-lvmpolld.service loaded inactive dead LVM2 poll daemon
lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
lvm2-pvscan@8:2.service loaded active exited LVM2 PV scan on device 8:2
mariadb.service loaded active running MariaDB database server
microcode.service loaded inactive dead Load CPU microcode update
network.service loaded active exited LSB: Bring up/down networking
NetworkManager-wait-online.service loaded active exited Network Manager Wait Online
NetworkManager.service loaded active running Network Manager
● ntpd.service not-found inactive dead ntpd.service
● ntpdate.service not-found inactive dead ntpdate.service
plymouth-quit-wait.service loaded inactive dead Wait for Plymouth Boot Screen to Quit
plymouth-quit.service loaded inactive dead Terminate Plymouth Boot Screen
plymouth-read-write.service loaded inactive dead Tell Plymouth To Write Out Runtime Data
plymouth-start.service loaded inactive dead Show Plymouth Boot Screen
polkit.service loaded active running Authorization Manager
postfix.service loaded active running Postfix Mail Transport Agent
rc-local.service loaded inactive dead /etc/rc.d/rc.local Compatibility
rescue.service loaded inactive dead Rescue Shell
rhel-autorelabel-mark.service loaded inactive dead Mark the need to relabel after reboot
rhel-autorelabel.service loaded inactive dead Relabel all filesystems, if necessary
rhel-configure.service loaded inactive dead Reconfigure the system on administrator request
rhel-dmesg.service loaded active exited Dump dmesg to /var/log/dmesg
rhel-domainname.service loaded active exited Read and set NIS domainname from /etc/sysconfig/network
rhel-import-state.service loaded active exited Import network configuration from initramfs

Probably best to just look at active and inactive.

IRJ

I tried using FirstState and still receiving failed from suricata.

`

DustinB3403

@IRJ Did you look at journalctl to see what the logs say?

JaredBusch

@DustinB3403 said in What is the difference between Dead and Failed for service status?:

@IRJ Did you look at journalctl to see what the logs say?

He doesn't care about why things are failed. He just wants the states. It is for monitoring and automation.