如何新增 LibreNMS 不支援設備的監控

 不確定是不是因為 LibreNMS 版本太舊,還是 MIB 檔沒更新,導致無法讓 LibreNMS 無法識別出

Ubiquiti Unifi WiFi-AP / Switch , 也可能設備太新導致.經過一番 google 後,有看到相關資訊,如下 

URL https://community.librenms.org/t/ubiquiti-devices/11341

官方相關的 URL為  https://docs.librenms.org/Developing/os/Initial-Detection/

看了一下目前設定,並修改如下:

[root@LibreNMS definitions]# pwd

/opt/librenms/includes/definitions

[root@LibreNMS definitions]#

[root@LibreNMS definitions]# cat unifi.yaml 

os: unifi

text: 'Ubiquiti UniFi'

type: wireless

icon: ubiquiti

mib_dir: ubnt

over:

    - { graph: device_bits, text: 'Device Traffic' }

    - { graph: device_wireless_clients, text: 'Connected Clients' }

    - { graph: device_wireless_ccq, text: 'Connection Quality' }

discovery:

    -

        sysObjectID:

            - .1.3.6.1.4.1.41112

            - .1.3.6.1.4.1.8072.3.2.10

        sysDescr_regex:

            - '/^UAP/'

            - '/^U6/'

            - '/^U7-Pro/'

    -

        sysObjectID: .1.3.6.1.4.1.10002.1

        sysDescr: Linux

        snmpwalk:

            oid: 'IEEE802dot11-MIB::dot11manufacturerProductName'

            op: 'contains'

            value: 'UAP'

[root@LibreNMS definitions]# cat edgeswitch.yaml 

os: edgeswitch

text: EdgeSwitch

type: network

icon: ubiquiti

ifname: true

bad_snmpEngineTime: true

over:

    - { graph: device_bits, text: 'Device Traffic' }

    - { graph: device_processor, text: 'CPU Usage' }

    - { graph: device_mempool, text: 'Memory Usage' }

discovery:

    -

        sysObjectID:

            - .1.3.6.1.2.1.1.1.0

            - .1.3.6.1.4.1.4413

            - .1.3.6.1.4.1.41112

            - .1.3.6.1.4.1.10002

        sysDescr_regex:

            - '/^EdgeSwitch/'

            - '/^EdgePoint/'

            - '/^USW[ -]/'

            - '/^UBNT US/'

            - '/^US[ -]/'

    -

        sysObjectID:

            - .1.3.6.1.4.1.8072.3.2.10

            - .1.3.6.1.4.1.27282.3.2.10

        sysDescr_regex:

            - '/^Linux UBNT/'

            - '/^RTL8380/'

            - '/^Linux.* 4.4.153/'

            - '/^Linux.* 3.18.24/'

以上檔案的內容定義可能會跟所使用的 LibreNMS 的版本不同,而有所不一樣.

如  LibreNMS　Version 24.11.0-66-g47a9e2b7d

[root@node1 definitions]# cat unifi.yaml 

os: unifi

text: 'Ubiquiti UniFi'

type: wireless

icon: ubiquiti

mib_dir: ubnt

over:

    - { graph: device_bits, text: 'Device Traffic' }

    - { graph: device_wireless_clients, text: 'Connected Clients' }

    - { graph: device_wireless_ccq, text: 'Connection Quality' }

discovery:

    -

        sysObjectID:

            - .1.3.6.1.4.1.41112

            - .1.3.6.1.4.1.8072.3.2.10

        sysDescr_regex:

            - '/^UAP/'

            - '/^U6/'

            - '/^U7/'

            - '/^U-LTE/'

    -

        sysObjectID: .1.3.6.1.4.1.10002.1

        sysDescr: Linux

        snmpwalk:

            oid: 'IEEE802dot11-MIB::dot11manufacturerProductName'

            op: 'contains'

            value: 'UAP'

[root@node1 definitions]# 

[root@node1 definitions]# cat edgeswitch.yaml 

os: edgeswitch

text: EdgeSwitch

type: network

icon: ubiquiti

ifname: true

bad_snmpEngineTime: true

over:

    - { graph: device_bits, text: 'Device Traffic' }

    - { graph: device_processor, text: 'CPU Usage' }

    - { graph: device_mempool, text: 'Memory Usage' }

discovery:

    -

        sysObjectID:

            - .1.3.6.1.4.1.4413

            - .1.3.6.1.4.1.41112

            - .1.3.6.1.4.1.10002

            - .1.3.6.1.4.1.8072

            - .1.3.6.1.4.1

        sysDescr_regex:

            - '/^EdgeSwitch/'

            - '/^EdgePoint/'

            - '/^USW[ -]/'

            - '/^UBNT US/'

            - '/^US[ -]/'

    -

        sysObjectID:

            - .1.3.6.1.4.1.8072.3.2.10

            - .1.3.6.1.4.1.27282.3.2.10

        sysDescr_regex:

            - '/^Linux UBNT/'

            - '/^RTL8380/'

[root@node1 definitions]# 

另外有一個地方的設定,也是相關的.

[root@node1 discovery]# pwd

/opt/librenms/includes/definitions/discovery

[root@node1 discovery]# 

[root@node1 discovery]# cat unifi.yaml 

modules:

    mempools:

        data:

            -

                total: FROGFOOT-RESOURCES-MIB::memTotal

                free: FROGFOOT-RESOURCES-MIB::memFree

                precision: 1024

[root@node1 discovery]# 

[root@node1 discovery]# cat edgeswitch.yaml 

mib: EdgeSwitch-BOXSERVICES-PRIVATE-MIB

modules:

    mempools:

        data:

            -

                total: EdgeSwitch-SWITCHING-MIB::agentSwitchCpuProcessMemAvailable

                free: EdgeSwitch-SWITCHING-MIB::agentSwitchCpuProcessMemFree

                precision: 1024

    os:

        sysDescr_regex: '/^(?<hardware>EdgeSwitch[ \d\w\-]*|EdgePoint Switch[ \d\w\-]*|US[ \d\w\-]*)(,)* (firmware )*(?<version>\S*)(, Linux .*)*$/'

        serial: ENTITY-MIB::entPhysicalSerialNum.1

    sensors:

        state:

            data:

                -

                    oid: boxServicesTempUnitEntry

                    value: boxServicesTempUnitState

                    num_oid: '.1.3.6.1.4.1.4413.1.1.43.1.15.1.2.{{ $index }}'

                    descr: 'Chassis state'

                    index: 'boxServicesTempUnitState.{{ $index }}'

                    states:

                        - { value: 0, descr: low, graph: 1, generic: 1 }

                        - { value: 1, descr: normal, graph: 1, generic: 0 }

                        - { value: 2, descr: warning, graph: 1, generic: 1 }

                        - { value: 3, descr: critical, graph: 1, generic: 2 }

                        - { value: 4, descr: shutdown, graph: 1, generic: 2 }

                        - { value: 5, descr: notpresent, graph: 1, generic: 3 }

                        - { value: 6, descr: notoperational, graph: 1, generic: 2 }

                -

                    oid: boxServicesTempSensorsEntry

                    value: boxServicesTempSensorState

                    num_oid: '.1.3.6.1.4.1.4413.1.1.43.1.8.1.4.{{ $index }}'

                    descr: 'Temp Sensor {{ $index }}'

                    index: 'boxServicesTempSensorState.{{ $index }}'

                    states:

                        - { value: 0, descr: low, graph: 1, generic: 1 }

                        - { value: 1, descr: normal, graph: 1, generic: 0 }

                        - { value: 2, descr: warning, graph: 1, generic: 1 }

                        - { value: 3, descr: critical, graph: 1, generic: 2 }

                        - { value: 4, descr: shutdown, graph: 1, generic: 2 }

                        - { value: 5, descr: notpresent, graph: 1, generic: 3 }

                        - { value: 6, descr: notoperational, graph: 1, generic: 2 }

                -

                    oid: boxServicesFansEntry

                    value: boxServicesFanItemState

                    num_oid: '.1.3.6.1.4.1.4413.1.1.43.1.6.1.3.{{ $index }}'

                    descr: 'Fan {{ $index }}'

                    index: 'boxServicesFanItemState.{{ $index }}'

                    states:

                        - { value: 1, descr: notpresent, graph: 1, generic: 3 }

                        - { value: 2, descr: operational, graph: 1, generic: 0 }

                        - { value: 3, descr: failed, graph: 1, generic: 2 }

                        - { value: 4, descr: powering, graph: 1, generic: 0 }

                        - { value: 5, descr: nopower, graph: 1, generic: 2 }

                        - { value: 6, descr: notpowering, graph: 1, generic: 2 }

                        - { value: 7, descr: incompatible, graph: 1, generic: 1 }

        fanspeed:

            data:

                -

                    oid: boxServicesFansEntry

                    value: boxServicesFanSpeed

                    num_oid: '.1.3.6.1.4.1.4413.1.1.43.1.6.1.4.{{ $index }}'

                    index: 'boxServicesFanSpeed.{{ $index }}'

                    descr: 'Fan {{ $index }}'

        temperature:

            data:

                -

                    oid: boxServicesTempSensorsEntry

                    value: boxServicesTempSensorTemperature

                    num_oid: '.1.3.6.1.4.1.4413.1.1.43.1.8.1.5.{{ $index }}'

                    index: 'boxServicesTempSensorTemperature.{{ $index }}'

                    descr: 'Temperature {{ $index }}'

[root@node1 discovery]# 

[root@LibreNMS definitions]#  snmpwalk -Os -c public  -v 2c UniFi-Switch system

sysDescr.0 = STRING: Linux UniFi-Switch  4.4.153 #0 Thu Aug 30 12:10:54 2018 mips

sysObjectID.0 = OID: netSnmpAgentOIDs.10

sysUpTimeInstance = Timeticks: (213856100) 24 days, 18:02:41.00

sysContact.0 = STRING: Default

sysName.0 = STRING: UniFi-Switch

sysLocation.0 = STRING: Default

[root@LibreNMS definitions]# 

使用 PHP 試寫 LibreNMS 串接 Jandi 通知

 LibreNMS 預設是支援常見的 Line Notify,但不支援 Jandi 的.

所以試寫一個 Jandi 的 alert-transports 

Demo 如下所示.

PS:以下程式的串接寫下僅適用於 librenms-22.12.x 以下的版本.

而且是抄改自  Linenotify.php 原碼,如下所示

[root@Oracle9 Transport]# pwd

/opt/librenms/LibreNMS/Alert/Transport

[root@Oracle9 Transport]# cat Linenotify.php

<?php

/**

 * LINE Notify Transport

 */

namespace LibreNMS\Alert\Transport;

use LibreNMS\Alert\Transport;

use LibreNMS\Util\Proxy;

class Linenotify extends Transport

{

    protected $name = 'LINE Notify';

    public function deliverAlert($obj, $opts)

    {

        $opts['line-notify-access-token'] = $this->config['line-notify-access-token'];

        return $this->contactLinenotify($obj, $opts);

    }

    private function contactLinenotify($obj, $opts)

    {

        $lineUrl = 'https://notify-api.line.me/api/notify';

        $lineHead = ['Authorization: Bearer ' . $opts['line-notify-access-token']];

        $lineFields = ['message' => $obj['msg']];

        $curl = curl_init();

        Proxy::applyToCurl($curl);

        curl_setopt($curl, CURLOPT_URL, $lineUrl);

        curl_setopt($curl, CURLOPT_HTTPHEADER, $lineHead);

        curl_setopt($curl, CURLOPT_NOBODY, false);

        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

        curl_setopt($curl, CURLOPT_POST, true);

        curl_setopt($curl, CURLOPT_POSTFIELDS, $lineFields);

        curl_exec($curl);

        $code = curl_getinfo($curl, CURLINFO_HTTP_CODE);

        curl_close($curl);

        if ($code != 200) {

            return 'HTTP Status code ' . $code;

        }

        return true;

    }

    public static function configTemplate()

    {

        return [

            'config' => [

                [

                    'title' => 'Token',

                    'name' => 'line-notify-access-token',

                    'descr' => 'LINE Notify Token',

                    'type' => 'text',

                ],

            ],

            'validation' => [

                'line-notify-access-token' => 'required|string',

            ],

        ];

    }

}

[root@Oracle9 Transport]#

抄改成 Jandi 的程式如下所示 

[root@Oracle9 Transport]# pwd

/opt/librenms/LibreNMS/Alert/Transport

[root@Oracle9 Transport]# cat Jandi.php

<?php

/**

 * Jandi webhook Transport

 */

namespace LibreNMS\Alert\Transport;

use LibreNMS\Alert\Transport;

use LibreNMS\Util\Proxy;

class Jandi extends Transport

{

    protected $name = 'Jandi';

    public function deliverAlert($obj, $opts)

    {

        $opts['jandi-webhook-access-token'] = $this->config['jandi-webhook-access-token'];

        $opts['jandi-webhook-access-rec'] = $this->config['jandi-webhook-access-rec'];

        return $this->contactLineNotify($obj, $opts);

    }

    private function contactLinenotify($obj, $opts)

    {

        $webhooktoken=$opts['jandi-webhook-access-token'];

        $webhookid=$opts['jandi-webhook-access-rec'];

        $lineUrl = "https://wh.jandi.com/connect-api/webhook/$webhooktoken/$webhookid";

        $lineHead = ['Accept: application/vnd.tosslab.jandi-v2+json','Content-Type: application/json'];

        // $lineFields = [body' => body , 'connectColor' => '#FAC11B' , 'connectInfo' => [ 'title' => title , 'description' => $obj['msg'] ] ];

        $jmsg=$obj['msg'];

        $nowtime=date('Y-m-d H:i:s');

        $lineFields = <<<DATA

        {

        "body" : "$nowtime",

        "connectColor" : "#FAC11B",

        "connectInfo" : [

           { "title" : "AlertMessage",

             "description" : "$jmsg"

           }

         ]

        }

        DATA;

        // $lineFields = ["body" => body];

        $curl = curl_init();

        curl_setopt($curl, CURLOPT_URL, $lineUrl);

        curl_setopt($curl, CURLOPT_HTTPHEADER, $lineHead);

        curl_setopt($curl, CURLOPT_NOBODY, false);

        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

        curl_setopt($curl, CURLOPT_POST, true);

        curl_setopt($curl, CURLOPT_POSTFIELDS, $lineFields);

        curl_exec($curl);

        $code = curl_getinfo($curl, CURLINFO_HTTP_CODE);

        curl_close($curl);

        if ($code != 200) {

            return 'HTTP Status code ' . $code;

        }

        return true;

    }

    public static function configTemplate()

    {

        return [

            'config' => [

                [

                    'title' => 'Jandi WebHook Token',

                    'name' => 'jandi-webhook-access-token',

                    'descr' => 'Jandi WEBHOOK Token',

                    'type' => 'text',

                ],

                [

                    'title' => 'Jandi WebHook Recipient',

                    'name' => 'jandi-webhook-access-rec',

                    'descr' => 'Jandi WEBHOOK Recipient',

                    'type' => 'text',

                ],

            ],

            'validation' => [

                'jandi-webhook-access-token' => 'required|string',

                'jandi-webhook-access-rec' => 'required|string',

            ],

        ];

    }

}

[root@Oracle9 Transport]#

DEMO

將 Oxidized 外掛上 Librenms

 最近又有人在問我如何做 Cisco 或 FortiGate 設定組態 (configuration)

如何自動備份?

N年前我就有寫過類似的東西,只不過在當時我是用 Perl + telnet DIY小程式去做.

當然也是改成 Perl + ssh DIY 的小程式去做.但 N 年過去了,

目前已有太多工具可以逹成這個目地.

接下來的筆記是記錄,我是如何透過 Oxidized  去做這件事,

順便將其整合進 LibreNMS內,

當然只使用 Oxidized 也是 OK 的.

參考文件

http://blog.jason.tools/2021/02/librenms-oxidized.html

官方 URL

https://docs.librenms.org/Extensions/Oxidized/

https://github.com/ytti/oxidized

之前寫的東西

http://xrcd2.blogspot.com/2013/02/cisco-show-run-config-perl.html

自動備份 cisco 設備 show run 的 config ( 使用 perl )

http://xrcd2.blogspot.com/2016/01/cisco-ios-configuration-cisco-config.html

Cisco IOS 設備 configuration 自動版本管理的方式 ( cisco config auto 2 svn )

------------------------------------------------------------------------

安裝 Oxidized (OS CentOS 7.9)

yum install -y centos-release-scl-rh

yum install -y rh-ruby24 rh-ruby24-ruby-devel

yum install make cmake which sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel libicu-devel gcc-c++

scl enable rh-ruby24 bash

Install the gems: 




gem install oxidized oxidized-web

===============

git 設定

git config --global user.name "cisco"

git config --global user.email "cisco@cisco.lab.local"

===============

[root@centos7 ~]# pwd

/root

[root@centos7 ~]# 

[root@centos7 ~]# oxidized

edit ~/.config/oxidized/config

[root@centos7 ~]# oxidized

edit ~/.config/oxidized/router.db 

設定檔參考如下:

[root@centos7 oxidized]# pwd

/root/.config/oxidized

[root@centos7 oxidized]# cat config

---

username: username

password: password

model: ios

resolve_dns: false

interval: 300

log: /root/.config/oxidized/logs/oxidized.log

use_syslog: false

debug: false

threads: 30

timeout: 20

retries: 3

prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/

rest: 192.168.100.105:8888

next_adds_job: false

vars: {}

groups: {}

models: {}

pid: "/root/.config/oxidized/pid"

crash:

  directory: "/root/.config/oxidized/crashes"

  hostnames: false

stats:

  history_size: 10

input:

  default: ssh, telnet

  debug: false

  ssh:

    secure: false

  ftp:

    passive: true

  utf8_encoded: true

output:

  default: git

  git:

    user: cisco

    email: cisco@cisco.lab.local

    repo: "/root/.config/oxidized/configs/configs.git"

source:

  default: csv

  csv:

    file: "/root/.config/oxidized/router.db"

    delimiter: !ruby/regexp /:/

    map:

      name: 0

      model: 1

      ip: 2

      username: 3

      password: 4

      group: 5

    gpg: false

model_map:

  juniper: junos

  cisco: ios

============================

[root@centos7 oxidized]# cat router.db 

csr1000v:ios:192.168.100.222:cisco:cisco:configs

[root@centos7 oxidized]# 

=============================

整合進 librenms

===================

[root@centos7 oxidized]# su - librenms

Last login: Sat Jul 16 12:39:31 CST 2022 on pts/0

-bash-4.2$ lnms config:set oxidized.enabled true

-bash-4.2$ lnms config:set oxidized.url http://192.168.100.105:8888

-bash-4.2$ lnms config:set oxidized.features.versioning true

-bash-4.2$ lnms config:set oxidized.group_support true

-bash-4.2$ lnms config:set oxidized.default_group default

-bash-4.2$ lnms config:set oxidized.reload_nodes true

-bash-4.2$ 

=======================

Cisco CSR 1000v  show run 

==================

csr1000v#sh run 

Building configuration...

Current configuration : 1204 bytes

!

! Last configuration change at 13:29:30 TPE Sat Jul 16 2022 by cisco

!

version 15.5

service timestamps debug datetime localtime

service timestamps log datetime localtime

no platform punt-keepalive disable-kernel-core

platform console auto

!

hostname csr1000v

!

boot-start-marker

boot-end-marker

!

!

!

no aaa new-model

clock timezone TPE 8 0

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!

!         

subscriber templating

!

multilink bundle-name authenticated

!

!

!

license udi pid CSR1000V sn XXXXXXXXXXXXX

license boot level ax

spanning-tree extend system-id

!

username cisco privilege 15 secret 5 $1$7wax$evNlQZGH2VorRL3bm/SRV0

!

redundancy

!

!

!

!

!

!

! 

!

!

!

!

!

!

!

!

!

!

!

!

! 

! 

!

interface GigabitEthernet1

 ip address 192.168.100.222 255.255.255.0

 negotiation auto

!

interface GigabitEthernet2

 no ip address

 shutdown

 negotiation auto

!

interface GigabitEthernet3

 no ip address

 shutdown

 negotiation auto

!

!

virtual-service csr_mgmt

 ip shared host-interface GigabitEthernet1

!

ip forward-protocol nd

!

no ip http server

ip http secure-server

!

!

snmp-server community cisco RO

!

!

control-plane

!

!

line con 0

 stopbits 1

line vty 0 4

 login local

 transport input ssh

!

ntp server 168.95.195.12

!

end

csr1000v# 

=====================

將  oxidized 設定成服務

====================

vi /etc/profile.d/rh-ruby24.sh

#!/bin/bash

source /opt/rh/rh-ruby24/enable

export X_SCLS="`scl enable rh-ruby24 'echo $X_SCLS'`"

export PATH=$PATH:/opt/rh/rh-ruby24/root/usr/bin/ruby

ln -s /opt/rh/rh-ruby24/root/usr/local/bin/oxidized /usr/local/bin/oxidized

====================

vi  /lib/systemd/system/oxidized.service

 

# /lib/systemd/system/oxidized.service

[Unit]

Description=Oxidized - Network Device Configuration Backup Tool

After=network-online.target multi-user.target

Wants=network-online.target

 

[Service]

ExecStart=/usr/local/bin/oxidized

KillSignal=SIGKILL

User=root

 

[Install]

WantedBy=multi-user.target

===========================

 

chmod +x /lib/systemd/system/oxidized.service

 

vi /etc/ld.so.conf

#增加 /opt/rh/rh-ruby24/root/usr/lib64

 

ldconfig

ldconfig -v

#檢查 ruby ldconf

 

chmod +x /lib/systemd/system/oxidized.service

 

systemctl enable oxidized.service

systemctl start oxidized.service

systemctl status oxidized.service

====================

Demo

LibreNMS平均CPU使用率監控功能

  最近有一個需求是要去CPU的監控,但使用的方式為 

processors.processor_usage >= processor_perc_warn

這個方式去監控是依據每一顆CPU去做監控的.如果該主機有 8 顆CPU,

其中有 1 顆 超過告警值 即會發出告警,但其它 7 顆可能沒那麼重的 loading .

這種方式的監控似乎不是那麼合理.理想的情境應是依 這 8 顆的平均值去做告警才是.

在 官方 https://docs.librenms.org/Alerting/Rules/ 的 Advanced 處,

有看到一個 SQL 語法.

SELECT *,AVG(processors.processor_usage) as cpu_avg FROM devices,processors WHERE (devices.device_id = ? AND devices.device_id = processors.device_id) AND (devices.status = 1 && (devices.disabled = 0 && devices.ignore = 0)) = 1 HAVING AVG(processors.processor_usage) > 10

後來試一下真的可以用.

壓測方式

[root@librenms CPULoadGenerator]# python3.6 -m cpu_load_generator -l 0.8 -d 600 -c -1

使用工具

https://pypi.org/project/cpu-load-generator/

Project description

CPU Load Generator

This package allows to generate a fixed CPU load for a finite time period. 

The script takes in input the desired CPU load, the duration of the experiment 

and the CPU core or all cores on which the load has to be generated.

Python versions This master branch refers to Python versions 3.x

Dependencies Installment of psutil is required 

(it is already set as dependency in the setup.py)

Install psutil:

pip install psutil

Other dependencies for unit tests and linters:

pip install mock pytest flake8 tox

Usage To generate 20% of load on core 0 for 20 seconds run:

python -m cpu_load_generator -l 0.2 -d 20 -c 0

To generate 50% of load on all logical cores for 20 seconds run:

python -m cpu_load_generator -l 0.5 -d 20 -c -1

There is an option to run CPU load based on profile file. An exemplary profile 

is under load_profiles/default_profile.json. In order to run generator based on

a profile issue the following command:

python -m cpu_load_generator -p <path_to_profile_json>

To use the package features from python code:

Install the package from PyPi by issuing the following command:

python -m pip instal cpu-load-generator

To use its features from your code:

from cpu_load_generator import load_single_core, load_all_cores, from_profile

load_single_core(core_num=0, duration_s=20, target_load=0.4)  # generate load on single core (0)

load_all_cores(duration_s=30, target_load=0.2)  # generates load on all cores

from_profile(path_to_profile_json=r"c:\profiles\profile1.json")

DEMO

LibreNMS 外掛 Service 監控功能

源自 Nagios Plugins - Services 

https://docs.librenms.org/Extensions/Services/

Demo

Setting

======================================================

Service

{{ $alert->title }}

@if ($alert->faults)

@foreach ($alert->faults as $key => $value) {{ $value['service_desc'] }} - {{ $value['service_type'] }}

{{ $value['service_message'] }}

@endforeach

@endif

Error=========

Service: {{ $value['service_name'] }} {{ $value['service_type'] }}

=============

=========================================================

Other

SELECT * FROM devices,services WHERE (devices.device_id = ? AND devices.device_id = services.device_id) AND services.service_status != 0 AND (devices.status = 1 && (devices.disabled = 0 && devices.ignore = 0)) = 1