M525 stopped working



  • Dear Unipi Support Team,

    After 10 days of smoth operation, my new M525 PLC has stopped working and I have the following error with dmesg:

    [ 1696.881917] EXT4-fs (mmcblk0p2): re-mounted. Opts: errors=remount-ro,commit=120,discard
    [ 1696.942839] EXT4-fs (mmcblk0p2): re-mounted. Opts: errors=remount-ro,commit=120,discard
    [ 1697.239529] RTL8211E Gigabit Ethernet stmmac-0:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
    [ 1697.242547] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available
    [ 1697.242559] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
    [ 1697.242990] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [ 1701.336275] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
    [ 1701.336369] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [12755.149390] hrtimer: interrupt took 656215 ns
    [863475.515873] i2c i2c-0: mv64xxx_i2c_fsm: Ctlr Error -- state: 0x2, status: 0x0, addr: 0x18, flags: 0x1
    

    After PLC reboot, it still did not work properly, especially modbus communication trought
    serial port RS485-1.1 with 3 energy meters (Inepro PRO380)

    I did a complete reflash of the Mervis image and redeployed my projet and now it works again.
    Do you have an idea what happened?

    Many thanks in advance for your help and kind regards


  • administrators

    Hello @pascal37,
    can you please described how it stopped working? What software do you use?

    Thank you and have a nice day,
    Martin



  • I am running latest available Mervis version (2.2.0.30). After 10 days of operation, the modbus communication over serial port RS485-1.1 failed. I am reading some values from 3 Inepro power meters.



  • Hi @Martin-Kudláček,

    I found article: https://kb.unipi.technology/en:sw:04-unipi-firmware?s[]=firmware

    I had firmware v5.40 and just updated now to v5.50.


  • administrators

    Hello @pascal37,
    all of that is ok. You can check what is going on on the serial port in Mervis IDE directly. Switch to debugging mode and in the bottom panel look for "port monitor". Start the monitoring of the affected port. You should see more information.

    Also, try disconnecting the meters to see if the problem is caused by one of them. And also make sure, that you use shielded twist pair cables, with the shielding connected to earth ground on one end and use proper termination of the bus.

    Best regards,
    Martin



  • Hi @martin-kudláček,

    The problem recurred, again after 10 days of smooth operation.

    Here is the output of the serial port:

    0_1587715636275_bfb576df-88b0-4c6c-8428-23404e352219-image.png

    Since the solution works fine during multiple days without any error on the serial interface, I do not think there is an issue on the RS485 bus. Yes, I am using am appropriate shielded cable with terminations on both ends.

    I recently replaced the M505 with an M525 because of relay issues but I never had this RS485 serial bus issue with the M505 !?

    There are 3 Inepro PRO380 devices on the bus, I tried to disconnect each Meter, one after the other, but the issue seems not caused by one of them. I have also connected each meter individually direcly to the M525 with another cable but no success either.

    Anything else we can do on the M525 to further troublehoot before I try restart/redeploy the solution ?

    Many thanks in advance and kind regards,

    Pascal



  • Hi @Martin-Kudláček,

    A simple "sudo reboot" of the PLC has solved the issue and communication works again error free:

    0_1587828876138_421e417e-cd95-4f47-9c5d-16898c910ff4-image.png

    Current serial port is RS485-1.1, I will try if same issue occurs after 10 days if using port RS485-1.3 ...



  • Hi @Martin-Kudláček,

    No issue after 15 days of uptime using port RS485-1.3 :

    0_1589308492689_8fae0eef-f7df-4118-aa5a-82b5300d6a28-image.png

    I switch now back to port RS485-1.1 to check if issue occurs again after 10 days...


  • administrators

    @pascal37 if it occurs again, please send the captured data on the port monitor in txt file from a longer period so we could get better overview



  • Hi @tomas_hora,

    The issue occured again with port RS485-1.1 just after 5 days of uptime, but sorry, I was not capturing serial data before the "crash". This is quite difficult to do: it means leave IDE open in debugging mode during up to 10 days and be able to select all the output (> 50 millions packets) !? Is there any better why to archieve that ?

    Workaround is to redeploy the solution from IDE (cold restart), and serial communication works again immedately.
    The issue is definitvely not on the RS485 bus or Inepro devices.

    I had this issue now 3 times with the port 1.1 on the Axon M525, but not with port 1.3, nor with an Axon M505.
    Since a solution deploy does not reboot the Debian OS of the PLC, the issue should be somewhere in the Mervis runtime !?

    The first time the issue occured, I did a complete reboot of the PLC, but this is not a workaround... as if a counter in the Mervis runtime is not reseted properly like a solution deploy with cold restart.

    Many thanks in advance for your help and kind regards.


  • administrators

    Hello @pascal37, I did not men to capture the data at the time it stops responding, but jus to get a longer period of the caputed data after it stops communicatig to find out if it's all timeout or what is going on the line.

    What should be the configuration of the RS485 (baudrate, parity, ...)

    Could you execute this command in linux while the communication stops?

    cat /run/unipi/io_group1/uart_config
    


  • Hi @tomas_hora,

    Issue occurred again with port RS485-1.1

    This time, a simple reboot (from Debian OS) was enough to regain data acquisition from the 3 Inepro PRO380 meters.

    Port configuration:

    serial_port_config.jpg

    cat /run/unipi/io_group1/uart_config
    9bd
    

    Data captured after communication stops working:

    09:02:36.2939 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:36.3139 <= Receiving: CB EC EB 
    09:02:36.3139 Timeout
    09:02:36.3239 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:36.3439 <= Receiving: CB EC EB 
    09:02:36.3439 Timeout
    09:02:36.3639 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:36.3839 <= Receiving: EC EB 
    09:02:36.3839 Timeout
    09:02:36.3939 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:36.4139 <= Receiving: 6D D5 
    09:02:36.4139 Timeout
    09:02:36.4239 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:36.4539 <= Receiving: 6D D5 
    09:02:36.4539 Timeout
    09:02:36.4739 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:36.4939 <= Receiving: 6D D5 
    09:02:36.4939 Timeout
    09:02:36.5039 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:36.5359 <= Receiving: 68 5B 
    09:02:36.5359 Timeout
    09:02:36.5459 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:36.5659 <= Receiving: 6F 68 5B 
    09:02:36.5659 Timeout
    09:02:36.5859 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:36.6059 <= Receiving: 68 5B 
    09:02:36.6059 Timeout
    09:02:36.6159 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:36.6459 <= Receiving: 3D F3 8F 
    09:02:36.6459 Timeout
    09:02:36.6559 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:36.6759 <= Receiving: F3 8F 
    09:02:36.6759 Timeout
    09:02:36.7009 => Sending: 01 03 50 02 00 06 75 08 
    09:02:36.7209 <= Receiving: F3 8F 
    09:02:36.7209 Timeout
    09:02:36.7309 => Sending: 01 03 50 02 00 06 75 08 
    09:02:36.7599 <= Receiving: 43 
    09:02:36.7599 Timeout
    09:02:36.7699 => Sending: 01 03 50 02 00 06 75 08 
    09:02:36.8399 Timeout
    09:02:36.8599 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:36.8599 <= Receiving: 43 73 00 00 43 
    09:02:36.8599 <= Receiving: 72 99 9A A3 
    09:02:36.8599 Wrong telegram: ???
    09:02:36.8699 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:36.9399 Timeout
    09:02:36.9499 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:36.9699 <= Receiving: 62 23 
    09:02:36.9699 Timeout
    09:02:36.9899 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:37.0099 <= Receiving: 62 23 
    09:02:37.0099 Timeout
    09:02:37.0199 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:37.0519 <= Receiving: 9F 
    09:02:37.0519 Timeout
    09:02:37.0619 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:37.1319 Timeout
    09:02:37.1519 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:37.1719 <= Receiving: 00 5C 
    09:02:37.1719 Timeout
    09:02:37.1819 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:37.2129 <= Receiving: 69 26 
    09:02:37.2129 Timeout
    09:02:37.2229 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:37.2429 <= Receiving: 69 26 
    09:02:37.2429 Timeout
    09:02:37.2629 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:37.2829 <= Receiving: 52 69 26 
    09:02:37.2829 Timeout
    09:02:37.2929 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:37.3129 <= Receiving: EA 1D 
    09:02:37.3129 Timeout
    09:02:37.3229 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:37.3429 <= Receiving: EA 1D 
    09:02:37.3429 Timeout
    09:02:37.3629 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:37.3839 <= Receiving: EA 1D 
    09:02:37.3839 Timeout
    09:02:37.3939 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:37.4139 <= Receiving: 6D D5 
    09:02:37.4139 Timeout
    09:02:37.4239 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:37.4439 <= Receiving: 6D D5 
    09:02:37.4439 Timeout
    09:02:37.4639 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:37.4849 <= Receiving: 6D D5 
    09:02:37.4849 Timeout
    09:02:37.4949 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:37.5159 <= Receiving: 68 5B 
    09:02:37.5159 Timeout
    09:02:37.5259 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:37.5459 <= Receiving: 68 5B 
    09:02:37.5459 Timeout
    09:02:37.5659 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:37.5859 <= Receiving: 68 5B 
    09:02:37.5859 Timeout
    09:02:37.5959 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:37.6279 <= Receiving: F3 8F 
    09:02:37.6279 Timeout
    09:02:37.6628 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:37.6828 <= Receiving: F3 8F 
    09:02:37.6828 Timeout
    09:02:37.7088 => Sending: 01 03 50 02 00 06 75 08 
    09:02:37.7288 <= Receiving: F3 8F 
    09:02:37.7288 Timeout
    09:02:37.7388 => Sending: 01 03 50 02 00 06 75 08 
    09:02:37.7708 <= Receiving: 43 73 
    09:02:37.7708 Timeout
    09:02:37.7808 => Sending: 01 03 50 02 00 06 75 08 
    09:02:37.8508 Timeout
    09:02:37.8708 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:37.8708 <= Receiving: CD 43 73 00 00 
    09:02:37.8708 <= Receiving: 43 72 99 9A 
    09:02:37.8708 Wrong telegram: ???
    09:02:37.8808 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:37.9508 Timeout
    09:02:37.9608 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:37.9808 <= Receiving: 62 23 
    09:02:37.9808 Timeout
    09:02:38.0008 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:38.0218 <= Receiving: 62 23 
    09:02:38.0218 Timeout
    09:02:38.0318 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:38.0628 <= Receiving: 00 5C 
    09:02:38.0628 Timeout
    09:02:38.0728 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:38.0928 <= Receiving: 00 5C 
    09:02:38.0928 Timeout
    09:02:38.1128 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:38.1338 <= Receiving: 00 5C 
    09:02:38.1338 Timeout
    09:02:38.1438 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:38.1638 <= Receiving: 69 26 
    09:02:38.1638 Timeout
    09:02:38.1738 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:38.1938 <= Receiving: 69 26 
    09:02:38.1938 Timeout
    09:02:38.2138 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:38.2338 <= Receiving: 69 26 
    09:02:38.2338 Timeout
    09:02:38.2438 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:38.2638 <= Receiving: EA 1D 
    09:02:38.2638 Timeout
    09:02:38.2738 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:38.2938 <= Receiving: EA 1D 
    09:02:38.2938 Timeout
    09:02:38.3138 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:38.3338 <= Receiving: EA 1D 
    09:02:38.3338 Timeout
    09:02:38.3438 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:38.3638 <= Receiving: 6D D5 
    09:02:38.3638 Timeout
    09:02:38.3738 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:38.4048 <= Receiving: 6D D5 
    09:02:38.4048 Timeout
    09:02:38.4248 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:38.4448 <= Receiving: 6D D5 
    09:02:38.4448 Timeout
    09:02:38.4548 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:38.4748 <= Receiving: 68 5B 
    09:02:38.4748 Timeout
    09:02:38.4848 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:38.5158 <= Receiving: 68 5B 
    09:02:38.5158 Timeout
    09:02:38.5358 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:38.5558 <= Receiving: 68 5B 
    09:02:38.5558 Timeout
    09:02:38.5658 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:38.5858 <= Receiving: F3 8F 
    09:02:38.5858 Timeout
    09:02:38.5958 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:38.6158 <= Receiving: F3 8F 
    09:02:38.6158 Timeout
    09:02:38.6408 => Sending: 01 03 50 02 00 06 75 08 
    09:02:38.6608 <= Receiving: F3 8F 
    09:02:38.6608 Timeout
    09:02:38.6708 => Sending: 01 03 50 02 00 06 75 08 
    09:02:38.7028 <= Receiving: CD 
    09:02:38.7028 Timeout
    09:02:38.7128 => Sending: 01 03 50 02 00 06 75 08 
    09:02:38.7828 Timeout
    09:02:38.8028 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:38.8028 <= Receiving: 43 73 19 9A 43 
    09:02:38.8028 <= Receiving: 72 B3 33 26 
    09:02:38.8028 Wrong telegram: ???
    09:02:38.8128 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:38.8828 Timeout
    09:02:38.8928 => Sending: 01 03 50 12 00 02 75 0E 
    Free space in internal port monitor buffer: 59.7 kB
    09:02:38.9628 Timeout
    09:02:38.9828 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:39.0028 <= Receiving: 1B EE 
    09:02:39.0028 Timeout
    09:02:39.0128 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:39.0448 <= Receiving: 00 5C 
    09:02:39.0448 Timeout
    09:02:39.0548 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:39.0748 <= Receiving: 00 5C 
    09:02:39.0748 Timeout
    09:02:39.0948 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:39.1148 <= Receiving: 00 5C 
    09:02:39.1148 Timeout
    09:02:39.1512 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:39.1712 <= Receiving: 69 26 
    09:02:39.1712 Timeout
    09:02:39.1812 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:39.2012 <= Receiving: 69 26 
    09:02:39.2012 Timeout
    09:02:39.2212 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:39.2412 <= Receiving: 69 26 
    09:02:39.2412 Timeout
    09:02:39.2512 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:39.2712 <= Receiving: EA 1D 
    09:02:39.2712 Timeout
    09:02:39.2812 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:39.3012 <= Receiving: EA 1D 
    09:02:39.3012 Timeout
    09:02:39.3212 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:39.3422 <= Receiving: EA 1D 
    09:02:39.3422 Timeout
    09:02:39.3522 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:39.3722 <= Receiving: 9A 6D D5 
    09:02:39.3722 Timeout
    09:02:39.3822 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:39.4022 <= Receiving: 6D D5 
    09:02:39.4022 Timeout
    09:02:39.4222 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:39.4422 <= Receiving: 6D D5 
    09:02:39.4422 Timeout
    09:02:39.4522 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:39.4722 <= Receiving: 68 5B 
    09:02:39.4722 Timeout
    09:02:39.4822 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:39.5022 <= Receiving: 68 5B 
    09:02:39.5022 Timeout
    09:02:39.5222 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:39.5422 <= Receiving: 68 5B 
    09:02:39.5422 Timeout
    09:02:39.5522 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:39.5722 <= Receiving: F3 8F 
    09:02:39.5722 Timeout
    09:02:39.5822 => Sending: 03 03 60 00 00 02 DB E9 
    09:02:39.6032 <= Receiving: F3 8F 
    09:02:39.6032 Timeout
    09:02:39.6282 => Sending: 01 03 50 02 00 06 75 08 
    09:02:39.6482 <= Receiving: F3 8F 
    09:02:39.6482 Timeout
    09:02:39.6582 => Sending: 01 03 50 02 00 06 75 08 
    09:02:39.6892 <= Receiving: CD 
    09:02:39.6892 Timeout
    09:02:39.6992 => Sending: 01 03 50 02 00 06 75 08 
    09:02:39.7692 Timeout
    09:02:39.7892 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:39.7892 <= Receiving: 43 73 00 00 43 
    09:02:39.7892 <= Receiving: 72 99 9A A3 
    09:02:39.7892 Wrong telegram: ???
    09:02:39.7992 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:39.8692 Timeout
    09:02:39.8792 => Sending: 01 03 50 12 00 02 75 0E 
    09:02:39.9492 Timeout
    09:02:39.9692 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:39.9892 <= Receiving: 5C 1B EE 
    09:02:39.9892 Timeout
    09:02:39.9992 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:40.0292 <= Receiving: 3D 
    09:02:40.0292 Timeout
    09:02:40.0392 => Sending: 01 03 60 0C 00 02 1A 08 
    09:02:40.1092 Timeout
    09:02:40.1292 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:40.1492 <= Receiving: 00 5C 
    09:02:40.1492 Timeout
    09:02:40.1592 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:40.1792 <= Receiving: 69 26 
    09:02:40.1792 Timeout
    09:02:40.1892 => Sending: 01 03 60 18 00 02 5A 0C 
    09:02:40.2212 <= Receiving: 69 26 
    09:02:40.2212 Timeout
    09:02:40.2412 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:40.2612 <= Receiving: 52 69 26 
    09:02:40.2612 Timeout
    09:02:40.2712 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:40.2912 <= Receiving: DD 3E 
    09:02:40.2912 Timeout
    09:02:40.3012 => Sending: 02 03 50 12 00 02 75 3D 
    09:02:40.3212 <= Receiving: DD 3E 
    09:02:40.3212 Timeout
    09:02:40.3412 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:40.3612 <= Receiving: DD 3E 
    09:02:40.3612 Timeout
    09:02:40.3712 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:40.4032 <= Receiving: 6D D5 
    09:02:40.4032 Timeout
    09:02:40.4132 => Sending: 02 03 60 00 00 02 DA 38 
    09:02:40.4332 <= Receiving: 6D D5 
    09:02:40.4332 Timeout
    09:02:40.4532 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:40.4732 <= Receiving: 6D D5 
    09:02:40.4732 Timeout
    09:02:40.4832 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:40.5032 <= Receiving: 68 5B 
    09:02:40.5032 Timeout
    09:02:40.5132 => Sending: 03 03 50 12 00 02 74 EC 
    09:02:40.5332 <= Receiving: 68 5B 
    09:02:40.5332 Timeout
    

    Many thanks in advance for your help and kind regards


  • administrators

    @pascal37 said in M525 stopped working:

    cat /run/unipi/io_group1/uart_config

    Did you run this command when you received the Timeout or afer you restarted it?



  • The command was run while serial communication timeouts, but output is the same after a PLC restart.


  • administrators

    This post is deleted!

  • administrators

    @pascal37 Can you try increasing the Max telegram duration a bit higher? For example 300ms? If that won't help. You can also try to increase the pause between telegrams to 5ms.



  • Hi @tomas_hora,

    I have increased Max telegram duration to 300ms + added a 5ms pause between telegrams.

    I have also update my M525 to latest Axon runtime (2.3.0.8) with latest firmwares/drivers.

    I will let you know if issue occurs again.
    Many thanks and kind regards.



  • Hi @tomas_hora, issue occured again again with port 1.1 and new settings (Max telegram duration to 300ms + added a 5ms pause between telegrams.)
    A simple PLC reboot (from Debian OS) was enough to regain data acquisition from the 3 Inepro PRO380 meters.

    I do not understand why problem never occured when using port 1.3 of same PLC, nor with my previous M505.
    What is the difference between ports 1.1 and 1.3 ?
    If same hardware, could it be a hardware issue with port 1.1 on my M525?

    22:13:08.5230 => Send: 03 03 60 00 00 02 DB E9 
    22:13:08.5530 <= Recv: CA CB 
    22:13:08.5530 Timeout
    22:13:08.5630 => Send: 03 03 60 00 00 02 DB E9 
    22:13:08.5930 <= Recv: CA CB 
    22:13:08.5930 Timeout
    22:13:08.6230 => Send: 01 03 50 02 00 06 75 08 
    22:13:08.6530 <= Recv: CA CB 
    22:13:08.6530 Timeout
    22:13:08.6630 => Send: 01 03 50 02 00 06 75 08 
    22:13:08.6630 <= Recv: 43 71 CC CD 43 
    22:13:08.7040 <= Recv: 72 E6 66 
    22:13:08.7040 Wrong telegram: ???
    22:13:08.7140 => Send: 01 03 50 02 00 06 75 08 
    22:13:09.0140 Timeout
    22:13:09.0390 => Send: 01 03 50 12 00 02 75 0E 
    22:13:09.0390 <= Recv: 43 71 CC CD 43 
    22:13:09.0390 <= Recv: 72 E6 66 52 
    22:13:09.0390 Wrong telegram: ???
    22:13:09.0490 => Send: 01 03 50 12 00 02 75 0E 
    22:13:09.3490 Timeout
    22:13:09.3590 => Send: 01 03 50 12 00 02 75 0E 
    22:13:09.3890 <= Recv: 4B E3 
    22:13:09.3890 Timeout
    22:13:09.4140 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:09.4440 <= Recv: EC 4B E3 
    22:13:09.4440 Timeout
    22:13:09.4540 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:09.4840 <= Recv: CC D8 
    22:13:09.4840 Timeout
    22:13:09.4940 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:09.5240 <= Recv: CC D8 
    22:13:09.5240 Timeout
    22:13:09.5490 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:09.5790 <= Recv: CC D8 
    22:13:09.5790 Timeout
    22:13:09.5890 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:09.6190 <= Recv: 72 00 
    22:13:09.6190 Timeout
    22:13:09.6290 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:09.6590 <= Recv: 72 00 
    22:13:09.6590 Timeout
    22:13:09.6840 => Send: 02 03 50 12 00 02 75 3D 
    22:13:09.7140 <= Recv: 72 00 
    22:13:09.7140 Timeout
    22:13:09.7240 => Send: 02 03 50 12 00 02 75 3D 
    22:13:09.7540 <= Recv: 46 88 
    22:13:09.7540 Timeout
    22:13:09.7640 => Send: 02 03 50 12 00 02 75 3D 
    22:13:09.7940 <= Recv: 46 88 
    22:13:09.7940 Timeout
    22:13:09.8190 => Send: 02 03 60 00 00 02 DA 38 
    22:13:09.8490 <= Recv: 46 88 
    22:13:09.8490 Timeout
    22:13:09.8590 => Send: 02 03 60 00 00 02 DA 38 
    22:13:09.8890 <= Recv: 85 9B 
    22:13:09.8890 Timeout
    22:13:09.8990 => Send: 02 03 60 00 00 02 DA 38 
    22:13:09.9290 <= Recv: 85 9B 
    22:13:09.9290 Timeout
    22:13:09.9540 => Send: 03 03 50 12 00 02 74 EC 
    22:13:09.9840 <= Recv: 85 9B 
    22:13:09.9840 Timeout
    22:13:09.9940 => Send: 03 03 50 12 00 02 74 EC 
    22:13:10.0240 <= Recv: 68 5B 
    22:13:10.0240 Timeout
    22:13:10.0340 => Send: 03 03 50 12 00 02 74 EC 
    22:13:10.0640 <= Recv: 68 5B 
    22:13:10.0640 Timeout
    22:13:10.0890 => Send: 03 03 60 00 00 02 DB E9 
    22:13:10.1190 <= Recv: 68 5B 
    22:13:10.1190 Timeout
    22:13:10.1290 => Send: 03 03 60 00 00 02 DB E9 
    22:13:10.1700 <= Recv: 04 
    22:13:10.1700 Timeout
    22:13:10.1800 => Send: 03 03 60 00 00 02 DB E9 
    22:13:10.4800 Timeout
    22:13:10.5100 => Send: 01 03 50 02 00 06 75 08 
    22:13:10.5623 <= Recv: CA CB 
    22:13:10.5623 Timeout
    22:13:10.5723 => Send: 01 03 50 02 00 06 75 08 
    22:13:10.5723 <= Recv: 43 71 CC CD 43 
    22:13:10.6133 <= Recv: 72 E6 66 52 
    22:13:10.6133 Wrong telegram: ???
    22:13:10.6233 => Send: 01 03 50 02 00 06 75 08 
    22:13:10.9233 Timeout
    22:13:10.9483 => Send: 01 03 50 12 00 02 75 0E 
    22:13:10.9483 <= Recv: 43 71 CC CD 43 
    22:13:10.9483 <= Recv: 72 E6 66 52 
    22:13:10.9483 Wrong telegram: ???
    22:13:10.9583 => Send: 01 03 50 12 00 02 75 0E 
    22:13:11.2583 Timeout
    22:13:11.2683 => Send: 01 03 50 12 00 02 75 0E 
    22:13:11.2993 <= Recv: 4B E3 
    22:13:11.2993 Timeout
    22:13:11.3243 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:11.3543 <= Recv: 4B E3 
    22:13:11.3543 Timeout
    22:13:11.3643 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:11.3943 <= Recv: CC D8 
    22:13:11.3943 Timeout
    22:13:11.4043 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:11.4343 <= Recv: CC D8 
    22:13:11.4343 Timeout
    22:13:11.4593 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:11.4893 <= Recv: CC D8 
    22:13:11.4893 Timeout
    22:13:11.4993 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:11.5293 <= Recv: 72 00 
    22:13:11.5293 Timeout
    22:13:11.5393 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:11.5693 <= Recv: E1 72 00 
    22:13:11.5693 Timeout
    22:13:11.5943 => Send: 02 03 50 12 00 02 75 3D 
    22:13:11.6243 <= Recv: 72 00 
    22:13:11.6243 Timeout
    22:13:11.6343 => Send: 02 03 50 12 00 02 75 3D 
    22:13:11.6643 <= Recv: 46 88 
    22:13:11.6643 Timeout
    22:13:11.6743 => Send: 02 03 50 12 00 02 75 3D 
    22:13:11.7043 <= Recv: 46 88 
    22:13:11.7043 Timeout
    22:13:11.7293 => Send: 02 03 60 00 00 02 DA 38 
    22:13:11.7593 <= Recv: 46 88 
    22:13:11.7593 Timeout
    22:13:11.7693 => Send: 02 03 60 00 00 02 DA 38 
    22:13:11.7993 <= Recv: 85 9B 
    22:13:11.7993 Timeout
    22:13:11.8093 => Send: 02 03 60 00 00 02 DA 38 
    22:13:11.8393 <= Recv: 85 9B 
    22:13:11.8393 Timeout
    22:13:11.8643 => Send: 03 03 50 12 00 02 74 EC 
    22:13:11.8943 <= Recv: 85 9B 
    22:13:11.8943 Timeout
    22:13:11.9043 => Send: 03 03 50 12 00 02 74 EC 
    22:13:11.9343 <= Recv: 68 5B 
    22:13:11.9343 Timeout
    22:13:11.9443 => Send: 03 03 50 12 00 02 74 EC 
    22:13:11.9743 <= Recv: 68 5B 
    22:13:11.9743 Timeout
    22:13:11.9993 => Send: 03 03 60 00 00 02 DB E9 
    22:13:12.0293 <= Recv: 68 5B 
    22:13:12.0293 Timeout
    22:13:12.0393 => Send: 03 03 60 00 00 02 DB E9 
    22:13:12.0693 <= Recv: CA CB 
    22:13:12.0693 Timeout
    22:13:12.0793 => Send: 03 03 60 00 00 02 DB E9 
    22:13:12.1093 <= Recv: CA CB 
    22:13:12.1093 Timeout
    22:13:12.1393 => Send: 01 03 50 02 00 06 75 08 
    22:13:12.1693 <= Recv: CA CB 
    22:13:12.1693 Timeout
    22:13:12.1793 => Send: 01 03 50 02 00 06 75 08 
    22:13:12.1793 <= Recv: 43 71 B3 33 43 
    22:13:12.2193 <= Recv: 72 E6 66 70 
    22:13:12.2193 Wrong telegram: ???
    22:13:12.2293 => Send: 01 03 50 02 00 06 75 08 
    22:13:12.5293 Timeout
    22:13:12.5543 => Send: 01 03 50 12 00 02 75 0E 
    22:13:12.5543 <= Recv: 43 71 B3 33 43 
    22:13:12.5543 <= Recv: 72 E6 66 70 
    22:13:12.5543 Wrong telegram: ???
    22:13:12.5643 => Send: 01 03 50 12 00 02 75 0E 
    22:13:12.8643 Timeout
    22:13:12.8743 => Send: 01 03 50 12 00 02 75 0E 
    22:13:12.9043 <= Recv: 4B E3 
    22:13:12.9043 Timeout
    22:13:12.9293 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:12.9593 <= Recv: 4B E3 
    22:13:12.9593 Timeout
    22:13:12.9693 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:12.9993 <= Recv: CC D8 
    22:13:12.9993 Timeout
    22:13:13.0093 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:13.0393 <= Recv: CC D8 
    22:13:13.0393 Timeout
    22:13:13.0643 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:13.0943 <= Recv: CC D8 
    22:13:13.0943 Timeout
    22:13:13.1043 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:13.1557 <= Recv: 72 00 
    22:13:13.1557 Timeout
    22:13:13.1657 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:13.1957 <= Recv: 72 00 
    22:13:13.1957 Timeout
    22:13:13.2207 => Send: 02 03 50 12 00 02 75 3D 
    22:13:13.2507 <= Recv: 72 00 
    22:13:13.2507 Timeout
    22:13:13.2607 => Send: 02 03 50 12 00 02 75 3D 
    22:13:13.2907 <= Recv: 46 88 
    22:13:13.2907 Timeout
    22:13:13.3007 => Send: 02 03 50 12 00 02 75 3D 
    22:13:13.3307 <= Recv: 46 88 
    22:13:13.3307 Timeout
    22:13:13.3557 => Send: 02 03 60 00 00 02 DA 38 
    22:13:13.3857 <= Recv: 46 88 
    22:13:13.3857 Timeout
    22:13:13.3957 => Send: 02 03 60 00 00 02 DA 38 
    22:13:13.4257 <= Recv: 85 9B 
    22:13:13.4257 Timeout
    22:13:13.4357 => Send: 02 03 60 00 00 02 DA 38 
    22:13:13.4657 <= Recv: 85 9B 
    22:13:13.4657 Timeout
    22:13:13.4907 => Send: 03 03 50 12 00 02 74 EC 
    22:13:13.5207 <= Recv: 85 9B 
    22:13:13.5207 Timeout
    22:13:13.5307 => Send: 03 03 50 12 00 02 74 EC 
    22:13:13.5607 <= Recv: 68 5B 
    22:13:13.5607 Timeout
    22:13:13.5707 => Send: 03 03 50 12 00 02 74 EC 
    22:13:13.6007 <= Recv: 68 5B 
    22:13:13.6007 Timeout
    22:13:13.6257 => Send: 03 03 60 00 00 02 DB E9 
    22:13:13.6557 <= Recv: 68 5B 
    22:13:13.6557 Timeout
    22:13:13.6657 => Send: 03 03 60 00 00 02 DB E9 
    22:13:13.6957 <= Recv: CA CB 
    22:13:13.6957 Timeout
    22:13:13.7057 => Send: 03 03 60 00 00 02 DB E9 
    22:13:13.7357 <= Recv: 0A CA CB 
    22:13:13.7357 Timeout
    22:13:13.7657 => Send: 01 03 50 02 00 06 75 08 
    22:13:13.7957 <= Recv: CA CB 
    22:13:13.7957 Timeout
    22:13:13.8057 => Send: 01 03 50 02 00 06 75 08 
    22:13:13.8057 <= Recv: 43 71 CC CD 43 
    22:13:13.8447 <= Recv: 72 CC CD BF 07 
    22:13:13.8447 Wrong telegram: ???
    22:13:13.8547 => Send: 01 03 50 02 00 06 75 08 
    22:13:13.8547 <= Recv: 43 71 CC CD 43 
    22:13:13.8847 <= Recv: 72 CC CD BF 07 
    22:13:13.8847 Wrong telegram: ???
    22:13:13.9097 => Send: 01 03 50 12 00 02 75 0E 
    22:13:13.9097 <= Recv: 43 71 CC CD 43 
    22:13:13.9097 <= Recv: 72 CC CD BF 
    22:13:13.9097 Wrong telegram: ???
    22:13:13.9197 => Send: 01 03 50 12 00 02 75 0E 
    22:13:14.2197 Timeout
    22:13:14.2297 => Send: 01 03 50 12 00 02 75 0E 
    22:13:14.2597 <= Recv: 4B E3 
    22:13:14.2597 Timeout
    22:13:14.2847 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:14.3147 <= Recv: 4B E3 
    22:13:14.3147 Timeout
    22:13:14.3247 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:14.3547 <= Recv: CC D8 
    22:13:14.3547 Timeout
    22:13:14.3647 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:14.3947 <= Recv: CC D8 
    22:13:14.3947 Timeout
    22:13:14.4197 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:14.4497 <= Recv: CC D8 
    22:13:14.4497 Timeout
    22:13:14.4597 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:14.4897 <= Recv: 72 00 
    22:13:14.4897 Timeout
    22:13:14.4997 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:14.5297 <= Recv: 72 00 
    22:13:14.5297 Timeout
    22:13:14.5547 => Send: 02 03 50 12 00 02 75 3D 
    22:13:14.5847 <= Recv: 72 00 
    22:13:14.5847 Timeout
    22:13:14.5947 => Send: 02 03 50 12 00 02 75 3D 
    22:13:14.6247 <= Recv: 46 88 
    22:13:14.6247 Timeout
    22:13:14.6347 => Send: 02 03 50 12 00 02 75 3D 
    22:13:14.6647 <= Recv: 46 88 
    22:13:14.6647 Timeout
    22:13:14.6897 => Send: 02 03 60 00 00 02 DA 38 
    22:13:14.7197 <= Recv: 46 88 
    22:13:14.7197 Timeout
    22:13:14.7297 => Send: 02 03 60 00 00 02 DA 38 
    22:13:14.7597 <= Recv: 85 9B 
    22:13:14.7597 Timeout
    22:13:14.7697 => Send: 02 03 60 00 00 02 DA 38 
    22:13:14.7997 <= Recv: 85 9B 
    22:13:14.7997 Timeout
    22:13:14.8247 => Send: 03 03 50 12 00 02 74 EC 
    22:13:14.8547 <= Recv: 85 9B 
    22:13:14.8547 Timeout
    22:13:14.8647 => Send: 03 03 50 12 00 02 74 EC 
    22:13:14.8947 <= Recv: 68 5B 
    22:13:14.9166 Timeout
    22:13:14.9266 => Send: 03 03 50 12 00 02 74 EC 
    22:13:14.9566 <= Recv: 68 5B 
    22:13:14.9566 Timeout
    22:13:14.9816 => Send: 03 03 60 00 00 02 DB E9 
    22:13:15.0116 <= Recv: 68 5B 
    22:13:15.0116 Timeout
    22:13:15.0216 => Send: 03 03 60 00 00 02 DB E9 
    22:13:15.0516 <= Recv: CA CB 
    22:13:15.0516 Timeout
    22:13:15.0616 => Send: 03 03 60 00 00 02 DB E9 
    22:13:15.0916 <= Recv: CA CB 
    22:13:15.0916 Timeout
    22:13:15.1216 => Send: 01 03 50 02 00 06 75 08 
    22:13:15.1516 <= Recv: CA CB 
    22:13:15.1516 Timeout
    22:13:15.1616 => Send: 01 03 50 02 00 06 75 08 
    22:13:15.1616 <= Recv: 43 71 B3 33 43 
    22:13:15.2016 <= Recv: 72 E6 66 C3 C3 
    22:13:15.2016 Wrong telegram: ???
    22:13:15.2116 => Send: 01 03 50 02 00 06 75 08 
    22:13:15.2116 <= Recv: 43 71 B3 33 43 
    22:13:15.2416 <= Recv: 72 E6 66 C3 C3 
    22:13:15.2416 Wrong telegram: ???
    22:13:15.2666 => Send: 01 03 50 12 00 02 75 0E 
    22:13:15.2666 <= Recv: 43 71 B3 33 43 
    22:13:15.2666 <= Recv: 72 E6 66 C3 
    22:13:15.2666 Wrong telegram: ???
    22:13:15.2766 => Send: 01 03 50 12 00 02 75 0E 
    22:13:15.5766 Timeout
    22:13:15.5866 => Send: 01 03 50 12 00 02 75 0E 
    22:13:15.6166 <= Recv: 4B E3 
    22:13:15.6166 Timeout
    22:13:15.6416 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:15.6716 <= Recv: 4B E3 
    22:13:15.6716 Timeout
    22:13:15.6816 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:15.7116 <= Recv: CC D8 
    22:13:15.7116 Timeout
    22:13:15.7216 => Send: 01 03 60 0C 00 02 1A 08 
    22:13:15.7516 <= Recv: CC D8 
    22:13:15.7516 Timeout
    22:13:15.7766 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:15.8066 <= Recv: CC D8 
    22:13:15.8066 Timeout
    22:13:15.8166 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:15.8466 <= Recv: 72 00 
    22:13:15.8466 Timeout
    22:13:15.8566 => Send: 01 03 60 18 00 02 5A 0C 
    22:13:15.8866 <= Recv: 72 00 
    22:13:15.8866 Timeout
    22:13:15.9116 => Send: 02 03 50 12 00 02 75 3D 
    22:13:15.9416 <= Recv: 72 00 
    22:13:15.9416 Timeout
    22:13:15.9516 => Send: 02 03 50 12 00 02 75 3D 
    22:13:15.9926 <= Recv: 04 
    22:13:15.9926 Timeout
    22:13:16.0026 => Send: 02 03 50 12 00 02 75 3D 
    22:13:16.3026 Timeout
    22:13:16.3276 => Send: 02 03 60 00 00 02 DA 38 
    22:13:16.3576 <= Recv: 46 88 
    22:13:16.3576 Timeout
    22:13:16.3676 => Send: 02 03 60 00 00 02 DA 38 
    22:13:16.3976 <= Recv: 85 9B 
    22:13:16.3976 Timeout
    22:13:16.4076 => Send: 02 03 60 00 00 02 DA 38 
    22:13:16.4376 <= Recv: 85 9B 
    22:13:16.4376 Timeout
    22:13:16.4626 => Send: 03 03 50 12 00 02 74 EC 
    22:13:16.4926 <= Recv: 85 9B 
    22:13:16.4926 Timeout
    22:13:16.5026 => Send: 03 03 50 12 00 02 74 EC 
    22:13:16.5326 <= Recv: 68 5B 
    22:13:16.5326 Timeout
    22:13:16.5426 => Send: 03 03 50 12 00 02 74 EC 
    22:13:16.5726 <= Recv: 68 5B 
    22:13:16.5726 Timeout
    22:13:16.5976 => Send: 03 03 60 00 00 02 DB E9 
    22:13:16.6276 <= Recv: 68 5B 
    22:13:16.6276 Timeout
    22:13:16.6376 => Send: 03 03 60 00 00 02 DB E9 
    22:13:16.6676 <= Recv: CA CB 
    22:13:16.6676 Timeout
    22:13:16.6776 => Send: 03 03 60 00 00 02 DB E9 
    22:13:16.7076 <= Recv: CA CB 
    22:13:16.7076 Timeout
    22:13:16.7376 => Send: 01 03 50 02 00 06 75 08 
    22:13:16.7686 <= Recv: CA CB 
    22:13:16.7686 Timeout
    

  • administrators

    Hello @pascal37,

    thank you for the information sent. Unfortunately, based on it, we are not able to determine the exact source of the issue.
    Can you please check the following:

    1. Again, make sure you have the latest firmware in the I/O board flashed. Type:
    /opt/unipi/tools/fwspi -i 0
    
    1. Can you confirm that the i2c-0 error in the dmesg log appears always in the moment of communication goes wrong? It seems there is a correlation (863475 seconds is almost exactly 10 days).

    2. Are you sure that no other running processes access the interface (/dev/ttyNS0)? Please check also your time-scheduled tasks (e.g. executed from cron).

    3. Are you sure that no noise or interference from the environment or power supply can infiltrate the unit?

    As the last resort, we are able to simulate your application with the same HW, but it will be pretty time-consuming. In the meantime, please use another RS485 interface.



  • Hi @martin_triska,

    1. I did a reflash (USB method) of my M525 with latest Mervis OS image (2.3.0.10), firmwares are already up-to-date:
    unipi@M525-sn8:~$ sudo /opt/unipi/tools/fwspi -i 0
    Boardset:     0 B-1000                         (v1.0)
    Baseboard:    0 B-1000                         (v2.0)
    Firmware: v5.50
    unipi@M525-sn8:~$ sudo /opt/unipi/tools/fwspi -i 1
    Boardset:    19 E-4Ai4Ao_U-R485Di4Ro5          (v1.0)
    Baseboard:   11 E-4Ai4Ao                       (v1.0)
    Firmware: v5.50
    
    1. From what I've seen after serial communications "crashes", I did not see anymore an i2c-0 error in demsg, so I am not sure this is correlated.

    2. Yes: it is a standard Mervis OS image + 2 pythons scripts running as daemons with pymodbus client over ModbusTCP, so no serial communications there.

    3. Yes: I am using shielded cable specific for RS485, the bus is properly terminated at each end. Cable shield is properly grounded at PLC side and also to PLC itself. Environment is stable and, so long the issue is not occurring, communication is working perfectly.
      With the increase of Max telegram duration to 300ms, there is absolutely no error during multiple days:

    PLC_status.JPG

    It seems most likely a driver or Mervis runtime issue related to M525 model and port 1.1 !?
    As I said, I never had the issue when using port 1.3 instead of port 1.1
    I have also another RS485 bus on port 2.1 and no issue either.
    This issue also never occurred with port 1.1 on my previous M505 PLC

    If you want to try reproduce the issue, you need 3 Inepro PRO380 meters and read constantly multiple values from them like power, energy, voltage.
    In my case, the maximum uptime before a crash is about 10 days.

    I can use port 1.3 in the meantime, but I will need the 3rd RS485 port later for my project...
    But let's check again if it is really stable with port 1.3 over >10 days !

    Many thanks in advance and kind regards


Log in to reply