M525 stopped working

[pascal37]

Dear Unipi Support Team,

After 10 days of smoth operation, my new M525 PLC has stopped working and I have the following error with dmesg:

[ 1696.881917] EXT4-fs (mmcblk0p2): re-mounted. Opts: errors=remount-ro,commit=120,discard
[ 1696.942839] EXT4-fs (mmcblk0p2): re-mounted. Opts: errors=remount-ro,commit=120,discard
[ 1697.239529] RTL8211E Gigabit Ethernet stmmac-0:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
[ 1697.242547] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available
[ 1697.242559] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[ 1697.242990] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1701.336275] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 1701.336369] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[12755.149390] hrtimer: interrupt took 656215 ns
[863475.515873] i2c i2c-0: mv64xxx_i2c_fsm: Ctlr Error -- state: 0x2, status: 0x0, addr: 0x18, flags: 0x1

After PLC reboot, it still did not work properly, especially modbus communication trought
serial port RS485-1.1 with 3 energy meters (Inepro PRO380)

I did a complete reflash of the Mervis image and redeployed my projet and now it works again.
Do you have an idea what happened?

Many thanks in advance for your help and kind regards

[Martin Kudláček]

Hello @pascal37,
can you please described how it stopped working? What software do you use?

Thank you and have a nice day,
Martin

[pascal37]

I am running latest available Mervis version (2.2.0.30). After 10 days of operation, the modbus communication over serial port RS485-1.1 failed. I am reading some values from 3 Inepro power meters.

[pascal37]

Hi @Martin-Kudláček,

I found article: https://kb.unipi.technology/en:sw:04-unipi-firmware?s[]=firmware

I had firmware v5.40 and just updated now to v5.50.

[Martin Kudláček]

Hello @pascal37,
all of that is ok. You can check what is going on on the serial port in Mervis IDE directly. Switch to debugging mode and in the bottom panel look for "port monitor". Start the monitoring of the affected port. You should see more information.

Also, try disconnecting the meters to see if the problem is caused by one of them. And also make sure, that you use shielded twist pair cables, with the shielding connected to earth ground on one end and use proper termination of the bus.

Best regards,
Martin

[pascal37]

Hi @martin-kudláček,

The problem recurred, again after 10 days of smooth operation.

Here is the output of the serial port:

Since the solution works fine during multiple days without any error on the serial interface, I do not think there is an issue on the RS485 bus. Yes, I am using am appropriate shielded cable with terminations on both ends.

I recently replaced the M505 with an M525 because of relay issues but I never had this RS485 serial bus issue with the M505 !?

There are 3 Inepro PRO380 devices on the bus, I tried to disconnect each Meter, one after the other, but the issue seems not caused by one of them. I have also connected each meter individually direcly to the M525 with another cable but no success either.

Anything else we can do on the M525 to further troublehoot before I try restart/redeploy the solution ?

Many thanks in advance and kind regards,

Pascal

[pascal37]

Hi @Martin-Kudláček,

A simple "sudo reboot" of the PLC has solved the issue and communication works again error free:

Current serial port is RS485-1.1, I will try if same issue occurs after 10 days if using port RS485-1.3 ...

[pascal37]

Hi @Martin-Kudláček,

No issue after 15 days of uptime using port RS485-1.3 :

I switch now back to port RS485-1.1 to check if issue occurs again after 10 days...

[tomas_hora]

@pascal37 if it occurs again, please send the captured data on the port monitor in txt file from a longer period so we could get better overview

[pascal37]

Hi @tomas_hora,

The issue occured again with port RS485-1.1 just after 5 days of uptime, but sorry, I was not capturing serial data before the "crash". This is quite difficult to do: it means leave IDE open in debugging mode during up to 10 days and be able to select all the output (> 50 millions packets) !? Is there any better why to archieve that ?

Workaround is to redeploy the solution from IDE (cold restart), and serial communication works again immedately.
The issue is definitvely not on the RS485 bus or Inepro devices.

I had this issue now 3 times with the port 1.1 on the Axon M525, but not with port 1.3, nor with an Axon M505.
Since a solution deploy does not reboot the Debian OS of the PLC, the issue should be somewhere in the Mervis runtime !?

The first time the issue occured, I did a complete reboot of the PLC, but this is not a workaround... as if a counter in the Mervis runtime is not reseted properly like a solution deploy with cold restart.

Many thanks in advance for your help and kind regards.

[tomas_hora]

Hello @pascal37, I did not men to capture the data at the time it stops responding, but jus to get a longer period of the caputed data after it stops communicatig to find out if it's all timeout or what is going on the line.

What should be the configuration of the RS485 (baudrate, parity, ...)

Could you execute this command in linux while the communication stops?

cat /run/unipi/io_group1/uart_config

[pascal37]

Hi @tomas_hora,

Issue occurred again with port RS485-1.1

This time, a simple reboot (from Debian OS) was enough to regain data acquisition from the 3 Inepro PRO380 meters.

Port configuration:

cat /run/unipi/io_group1/uart_config
9bd

Data captured after communication stops working:

09:02:36.2939 => Sending: 02 03 50 12 00 02 75 3D 
09:02:36.3139 <= Receiving: CB EC EB 
09:02:36.3139 Timeout
09:02:36.3239 => Sending: 02 03 50 12 00 02 75 3D 
09:02:36.3439 <= Receiving: CB EC EB 
09:02:36.3439 Timeout
09:02:36.3639 => Sending: 02 03 60 00 00 02 DA 38 
09:02:36.3839 <= Receiving: EC EB 
09:02:36.3839 Timeout
09:02:36.3939 => Sending: 02 03 60 00 00 02 DA 38 
09:02:36.4139 <= Receiving: 6D D5 
09:02:36.4139 Timeout
09:02:36.4239 => Sending: 02 03 60 00 00 02 DA 38 
09:02:36.4539 <= Receiving: 6D D5 
09:02:36.4539 Timeout
09:02:36.4739 => Sending: 03 03 50 12 00 02 74 EC 
09:02:36.4939 <= Receiving: 6D D5 
09:02:36.4939 Timeout
09:02:36.5039 => Sending: 03 03 50 12 00 02 74 EC 
09:02:36.5359 <= Receiving: 68 5B 
09:02:36.5359 Timeout
09:02:36.5459 => Sending: 03 03 50 12 00 02 74 EC 
09:02:36.5659 <= Receiving: 6F 68 5B 
09:02:36.5659 Timeout
09:02:36.5859 => Sending: 03 03 60 00 00 02 DB E9 
09:02:36.6059 <= Receiving: 68 5B 
09:02:36.6059 Timeout
09:02:36.6159 => Sending: 03 03 60 00 00 02 DB E9 
09:02:36.6459 <= Receiving: 3D F3 8F 
09:02:36.6459 Timeout
09:02:36.6559 => Sending: 03 03 60 00 00 02 DB E9 
09:02:36.6759 <= Receiving: F3 8F 
09:02:36.6759 Timeout
09:02:36.7009 => Sending: 01 03 50 02 00 06 75 08 
09:02:36.7209 <= Receiving: F3 8F 
09:02:36.7209 Timeout
09:02:36.7309 => Sending: 01 03 50 02 00 06 75 08 
09:02:36.7599 <= Receiving: 43 
09:02:36.7599 Timeout
09:02:36.7699 => Sending: 01 03 50 02 00 06 75 08 
09:02:36.8399 Timeout
09:02:36.8599 => Sending: 01 03 50 12 00 02 75 0E 
09:02:36.8599 <= Receiving: 43 73 00 00 43 
09:02:36.8599 <= Receiving: 72 99 9A A3 
09:02:36.8599 Wrong telegram: ???
09:02:36.8699 => Sending: 01 03 50 12 00 02 75 0E 
09:02:36.9399 Timeout
09:02:36.9499 => Sending: 01 03 50 12 00 02 75 0E 
09:02:36.9699 <= Receiving: 62 23 
09:02:36.9699 Timeout
09:02:36.9899 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:37.0099 <= Receiving: 62 23 
09:02:37.0099 Timeout
09:02:37.0199 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:37.0519 <= Receiving: 9F 
09:02:37.0519 Timeout
09:02:37.0619 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:37.1319 Timeout
09:02:37.1519 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:37.1719 <= Receiving: 00 5C 
09:02:37.1719 Timeout
09:02:37.1819 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:37.2129 <= Receiving: 69 26 
09:02:37.2129 Timeout
09:02:37.2229 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:37.2429 <= Receiving: 69 26 
09:02:37.2429 Timeout
09:02:37.2629 => Sending: 02 03 50 12 00 02 75 3D 
09:02:37.2829 <= Receiving: 52 69 26 
09:02:37.2829 Timeout
09:02:37.2929 => Sending: 02 03 50 12 00 02 75 3D 
09:02:37.3129 <= Receiving: EA 1D 
09:02:37.3129 Timeout
09:02:37.3229 => Sending: 02 03 50 12 00 02 75 3D 
09:02:37.3429 <= Receiving: EA 1D 
09:02:37.3429 Timeout
09:02:37.3629 => Sending: 02 03 60 00 00 02 DA 38 
09:02:37.3839 <= Receiving: EA 1D 
09:02:37.3839 Timeout
09:02:37.3939 => Sending: 02 03 60 00 00 02 DA 38 
09:02:37.4139 <= Receiving: 6D D5 
09:02:37.4139 Timeout
09:02:37.4239 => Sending: 02 03 60 00 00 02 DA 38 
09:02:37.4439 <= Receiving: 6D D5 
09:02:37.4439 Timeout
09:02:37.4639 => Sending: 03 03 50 12 00 02 74 EC 
09:02:37.4849 <= Receiving: 6D D5 
09:02:37.4849 Timeout
09:02:37.4949 => Sending: 03 03 50 12 00 02 74 EC 
09:02:37.5159 <= Receiving: 68 5B 
09:02:37.5159 Timeout
09:02:37.5259 => Sending: 03 03 50 12 00 02 74 EC 
09:02:37.5459 <= Receiving: 68 5B 
09:02:37.5459 Timeout
09:02:37.5659 => Sending: 03 03 60 00 00 02 DB E9 
09:02:37.5859 <= Receiving: 68 5B 
09:02:37.5859 Timeout
09:02:37.5959 => Sending: 03 03 60 00 00 02 DB E9 
09:02:37.6279 <= Receiving: F3 8F 
09:02:37.6279 Timeout
09:02:37.6628 => Sending: 03 03 60 00 00 02 DB E9 
09:02:37.6828 <= Receiving: F3 8F 
09:02:37.6828 Timeout
09:02:37.7088 => Sending: 01 03 50 02 00 06 75 08 
09:02:37.7288 <= Receiving: F3 8F 
09:02:37.7288 Timeout
09:02:37.7388 => Sending: 01 03 50 02 00 06 75 08 
09:02:37.7708 <= Receiving: 43 73 
09:02:37.7708 Timeout
09:02:37.7808 => Sending: 01 03 50 02 00 06 75 08 
09:02:37.8508 Timeout
09:02:37.8708 => Sending: 01 03 50 12 00 02 75 0E 
09:02:37.8708 <= Receiving: CD 43 73 00 00 
09:02:37.8708 <= Receiving: 43 72 99 9A 
09:02:37.8708 Wrong telegram: ???
09:02:37.8808 => Sending: 01 03 50 12 00 02 75 0E 
09:02:37.9508 Timeout
09:02:37.9608 => Sending: 01 03 50 12 00 02 75 0E 
09:02:37.9808 <= Receiving: 62 23 
09:02:37.9808 Timeout
09:02:38.0008 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:38.0218 <= Receiving: 62 23 
09:02:38.0218 Timeout
09:02:38.0318 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:38.0628 <= Receiving: 00 5C 
09:02:38.0628 Timeout
09:02:38.0728 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:38.0928 <= Receiving: 00 5C 
09:02:38.0928 Timeout
09:02:38.1128 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:38.1338 <= Receiving: 00 5C 
09:02:38.1338 Timeout
09:02:38.1438 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:38.1638 <= Receiving: 69 26 
09:02:38.1638 Timeout
09:02:38.1738 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:38.1938 <= Receiving: 69 26 
09:02:38.1938 Timeout
09:02:38.2138 => Sending: 02 03 50 12 00 02 75 3D 
09:02:38.2338 <= Receiving: 69 26 
09:02:38.2338 Timeout
09:02:38.2438 => Sending: 02 03 50 12 00 02 75 3D 
09:02:38.2638 <= Receiving: EA 1D 
09:02:38.2638 Timeout
09:02:38.2738 => Sending: 02 03 50 12 00 02 75 3D 
09:02:38.2938 <= Receiving: EA 1D 
09:02:38.2938 Timeout
09:02:38.3138 => Sending: 02 03 60 00 00 02 DA 38 
09:02:38.3338 <= Receiving: EA 1D 
09:02:38.3338 Timeout
09:02:38.3438 => Sending: 02 03 60 00 00 02 DA 38 
09:02:38.3638 <= Receiving: 6D D5 
09:02:38.3638 Timeout
09:02:38.3738 => Sending: 02 03 60 00 00 02 DA 38 
09:02:38.4048 <= Receiving: 6D D5 
09:02:38.4048 Timeout
09:02:38.4248 => Sending: 03 03 50 12 00 02 74 EC 
09:02:38.4448 <= Receiving: 6D D5 
09:02:38.4448 Timeout
09:02:38.4548 => Sending: 03 03 50 12 00 02 74 EC 
09:02:38.4748 <= Receiving: 68 5B 
09:02:38.4748 Timeout
09:02:38.4848 => Sending: 03 03 50 12 00 02 74 EC 
09:02:38.5158 <= Receiving: 68 5B 
09:02:38.5158 Timeout
09:02:38.5358 => Sending: 03 03 60 00 00 02 DB E9 
09:02:38.5558 <= Receiving: 68 5B 
09:02:38.5558 Timeout
09:02:38.5658 => Sending: 03 03 60 00 00 02 DB E9 
09:02:38.5858 <= Receiving: F3 8F 
09:02:38.5858 Timeout
09:02:38.5958 => Sending: 03 03 60 00 00 02 DB E9 
09:02:38.6158 <= Receiving: F3 8F 
09:02:38.6158 Timeout
09:02:38.6408 => Sending: 01 03 50 02 00 06 75 08 
09:02:38.6608 <= Receiving: F3 8F 
09:02:38.6608 Timeout
09:02:38.6708 => Sending: 01 03 50 02 00 06 75 08 
09:02:38.7028 <= Receiving: CD 
09:02:38.7028 Timeout
09:02:38.7128 => Sending: 01 03 50 02 00 06 75 08 
09:02:38.7828 Timeout
09:02:38.8028 => Sending: 01 03 50 12 00 02 75 0E 
09:02:38.8028 <= Receiving: 43 73 19 9A 43 
09:02:38.8028 <= Receiving: 72 B3 33 26 
09:02:38.8028 Wrong telegram: ???
09:02:38.8128 => Sending: 01 03 50 12 00 02 75 0E 
09:02:38.8828 Timeout
09:02:38.8928 => Sending: 01 03 50 12 00 02 75 0E 
Free space in internal port monitor buffer: 59.7 kB
09:02:38.9628 Timeout
09:02:38.9828 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.0028 <= Receiving: 1B EE 
09:02:39.0028 Timeout
09:02:39.0128 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.0448 <= Receiving: 00 5C 
09:02:39.0448 Timeout
09:02:39.0548 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.0748 <= Receiving: 00 5C 
09:02:39.0748 Timeout
09:02:39.0948 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:39.1148 <= Receiving: 00 5C 
09:02:39.1148 Timeout
09:02:39.1512 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:39.1712 <= Receiving: 69 26 
09:02:39.1712 Timeout
09:02:39.1812 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:39.2012 <= Receiving: 69 26 
09:02:39.2012 Timeout
09:02:39.2212 => Sending: 02 03 50 12 00 02 75 3D 
09:02:39.2412 <= Receiving: 69 26 
09:02:39.2412 Timeout
09:02:39.2512 => Sending: 02 03 50 12 00 02 75 3D 
09:02:39.2712 <= Receiving: EA 1D 
09:02:39.2712 Timeout
09:02:39.2812 => Sending: 02 03 50 12 00 02 75 3D 
09:02:39.3012 <= Receiving: EA 1D 
09:02:39.3012 Timeout
09:02:39.3212 => Sending: 02 03 60 00 00 02 DA 38 
09:02:39.3422 <= Receiving: EA 1D 
09:02:39.3422 Timeout
09:02:39.3522 => Sending: 02 03 60 00 00 02 DA 38 
09:02:39.3722 <= Receiving: 9A 6D D5 
09:02:39.3722 Timeout
09:02:39.3822 => Sending: 02 03 60 00 00 02 DA 38 
09:02:39.4022 <= Receiving: 6D D5 
09:02:39.4022 Timeout
09:02:39.4222 => Sending: 03 03 50 12 00 02 74 EC 
09:02:39.4422 <= Receiving: 6D D5 
09:02:39.4422 Timeout
09:02:39.4522 => Sending: 03 03 50 12 00 02 74 EC 
09:02:39.4722 <= Receiving: 68 5B 
09:02:39.4722 Timeout
09:02:39.4822 => Sending: 03 03 50 12 00 02 74 EC 
09:02:39.5022 <= Receiving: 68 5B 
09:02:39.5022 Timeout
09:02:39.5222 => Sending: 03 03 60 00 00 02 DB E9 
09:02:39.5422 <= Receiving: 68 5B 
09:02:39.5422 Timeout
09:02:39.5522 => Sending: 03 03 60 00 00 02 DB E9 
09:02:39.5722 <= Receiving: F3 8F 
09:02:39.5722 Timeout
09:02:39.5822 => Sending: 03 03 60 00 00 02 DB E9 
09:02:39.6032 <= Receiving: F3 8F 
09:02:39.6032 Timeout
09:02:39.6282 => Sending: 01 03 50 02 00 06 75 08 
09:02:39.6482 <= Receiving: F3 8F 
09:02:39.6482 Timeout
09:02:39.6582 => Sending: 01 03 50 02 00 06 75 08 
09:02:39.6892 <= Receiving: CD 
09:02:39.6892 Timeout
09:02:39.6992 => Sending: 01 03 50 02 00 06 75 08 
09:02:39.7692 Timeout
09:02:39.7892 => Sending: 01 03 50 12 00 02 75 0E 
09:02:39.7892 <= Receiving: 43 73 00 00 43 
09:02:39.7892 <= Receiving: 72 99 9A A3 
09:02:39.7892 Wrong telegram: ???
09:02:39.7992 => Sending: 01 03 50 12 00 02 75 0E 
09:02:39.8692 Timeout
09:02:39.8792 => Sending: 01 03 50 12 00 02 75 0E 
09:02:39.9492 Timeout
09:02:39.9692 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.9892 <= Receiving: 5C 1B EE 
09:02:39.9892 Timeout
09:02:39.9992 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:40.0292 <= Receiving: 3D 
09:02:40.0292 Timeout
09:02:40.0392 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:40.1092 Timeout
09:02:40.1292 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:40.1492 <= Receiving: 00 5C 
09:02:40.1492 Timeout
09:02:40.1592 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:40.1792 <= Receiving: 69 26 
09:02:40.1792 Timeout
09:02:40.1892 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:40.2212 <= Receiving: 69 26 
09:02:40.2212 Timeout
09:02:40.2412 => Sending: 02 03 50 12 00 02 75 3D 
09:02:40.2612 <= Receiving: 52 69 26 
09:02:40.2612 Timeout
09:02:40.2712 => Sending: 02 03 50 12 00 02 75 3D 
09:02:40.2912 <= Receiving: DD 3E 
09:02:40.2912 Timeout
09:02:40.3012 => Sending: 02 03 50 12 00 02 75 3D 
09:02:40.3212 <= Receiving: DD 3E 
09:02:40.3212 Timeout
09:02:40.3412 => Sending: 02 03 60 00 00 02 DA 38 
09:02:40.3612 <= Receiving: DD 3E 
09:02:40.3612 Timeout
09:02:40.3712 => Sending: 02 03 60 00 00 02 DA 38 
09:02:40.4032 <= Receiving: 6D D5 
09:02:40.4032 Timeout
09:02:40.4132 => Sending: 02 03 60 00 00 02 DA 38 
09:02:40.4332 <= Receiving: 6D D5 
09:02:40.4332 Timeout
09:02:40.4532 => Sending: 03 03 50 12 00 02 74 EC 
09:02:40.4732 <= Receiving: 6D D5 
09:02:40.4732 Timeout
09:02:40.4832 => Sending: 03 03 50 12 00 02 74 EC 
09:02:40.5032 <= Receiving: 68 5B 
09:02:40.5032 Timeout
09:02:40.5132 => Sending: 03 03 50 12 00 02 74 EC 
09:02:40.5332 <= Receiving: 68 5B 
09:02:40.5332 Timeout

Many thanks in advance for your help and kind regards

[tomas_hora]

@pascal37 said in M525 stopped working:

cat /run/unipi/io_group1/uart_config

Did you run this command when you received the Timeout or afer you restarted it?

[pascal37]

The command was run while serial communication timeouts, but output is the same after a PLC restart.

[Martin Kudláček]

This post is deleted!

[tomas_hora]

@pascal37 Can you try increasing the Max telegram duration a bit higher? For example 300ms? If that won't help. You can also try to increase the pause between telegrams to 5ms.

[pascal37]

Hi @tomas_hora,

I have increased Max telegram duration to 300ms + added a 5ms pause between telegrams.

I have also update my M525 to latest Axon runtime (2.3.0.8) with latest firmwares/drivers.

I will let you know if issue occurs again.
Many thanks and kind regards.

[pascal37]

Hi @tomas_hora, issue occured again again with port 1.1 and new settings (Max telegram duration to 300ms + added a 5ms pause between telegrams.)
A simple PLC reboot (from Debian OS) was enough to regain data acquisition from the 3 Inepro PRO380 meters.

I do not understand why problem never occured when using port 1.3 of same PLC, nor with my previous M505.
What is the difference between ports 1.1 and 1.3 ?
If same hardware, could it be a hardware issue with port 1.1 on my M525?

22:13:08.5230 => Send: 03 03 60 00 00 02 DB E9 
22:13:08.5530 <= Recv: CA CB 
22:13:08.5530 Timeout
22:13:08.5630 => Send: 03 03 60 00 00 02 DB E9 
22:13:08.5930 <= Recv: CA CB 
22:13:08.5930 Timeout
22:13:08.6230 => Send: 01 03 50 02 00 06 75 08 
22:13:08.6530 <= Recv: CA CB 
22:13:08.6530 Timeout
22:13:08.6630 => Send: 01 03 50 02 00 06 75 08 
22:13:08.6630 <= Recv: 43 71 CC CD 43 
22:13:08.7040 <= Recv: 72 E6 66 
22:13:08.7040 Wrong telegram: ???
22:13:08.7140 => Send: 01 03 50 02 00 06 75 08 
22:13:09.0140 Timeout
22:13:09.0390 => Send: 01 03 50 12 00 02 75 0E 
22:13:09.0390 <= Recv: 43 71 CC CD 43 
22:13:09.0390 <= Recv: 72 E6 66 52 
22:13:09.0390 Wrong telegram: ???
22:13:09.0490 => Send: 01 03 50 12 00 02 75 0E 
22:13:09.3490 Timeout
22:13:09.3590 => Send: 01 03 50 12 00 02 75 0E 
22:13:09.3890 <= Recv: 4B E3 
22:13:09.3890 Timeout
22:13:09.4140 => Send: 01 03 60 0C 00 02 1A 08 
22:13:09.4440 <= Recv: EC 4B E3 
22:13:09.4440 Timeout
22:13:09.4540 => Send: 01 03 60 0C 00 02 1A 08 
22:13:09.4840 <= Recv: CC D8 
22:13:09.4840 Timeout
22:13:09.4940 => Send: 01 03 60 0C 00 02 1A 08 
22:13:09.5240 <= Recv: CC D8 
22:13:09.5240 Timeout
22:13:09.5490 => Send: 01 03 60 18 00 02 5A 0C 
22:13:09.5790 <= Recv: CC D8 
22:13:09.5790 Timeout
22:13:09.5890 => Send: 01 03 60 18 00 02 5A 0C 
22:13:09.6190 <= Recv: 72 00 
22:13:09.6190 Timeout
22:13:09.6290 => Send: 01 03 60 18 00 02 5A 0C 
22:13:09.6590 <= Recv: 72 00 
22:13:09.6590 Timeout
22:13:09.6840 => Send: 02 03 50 12 00 02 75 3D 
22:13:09.7140 <= Recv: 72 00 
22:13:09.7140 Timeout
22:13:09.7240 => Send: 02 03 50 12 00 02 75 3D 
22:13:09.7540 <= Recv: 46 88 
22:13:09.7540 Timeout
22:13:09.7640 => Send: 02 03 50 12 00 02 75 3D 
22:13:09.7940 <= Recv: 46 88 
22:13:09.7940 Timeout
22:13:09.8190 => Send: 02 03 60 00 00 02 DA 38 
22:13:09.8490 <= Recv: 46 88 
22:13:09.8490 Timeout
22:13:09.8590 => Send: 02 03 60 00 00 02 DA 38 
22:13:09.8890 <= Recv: 85 9B 
22:13:09.8890 Timeout
22:13:09.8990 => Send: 02 03 60 00 00 02 DA 38 
22:13:09.9290 <= Recv: 85 9B 
22:13:09.9290 Timeout
22:13:09.9540 => Send: 03 03 50 12 00 02 74 EC 
22:13:09.9840 <= Recv: 85 9B 
22:13:09.9840 Timeout
22:13:09.9940 => Send: 03 03 50 12 00 02 74 EC 
22:13:10.0240 <= Recv: 68 5B 
22:13:10.0240 Timeout
22:13:10.0340 => Send: 03 03 50 12 00 02 74 EC 
22:13:10.0640 <= Recv: 68 5B 
22:13:10.0640 Timeout
22:13:10.0890 => Send: 03 03 60 00 00 02 DB E9 
22:13:10.1190 <= Recv: 68 5B 
22:13:10.1190 Timeout
22:13:10.1290 => Send: 03 03 60 00 00 02 DB E9 
22:13:10.1700 <= Recv: 04 
22:13:10.1700 Timeout
22:13:10.1800 => Send: 03 03 60 00 00 02 DB E9 
22:13:10.4800 Timeout
22:13:10.5100 => Send: 01 03 50 02 00 06 75 08 
22:13:10.5623 <= Recv: CA CB 
22:13:10.5623 Timeout
22:13:10.5723 => Send: 01 03 50 02 00 06 75 08 
22:13:10.5723 <= Recv: 43 71 CC CD 43 
22:13:10.6133 <= Recv: 72 E6 66 52 
22:13:10.6133 Wrong telegram: ???
22:13:10.6233 => Send: 01 03 50 02 00 06 75 08 
22:13:10.9233 Timeout
22:13:10.9483 => Send: 01 03 50 12 00 02 75 0E 
22:13:10.9483 <= Recv: 43 71 CC CD 43 
22:13:10.9483 <= Recv: 72 E6 66 52 
22:13:10.9483 Wrong telegram: ???
22:13:10.9583 => Send: 01 03 50 12 00 02 75 0E 
22:13:11.2583 Timeout
22:13:11.2683 => Send: 01 03 50 12 00 02 75 0E 
22:13:11.2993 <= Recv: 4B E3 
22:13:11.2993 Timeout
22:13:11.3243 => Send: 01 03 60 0C 00 02 1A 08 
22:13:11.3543 <= Recv: 4B E3 
22:13:11.3543 Timeout
22:13:11.3643 => Send: 01 03 60 0C 00 02 1A 08 
22:13:11.3943 <= Recv: CC D8 
22:13:11.3943 Timeout
22:13:11.4043 => Send: 01 03 60 0C 00 02 1A 08 
22:13:11.4343 <= Recv: CC D8 
22:13:11.4343 Timeout
22:13:11.4593 => Send: 01 03 60 18 00 02 5A 0C 
22:13:11.4893 <= Recv: CC D8 
22:13:11.4893 Timeout
22:13:11.4993 => Send: 01 03 60 18 00 02 5A 0C 
22:13:11.5293 <= Recv: 72 00 
22:13:11.5293 Timeout
22:13:11.5393 => Send: 01 03 60 18 00 02 5A 0C 
22:13:11.5693 <= Recv: E1 72 00 
22:13:11.5693 Timeout
22:13:11.5943 => Send: 02 03 50 12 00 02 75 3D 
22:13:11.6243 <= Recv: 72 00 
22:13:11.6243 Timeout
22:13:11.6343 => Send: 02 03 50 12 00 02 75 3D 
22:13:11.6643 <= Recv: 46 88 
22:13:11.6643 Timeout
22:13:11.6743 => Send: 02 03 50 12 00 02 75 3D 
22:13:11.7043 <= Recv: 46 88 
22:13:11.7043 Timeout
22:13:11.7293 => Send: 02 03 60 00 00 02 DA 38 
22:13:11.7593 <= Recv: 46 88 
22:13:11.7593 Timeout
22:13:11.7693 => Send: 02 03 60 00 00 02 DA 38 
22:13:11.7993 <= Recv: 85 9B 
22:13:11.7993 Timeout
22:13:11.8093 => Send: 02 03 60 00 00 02 DA 38 
22:13:11.8393 <= Recv: 85 9B 
22:13:11.8393 Timeout
22:13:11.8643 => Send: 03 03 50 12 00 02 74 EC 
22:13:11.8943 <= Recv: 85 9B 
22:13:11.8943 Timeout
22:13:11.9043 => Send: 03 03 50 12 00 02 74 EC 
22:13:11.9343 <= Recv: 68 5B 
22:13:11.9343 Timeout
22:13:11.9443 => Send: 03 03 50 12 00 02 74 EC 
22:13:11.9743 <= Recv: 68 5B 
22:13:11.9743 Timeout
22:13:11.9993 => Send: 03 03 60 00 00 02 DB E9 
22:13:12.0293 <= Recv: 68 5B 
22:13:12.0293 Timeout
22:13:12.0393 => Send: 03 03 60 00 00 02 DB E9 
22:13:12.0693 <= Recv: CA CB 
22:13:12.0693 Timeout
22:13:12.0793 => Send: 03 03 60 00 00 02 DB E9 
22:13:12.1093 <= Recv: CA CB 
22:13:12.1093 Timeout
22:13:12.1393 => Send: 01 03 50 02 00 06 75 08 
22:13:12.1693 <= Recv: CA CB 
22:13:12.1693 Timeout
22:13:12.1793 => Send: 01 03 50 02 00 06 75 08 
22:13:12.1793 <= Recv: 43 71 B3 33 43 
22:13:12.2193 <= Recv: 72 E6 66 70 
22:13:12.2193 Wrong telegram: ???
22:13:12.2293 => Send: 01 03 50 02 00 06 75 08 
22:13:12.5293 Timeout
22:13:12.5543 => Send: 01 03 50 12 00 02 75 0E 
22:13:12.5543 <= Recv: 43 71 B3 33 43 
22:13:12.5543 <= Recv: 72 E6 66 70 
22:13:12.5543 Wrong telegram: ???
22:13:12.5643 => Send: 01 03 50 12 00 02 75 0E 
22:13:12.8643 Timeout
22:13:12.8743 => Send: 01 03 50 12 00 02 75 0E 
22:13:12.9043 <= Recv: 4B E3 
22:13:12.9043 Timeout
22:13:12.9293 => Send: 01 03 60 0C 00 02 1A 08 
22:13:12.9593 <= Recv: 4B E3 
22:13:12.9593 Timeout
22:13:12.9693 => Send: 01 03 60 0C 00 02 1A 08 
22:13:12.9993 <= Recv: CC D8 
22:13:12.9993 Timeout
22:13:13.0093 => Send: 01 03 60 0C 00 02 1A 08 
22:13:13.0393 <= Recv: CC D8 
22:13:13.0393 Timeout
22:13:13.0643 => Send: 01 03 60 18 00 02 5A 0C 
22:13:13.0943 <= Recv: CC D8 
22:13:13.0943 Timeout
22:13:13.1043 => Send: 01 03 60 18 00 02 5A 0C 
22:13:13.1557 <= Recv: 72 00 
22:13:13.1557 Timeout
22:13:13.1657 => Send: 01 03 60 18 00 02 5A 0C 
22:13:13.1957 <= Recv: 72 00 
22:13:13.1957 Timeout
22:13:13.2207 => Send: 02 03 50 12 00 02 75 3D 
22:13:13.2507 <= Recv: 72 00 
22:13:13.2507 Timeout
22:13:13.2607 => Send: 02 03 50 12 00 02 75 3D 
22:13:13.2907 <= Recv: 46 88 
22:13:13.2907 Timeout
22:13:13.3007 => Send: 02 03 50 12 00 02 75 3D 
22:13:13.3307 <= Recv: 46 88 
22:13:13.3307 Timeout
22:13:13.3557 => Send: 02 03 60 00 00 02 DA 38 
22:13:13.3857 <= Recv: 46 88 
22:13:13.3857 Timeout
22:13:13.3957 => Send: 02 03 60 00 00 02 DA 38 
22:13:13.4257 <= Recv: 85 9B 
22:13:13.4257 Timeout
22:13:13.4357 => Send: 02 03 60 00 00 02 DA 38 
22:13:13.4657 <= Recv: 85 9B 
22:13:13.4657 Timeout
22:13:13.4907 => Send: 03 03 50 12 00 02 74 EC 
22:13:13.5207 <= Recv: 85 9B 
22:13:13.5207 Timeout
22:13:13.5307 => Send: 03 03 50 12 00 02 74 EC 
22:13:13.5607 <= Recv: 68 5B 
22:13:13.5607 Timeout
22:13:13.5707 => Send: 03 03 50 12 00 02 74 EC 
22:13:13.6007 <= Recv: 68 5B 
22:13:13.6007 Timeout
22:13:13.6257 => Send: 03 03 60 00 00 02 DB E9 
22:13:13.6557 <= Recv: 68 5B 
22:13:13.6557 Timeout
22:13:13.6657 => Send: 03 03 60 00 00 02 DB E9 
22:13:13.6957 <= Recv: CA CB 
22:13:13.6957 Timeout
22:13:13.7057 => Send: 03 03 60 00 00 02 DB E9 
22:13:13.7357 <= Recv: 0A CA CB 
22:13:13.7357 Timeout
22:13:13.7657 => Send: 01 03 50 02 00 06 75 08 
22:13:13.7957 <= Recv: CA CB 
22:13:13.7957 Timeout
22:13:13.8057 => Send: 01 03 50 02 00 06 75 08 
22:13:13.8057 <= Recv: 43 71 CC CD 43 
22:13:13.8447 <= Recv: 72 CC CD BF 07 
22:13:13.8447 Wrong telegram: ???
22:13:13.8547 => Send: 01 03 50 02 00 06 75 08 
22:13:13.8547 <= Recv: 43 71 CC CD 43 
22:13:13.8847 <= Recv: 72 CC CD BF 07 
22:13:13.8847 Wrong telegram: ???
22:13:13.9097 => Send: 01 03 50 12 00 02 75 0E 
22:13:13.9097 <= Recv: 43 71 CC CD 43 
22:13:13.9097 <= Recv: 72 CC CD BF 
22:13:13.9097 Wrong telegram: ???
22:13:13.9197 => Send: 01 03 50 12 00 02 75 0E 
22:13:14.2197 Timeout
22:13:14.2297 => Send: 01 03 50 12 00 02 75 0E 
22:13:14.2597 <= Recv: 4B E3 
22:13:14.2597 Timeout
22:13:14.2847 => Send: 01 03 60 0C 00 02 1A 08 
22:13:14.3147 <= Recv: 4B E3 
22:13:14.3147 Timeout
22:13:14.3247 => Send: 01 03 60 0C 00 02 1A 08 
22:13:14.3547 <= Recv: CC D8 
22:13:14.3547 Timeout
22:13:14.3647 => Send: 01 03 60 0C 00 02 1A 08 
22:13:14.3947 <= Recv: CC D8 
22:13:14.3947 Timeout
22:13:14.4197 => Send: 01 03 60 18 00 02 5A 0C 
22:13:14.4497 <= Recv: CC D8 
22:13:14.4497 Timeout
22:13:14.4597 => Send: 01 03 60 18 00 02 5A 0C 
22:13:14.4897 <= Recv: 72 00 
22:13:14.4897 Timeout
22:13:14.4997 => Send: 01 03 60 18 00 02 5A 0C 
22:13:14.5297 <= Recv: 72 00 
22:13:14.5297 Timeout
22:13:14.5547 => Send: 02 03 50 12 00 02 75 3D 
22:13:14.5847 <= Recv: 72 00 
22:13:14.5847 Timeout
22:13:14.5947 => Send: 02 03 50 12 00 02 75 3D 
22:13:14.6247 <= Recv: 46 88 
22:13:14.6247 Timeout
22:13:14.6347 => Send: 02 03 50 12 00 02 75 3D 
22:13:14.6647 <= Recv: 46 88 
22:13:14.6647 Timeout
22:13:14.6897 => Send: 02 03 60 00 00 02 DA 38 
22:13:14.7197 <= Recv: 46 88 
22:13:14.7197 Timeout
22:13:14.7297 => Send: 02 03 60 00 00 02 DA 38 
22:13:14.7597 <= Recv: 85 9B 
22:13:14.7597 Timeout
22:13:14.7697 => Send: 02 03 60 00 00 02 DA 38 
22:13:14.7997 <= Recv: 85 9B 
22:13:14.7997 Timeout
22:13:14.8247 => Send: 03 03 50 12 00 02 74 EC 
22:13:14.8547 <= Recv: 85 9B 
22:13:14.8547 Timeout
22:13:14.8647 => Send: 03 03 50 12 00 02 74 EC 
22:13:14.8947 <= Recv: 68 5B 
22:13:14.9166 Timeout
22:13:14.9266 => Send: 03 03 50 12 00 02 74 EC 
22:13:14.9566 <= Recv: 68 5B 
22:13:14.9566 Timeout
22:13:14.9816 => Send: 03 03 60 00 00 02 DB E9 
22:13:15.0116 <= Recv: 68 5B 
22:13:15.0116 Timeout
22:13:15.0216 => Send: 03 03 60 00 00 02 DB E9 
22:13:15.0516 <= Recv: CA CB 
22:13:15.0516 Timeout
22:13:15.0616 => Send: 03 03 60 00 00 02 DB E9 
22:13:15.0916 <= Recv: CA CB 
22:13:15.0916 Timeout
22:13:15.1216 => Send: 01 03 50 02 00 06 75 08 
22:13:15.1516 <= Recv: CA CB 
22:13:15.1516 Timeout
22:13:15.1616 => Send: 01 03 50 02 00 06 75 08 
22:13:15.1616 <= Recv: 43 71 B3 33 43 
22:13:15.2016 <= Recv: 72 E6 66 C3 C3 
22:13:15.2016 Wrong telegram: ???
22:13:15.2116 => Send: 01 03 50 02 00 06 75 08 
22:13:15.2116 <= Recv: 43 71 B3 33 43 
22:13:15.2416 <= Recv: 72 E6 66 C3 C3 
22:13:15.2416 Wrong telegram: ???
22:13:15.2666 => Send: 01 03 50 12 00 02 75 0E 
22:13:15.2666 <= Recv: 43 71 B3 33 43 
22:13:15.2666 <= Recv: 72 E6 66 C3 
22:13:15.2666 Wrong telegram: ???
22:13:15.2766 => Send: 01 03 50 12 00 02 75 0E 
22:13:15.5766 Timeout
22:13:15.5866 => Send: 01 03 50 12 00 02 75 0E 
22:13:15.6166 <= Recv: 4B E3 
22:13:15.6166 Timeout
22:13:15.6416 => Send: 01 03 60 0C 00 02 1A 08 
22:13:15.6716 <= Recv: 4B E3 
22:13:15.6716 Timeout
22:13:15.6816 => Send: 01 03 60 0C 00 02 1A 08 
22:13:15.7116 <= Recv: CC D8 
22:13:15.7116 Timeout
22:13:15.7216 => Send: 01 03 60 0C 00 02 1A 08 
22:13:15.7516 <= Recv: CC D8 
22:13:15.7516 Timeout
22:13:15.7766 => Send: 01 03 60 18 00 02 5A 0C 
22:13:15.8066 <= Recv: CC D8 
22:13:15.8066 Timeout
22:13:15.8166 => Send: 01 03 60 18 00 02 5A 0C 
22:13:15.8466 <= Recv: 72 00 
22:13:15.8466 Timeout
22:13:15.8566 => Send: 01 03 60 18 00 02 5A 0C 
22:13:15.8866 <= Recv: 72 00 
22:13:15.8866 Timeout
22:13:15.9116 => Send: 02 03 50 12 00 02 75 3D 
22:13:15.9416 <= Recv: 72 00 
22:13:15.9416 Timeout
22:13:15.9516 => Send: 02 03 50 12 00 02 75 3D 
22:13:15.9926 <= Recv: 04 
22:13:15.9926 Timeout
22:13:16.0026 => Send: 02 03 50 12 00 02 75 3D 
22:13:16.3026 Timeout
22:13:16.3276 => Send: 02 03 60 00 00 02 DA 38 
22:13:16.3576 <= Recv: 46 88 
22:13:16.3576 Timeout
22:13:16.3676 => Send: 02 03 60 00 00 02 DA 38 
22:13:16.3976 <= Recv: 85 9B 
22:13:16.3976 Timeout
22:13:16.4076 => Send: 02 03 60 00 00 02 DA 38 
22:13:16.4376 <= Recv: 85 9B 
22:13:16.4376 Timeout
22:13:16.4626 => Send: 03 03 50 12 00 02 74 EC 
22:13:16.4926 <= Recv: 85 9B 
22:13:16.4926 Timeout
22:13:16.5026 => Send: 03 03 50 12 00 02 74 EC 
22:13:16.5326 <= Recv: 68 5B 
22:13:16.5326 Timeout
22:13:16.5426 => Send: 03 03 50 12 00 02 74 EC 
22:13:16.5726 <= Recv: 68 5B 
22:13:16.5726 Timeout
22:13:16.5976 => Send: 03 03 60 00 00 02 DB E9 
22:13:16.6276 <= Recv: 68 5B 
22:13:16.6276 Timeout
22:13:16.6376 => Send: 03 03 60 00 00 02 DB E9 
22:13:16.6676 <= Recv: CA CB 
22:13:16.6676 Timeout
22:13:16.6776 => Send: 03 03 60 00 00 02 DB E9 
22:13:16.7076 <= Recv: CA CB 
22:13:16.7076 Timeout
22:13:16.7376 => Send: 01 03 50 02 00 06 75 08 
22:13:16.7686 <= Recv: CA CB 
22:13:16.7686 Timeout

[martin_triska]

Hello @pascal37,

thank you for the information sent. Unfortunately, based on it, we are not able to determine the exact source of the issue.
Can you please check the following:

1. Again, make sure you have the latest firmware in the I/O board flashed. Type:

/opt/unipi/tools/fwspi -i 0

2. Can you confirm that the i2c-0 error in the dmesg log appears always in the moment of communication goes wrong? It seems there is a correlation (863475 seconds is almost exactly 10 days).

3. Are you sure that no other running processes access the interface (/dev/ttyNS0)? Please check also your time-scheduled tasks (e.g. executed from cron).

4. Are you sure that no noise or interference from the environment or power supply can infiltrate the unit?

As the last resort, we are able to simulate your application with the same HW, but it will be pretty time-consuming. In the meantime, please use another RS485 interface.

[pascal37]

Hi @martin_triska,

1. I did a reflash (USB method) of my M525 with latest Mervis OS image (2.3.0.10), firmwares are already up-to-date:

unipi@M525-sn8:~$ sudo /opt/unipi/tools/fwspi -i 0
Boardset:     0 B-1000                         (v1.0)
Baseboard:    0 B-1000                         (v2.0)
Firmware: v5.50
unipi@M525-sn8:~$ sudo /opt/unipi/tools/fwspi -i 1
Boardset:    19 E-4Ai4Ao_U-R485Di4Ro5          (v1.0)
Baseboard:   11 E-4Ai4Ao                       (v1.0)
Firmware: v5.50

2. From what I've seen after serial communications "crashes", I did not see anymore an i2c-0 error in demsg, so I am not sure this is correlated.

3. Yes: it is a standard Mervis OS image + 2 pythons scripts running as daemons with pymodbus client over ModbusTCP, so no serial communications there.

4. Yes: I am using shielded cable specific for RS485, the bus is properly terminated at each end. Cable shield is properly grounded at PLC side and also to PLC itself. Environment is stable and, so long the issue is not occurring, communication is working perfectly.
   With the increase of Max telegram duration to 300ms, there is absolutely no error during multiple days:

It seems most likely a driver or Mervis runtime issue related to M525 model and port 1.1 !?
As I said, I never had the issue when using port 1.3 instead of port 1.1
I have also another RS485 bus on port 2.1 and no issue either.
This issue also never occurred with port 1.1 on my previous M505 PLC

If you want to try reproduce the issue, you need 3 Inepro PRO380 meters and read constantly multiple values from them like power, energy, voltage.
In my case, the maximum uptime before a crash is about 10 days.

I can use port 1.3 in the meantime, but I will need the 3rd RS485 port later for my project...
But let's check again if it is really stable with port 1.3 over >10 days !

Many thanks in advance and kind regards