M525 stopped working

pascal37

I found article: https://kb.unipi.technology/en:sw:04-unipi-firmware?s[]=firmware

I had firmware v5.40 and just updated now to v5.50.

Martin Kudláček

Hello @pascal37,
all of that is ok. You can check what is going on on the serial port in Mervis IDE directly. Switch to debugging mode and in the bottom panel look for "port monitor". Start the monitoring of the affected port. You should see more information.

Also, try disconnecting the meters to see if the problem is caused by one of them. And also make sure, that you use shielded twist pair cables, with the shielding connected to earth ground on one end and use proper termination of the bus.

Best regards,
Martin

pascal37

Hi @martin-kudláček,

The problem recurred, again after 10 days of smooth operation.

Here is the output of the serial port:

0_1587715636275_bfb576df-88b0-4c6c-8428-23404e352219-image.png

Since the solution works fine during multiple days without any error on the serial interface, I do not think there is an issue on the RS485 bus. Yes, I am using am appropriate shielded cable with terminations on both ends.

I recently replaced the M505 with an M525 because of relay issues but I never had this RS485 serial bus issue with the M505 !?

There are 3 Inepro PRO380 devices on the bus, I tried to disconnect each Meter, one after the other, but the issue seems not caused by one of them. I have also connected each meter individually direcly to the M525 with another cable but no success either.

Anything else we can do on the M525 to further troublehoot before I try restart/redeploy the solution ?

Many thanks in advance and kind regards,

Pascal

pascal37

Hi @Martin-Kudláček,

A simple "sudo reboot" of the PLC has solved the issue and communication works again error free:

0_1587828876138_421e417e-cd95-4f47-9c5d-16898c910ff4-image.png

Current serial port is RS485-1.1, I will try if same issue occurs after 10 days if using port RS485-1.3 ...

pascal37

Hi @Martin-Kudláček,

No issue after 15 days of uptime using port RS485-1.3 :

0_1589308492689_8fae0eef-f7df-4118-aa5a-82b5300d6a28-image.png

I switch now back to port RS485-1.1 to check if issue occurs again after 10 days...

tomas_hora

@pascal37 if it occurs again, please send the captured data on the port monitor in txt file from a longer period so we could get better overview

pascal37

Hi @tomas_hora,

The issue occured again with port RS485-1.1 just after 5 days of uptime, but sorry, I was not capturing serial data before the "crash". This is quite difficult to do: it means leave IDE open in debugging mode during up to 10 days and be able to select all the output (> 50 millions packets) !? Is there any better why to archieve that ?

Workaround is to redeploy the solution from IDE (cold restart), and serial communication works again immedately.
The issue is definitvely not on the RS485 bus or Inepro devices.

I had this issue now 3 times with the port 1.1 on the Axon M525, but not with port 1.3, nor with an Axon M505.
Since a solution deploy does not reboot the Debian OS of the PLC, the issue should be somewhere in the Mervis runtime !?

The first time the issue occured, I did a complete reboot of the PLC, but this is not a workaround... as if a counter in the Mervis runtime is not reseted properly like a solution deploy with cold restart.

Many thanks in advance for your help and kind regards.

tomas_hora

Hello @pascal37, I did not men to capture the data at the time it stops responding, but jus to get a longer period of the caputed data after it stops communicatig to find out if it's all timeout or what is going on the line.

What should be the configuration of the RS485 (baudrate, parity, ...)

Could you execute this command in linux while the communication stops?

cat /run/unipi/io_group1/uart_config

pascal37

Hi @tomas_hora,

Issue occurred again with port RS485-1.1

This time, a simple reboot (from Debian OS) was enough to regain data acquisition from the 3 Inepro PRO380 meters.

Port configuration:

cat /run/unipi/io_group1/uart_config
9bd

Data captured after communication stops working:

09:02:36.2939 => Sending: 02 03 50 12 00 02 75 3D 
09:02:36.3139 <= Receiving: CB EC EB 
09:02:36.3139 Timeout
09:02:36.3239 => Sending: 02 03 50 12 00 02 75 3D 
09:02:36.3439 <= Receiving: CB EC EB 
09:02:36.3439 Timeout
09:02:36.3639 => Sending: 02 03 60 00 00 02 DA 38 
09:02:36.3839 <= Receiving: EC EB 
09:02:36.3839 Timeout
09:02:36.3939 => Sending: 02 03 60 00 00 02 DA 38 
09:02:36.4139 <= Receiving: 6D D5 
09:02:36.4139 Timeout
09:02:36.4239 => Sending: 02 03 60 00 00 02 DA 38 
09:02:36.4539 <= Receiving: 6D D5 
09:02:36.4539 Timeout
09:02:36.4739 => Sending: 03 03 50 12 00 02 74 EC 
09:02:36.4939 <= Receiving: 6D D5 
09:02:36.4939 Timeout
09:02:36.5039 => Sending: 03 03 50 12 00 02 74 EC 
09:02:36.5359 <= Receiving: 68 5B 
09:02:36.5359 Timeout
09:02:36.5459 => Sending: 03 03 50 12 00 02 74 EC 
09:02:36.5659 <= Receiving: 6F 68 5B 
09:02:36.5659 Timeout
09:02:36.5859 => Sending: 03 03 60 00 00 02 DB E9 
09:02:36.6059 <= Receiving: 68 5B 
09:02:36.6059 Timeout
09:02:36.6159 => Sending: 03 03 60 00 00 02 DB E9 
09:02:36.6459 <= Receiving: 3D F3 8F 
09:02:36.6459 Timeout
09:02:36.6559 => Sending: 03 03 60 00 00 02 DB E9 
09:02:36.6759 <= Receiving: F3 8F 
09:02:36.6759 Timeout
09:02:36.7009 => Sending: 01 03 50 02 00 06 75 08 
09:02:36.7209 <= Receiving: F3 8F 
09:02:36.7209 Timeout
09:02:36.7309 => Sending: 01 03 50 02 00 06 75 08 
09:02:36.7599 <= Receiving: 43 
09:02:36.7599 Timeout
09:02:36.7699 => Sending: 01 03 50 02 00 06 75 08 
09:02:36.8399 Timeout
09:02:36.8599 => Sending: 01 03 50 12 00 02 75 0E 
09:02:36.8599 <= Receiving: 43 73 00 00 43 
09:02:36.8599 <= Receiving: 72 99 9A A3 
09:02:36.8599 Wrong telegram: ???
09:02:36.8699 => Sending: 01 03 50 12 00 02 75 0E 
09:02:36.9399 Timeout
09:02:36.9499 => Sending: 01 03 50 12 00 02 75 0E 
09:02:36.9699 <= Receiving: 62 23 
09:02:36.9699 Timeout
09:02:36.9899 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:37.0099 <= Receiving: 62 23 
09:02:37.0099 Timeout
09:02:37.0199 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:37.0519 <= Receiving: 9F 
09:02:37.0519 Timeout
09:02:37.0619 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:37.1319 Timeout
09:02:37.1519 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:37.1719 <= Receiving: 00 5C 
09:02:37.1719 Timeout
09:02:37.1819 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:37.2129 <= Receiving: 69 26 
09:02:37.2129 Timeout
09:02:37.2229 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:37.2429 <= Receiving: 69 26 
09:02:37.2429 Timeout
09:02:37.2629 => Sending: 02 03 50 12 00 02 75 3D 
09:02:37.2829 <= Receiving: 52 69 26 
09:02:37.2829 Timeout
09:02:37.2929 => Sending: 02 03 50 12 00 02 75 3D 
09:02:37.3129 <= Receiving: EA 1D 
09:02:37.3129 Timeout
09:02:37.3229 => Sending: 02 03 50 12 00 02 75 3D 
09:02:37.3429 <= Receiving: EA 1D 
09:02:37.3429 Timeout
09:02:37.3629 => Sending: 02 03 60 00 00 02 DA 38 
09:02:37.3839 <= Receiving: EA 1D 
09:02:37.3839 Timeout
09:02:37.3939 => Sending: 02 03 60 00 00 02 DA 38 
09:02:37.4139 <= Receiving: 6D D5 
09:02:37.4139 Timeout
09:02:37.4239 => Sending: 02 03 60 00 00 02 DA 38 
09:02:37.4439 <= Receiving: 6D D5 
09:02:37.4439 Timeout
09:02:37.4639 => Sending: 03 03 50 12 00 02 74 EC 
09:02:37.4849 <= Receiving: 6D D5 
09:02:37.4849 Timeout
09:02:37.4949 => Sending: 03 03 50 12 00 02 74 EC 
09:02:37.5159 <= Receiving: 68 5B 
09:02:37.5159 Timeout
09:02:37.5259 => Sending: 03 03 50 12 00 02 74 EC 
09:02:37.5459 <= Receiving: 68 5B 
09:02:37.5459 Timeout
09:02:37.5659 => Sending: 03 03 60 00 00 02 DB E9 
09:02:37.5859 <= Receiving: 68 5B 
09:02:37.5859 Timeout
09:02:37.5959 => Sending: 03 03 60 00 00 02 DB E9 
09:02:37.6279 <= Receiving: F3 8F 
09:02:37.6279 Timeout
09:02:37.6628 => Sending: 03 03 60 00 00 02 DB E9 
09:02:37.6828 <= Receiving: F3 8F 
09:02:37.6828 Timeout
09:02:37.7088 => Sending: 01 03 50 02 00 06 75 08 
09:02:37.7288 <= Receiving: F3 8F 
09:02:37.7288 Timeout
09:02:37.7388 => Sending: 01 03 50 02 00 06 75 08 
09:02:37.7708 <= Receiving: 43 73 
09:02:37.7708 Timeout
09:02:37.7808 => Sending: 01 03 50 02 00 06 75 08 
09:02:37.8508 Timeout
09:02:37.8708 => Sending: 01 03 50 12 00 02 75 0E 
09:02:37.8708 <= Receiving: CD 43 73 00 00 
09:02:37.8708 <= Receiving: 43 72 99 9A 
09:02:37.8708 Wrong telegram: ???
09:02:37.8808 => Sending: 01 03 50 12 00 02 75 0E 
09:02:37.9508 Timeout
09:02:37.9608 => Sending: 01 03 50 12 00 02 75 0E 
09:02:37.9808 <= Receiving: 62 23 
09:02:37.9808 Timeout
09:02:38.0008 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:38.0218 <= Receiving: 62 23 
09:02:38.0218 Timeout
09:02:38.0318 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:38.0628 <= Receiving: 00 5C 
09:02:38.0628 Timeout
09:02:38.0728 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:38.0928 <= Receiving: 00 5C 
09:02:38.0928 Timeout
09:02:38.1128 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:38.1338 <= Receiving: 00 5C 
09:02:38.1338 Timeout
09:02:38.1438 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:38.1638 <= Receiving: 69 26 
09:02:38.1638 Timeout
09:02:38.1738 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:38.1938 <= Receiving: 69 26 
09:02:38.1938 Timeout
09:02:38.2138 => Sending: 02 03 50 12 00 02 75 3D 
09:02:38.2338 <= Receiving: 69 26 
09:02:38.2338 Timeout
09:02:38.2438 => Sending: 02 03 50 12 00 02 75 3D 
09:02:38.2638 <= Receiving: EA 1D 
09:02:38.2638 Timeout
09:02:38.2738 => Sending: 02 03 50 12 00 02 75 3D 
09:02:38.2938 <= Receiving: EA 1D 
09:02:38.2938 Timeout
09:02:38.3138 => Sending: 02 03 60 00 00 02 DA 38 
09:02:38.3338 <= Receiving: EA 1D 
09:02:38.3338 Timeout
09:02:38.3438 => Sending: 02 03 60 00 00 02 DA 38 
09:02:38.3638 <= Receiving: 6D D5 
09:02:38.3638 Timeout
09:02:38.3738 => Sending: 02 03 60 00 00 02 DA 38 
09:02:38.4048 <= Receiving: 6D D5 
09:02:38.4048 Timeout
09:02:38.4248 => Sending: 03 03 50 12 00 02 74 EC 
09:02:38.4448 <= Receiving: 6D D5 
09:02:38.4448 Timeout
09:02:38.4548 => Sending: 03 03 50 12 00 02 74 EC 
09:02:38.4748 <= Receiving: 68 5B 
09:02:38.4748 Timeout
09:02:38.4848 => Sending: 03 03 50 12 00 02 74 EC 
09:02:38.5158 <= Receiving: 68 5B 
09:02:38.5158 Timeout
09:02:38.5358 => Sending: 03 03 60 00 00 02 DB E9 
09:02:38.5558 <= Receiving: 68 5B 
09:02:38.5558 Timeout
09:02:38.5658 => Sending: 03 03 60 00 00 02 DB E9 
09:02:38.5858 <= Receiving: F3 8F 
09:02:38.5858 Timeout
09:02:38.5958 => Sending: 03 03 60 00 00 02 DB E9 
09:02:38.6158 <= Receiving: F3 8F 
09:02:38.6158 Timeout
09:02:38.6408 => Sending: 01 03 50 02 00 06 75 08 
09:02:38.6608 <= Receiving: F3 8F 
09:02:38.6608 Timeout
09:02:38.6708 => Sending: 01 03 50 02 00 06 75 08 
09:02:38.7028 <= Receiving: CD 
09:02:38.7028 Timeout
09:02:38.7128 => Sending: 01 03 50 02 00 06 75 08 
09:02:38.7828 Timeout
09:02:38.8028 => Sending: 01 03 50 12 00 02 75 0E 
09:02:38.8028 <= Receiving: 43 73 19 9A 43 
09:02:38.8028 <= Receiving: 72 B3 33 26 
09:02:38.8028 Wrong telegram: ???
09:02:38.8128 => Sending: 01 03 50 12 00 02 75 0E 
09:02:38.8828 Timeout
09:02:38.8928 => Sending: 01 03 50 12 00 02 75 0E 
Free space in internal port monitor buffer: 59.7 kB
09:02:38.9628 Timeout
09:02:38.9828 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.0028 <= Receiving: 1B EE 
09:02:39.0028 Timeout
09:02:39.0128 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.0448 <= Receiving: 00 5C 
09:02:39.0448 Timeout
09:02:39.0548 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.0748 <= Receiving: 00 5C 
09:02:39.0748 Timeout
09:02:39.0948 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:39.1148 <= Receiving: 00 5C 
09:02:39.1148 Timeout
09:02:39.1512 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:39.1712 <= Receiving: 69 26 
09:02:39.1712 Timeout
09:02:39.1812 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:39.2012 <= Receiving: 69 26 
09:02:39.2012 Timeout
09:02:39.2212 => Sending: 02 03 50 12 00 02 75 3D 
09:02:39.2412 <= Receiving: 69 26 
09:02:39.2412 Timeout
09:02:39.2512 => Sending: 02 03 50 12 00 02 75 3D 
09:02:39.2712 <= Receiving: EA 1D 
09:02:39.2712 Timeout
09:02:39.2812 => Sending: 02 03 50 12 00 02 75 3D 
09:02:39.3012 <= Receiving: EA 1D 
09:02:39.3012 Timeout
09:02:39.3212 => Sending: 02 03 60 00 00 02 DA 38 
09:02:39.3422 <= Receiving: EA 1D 
09:02:39.3422 Timeout
09:02:39.3522 => Sending: 02 03 60 00 00 02 DA 38 
09:02:39.3722 <= Receiving: 9A 6D D5 
09:02:39.3722 Timeout
09:02:39.3822 => Sending: 02 03 60 00 00 02 DA 38 
09:02:39.4022 <= Receiving: 6D D5 
09:02:39.4022 Timeout
09:02:39.4222 => Sending: 03 03 50 12 00 02 74 EC 
09:02:39.4422 <= Receiving: 6D D5 
09:02:39.4422 Timeout
09:02:39.4522 => Sending: 03 03 50 12 00 02 74 EC 
09:02:39.4722 <= Receiving: 68 5B 
09:02:39.4722 Timeout
09:02:39.4822 => Sending: 03 03 50 12 00 02 74 EC 
09:02:39.5022 <= Receiving: 68 5B 
09:02:39.5022 Timeout
09:02:39.5222 => Sending: 03 03 60 00 00 02 DB E9 
09:02:39.5422 <= Receiving: 68 5B 
09:02:39.5422 Timeout
09:02:39.5522 => Sending: 03 03 60 00 00 02 DB E9 
09:02:39.5722 <= Receiving: F3 8F 
09:02:39.5722 Timeout
09:02:39.5822 => Sending: 03 03 60 00 00 02 DB E9 
09:02:39.6032 <= Receiving: F3 8F 
09:02:39.6032 Timeout
09:02:39.6282 => Sending: 01 03 50 02 00 06 75 08 
09:02:39.6482 <= Receiving: F3 8F 
09:02:39.6482 Timeout
09:02:39.6582 => Sending: 01 03 50 02 00 06 75 08 
09:02:39.6892 <= Receiving: CD 
09:02:39.6892 Timeout
09:02:39.6992 => Sending: 01 03 50 02 00 06 75 08 
09:02:39.7692 Timeout
09:02:39.7892 => Sending: 01 03 50 12 00 02 75 0E 
09:02:39.7892 <= Receiving: 43 73 00 00 43 
09:02:39.7892 <= Receiving: 72 99 9A A3 
09:02:39.7892 Wrong telegram: ???
09:02:39.7992 => Sending: 01 03 50 12 00 02 75 0E 
09:02:39.8692 Timeout
09:02:39.8792 => Sending: 01 03 50 12 00 02 75 0E 
09:02:39.9492 Timeout
09:02:39.9692 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:39.9892 <= Receiving: 5C 1B EE 
09:02:39.9892 Timeout
09:02:39.9992 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:40.0292 <= Receiving: 3D 
09:02:40.0292 Timeout
09:02:40.0392 => Sending: 01 03 60 0C 00 02 1A 08 
09:02:40.1092 Timeout
09:02:40.1292 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:40.1492 <= Receiving: 00 5C 
09:02:40.1492 Timeout
09:02:40.1592 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:40.1792 <= Receiving: 69 26 
09:02:40.1792 Timeout
09:02:40.1892 => Sending: 01 03 60 18 00 02 5A 0C 
09:02:40.2212 <= Receiving: 69 26 
09:02:40.2212 Timeout
09:02:40.2412 => Sending: 02 03 50 12 00 02 75 3D 
09:02:40.2612 <= Receiving: 52 69 26 
09:02:40.2612 Timeout
09:02:40.2712 => Sending: 02 03 50 12 00 02 75 3D 
09:02:40.2912 <= Receiving: DD 3E 
09:02:40.2912 Timeout
09:02:40.3012 => Sending: 02 03 50 12 00 02 75 3D 
09:02:40.3212 <= Receiving: DD 3E 
09:02:40.3212 Timeout
09:02:40.3412 => Sending: 02 03 60 00 00 02 DA 38 
09:02:40.3612 <= Receiving: DD 3E 
09:02:40.3612 Timeout
09:02:40.3712 => Sending: 02 03 60 00 00 02 DA 38 
09:02:40.4032 <= Receiving: 6D D5 
09:02:40.4032 Timeout
09:02:40.4132 => Sending: 02 03 60 00 00 02 DA 38 
09:02:40.4332 <= Receiving: 6D D5 
09:02:40.4332 Timeout
09:02:40.4532 => Sending: 03 03 50 12 00 02 74 EC 
09:02:40.4732 <= Receiving: 6D D5 
09:02:40.4732 Timeout
09:02:40.4832 => Sending: 03 03 50 12 00 02 74 EC 
09:02:40.5032 <= Receiving: 68 5B 
09:02:40.5032 Timeout
09:02:40.5132 => Sending: 03 03 50 12 00 02 74 EC 
09:02:40.5332 <= Receiving: 68 5B 
09:02:40.5332 Timeout

Many thanks in advance for your help and kind regards

tomas_hora

@pascal37 said in M525 stopped working:

cat /run/unipi/io_group1/uart_config

Did you run this command when you received the Timeout or afer you restarted it?

pascal37

The command was run while serial communication timeouts, but output is the same after a PLC restart.

Martin Kudláček

This post is deleted!

tomas_hora

@pascal37 Can you try increasing the Max telegram duration a bit higher? For example 300ms? If that won't help. You can also try to increase the pause between telegrams to 5ms.

pascal37

Hi @tomas_hora,

I have increased Max telegram duration to 300ms + added a 5ms pause between telegrams.

I have also update my M525 to latest Axon runtime (2.3.0.8) with latest firmwares/drivers.

I will let you know if issue occurs again.
Many thanks and kind regards.

pascal37

Hi @tomas_hora, issue occured again again with port 1.1 and new settings (Max telegram duration to 300ms + added a 5ms pause between telegrams.)
A simple PLC reboot (from Debian OS) was enough to regain data acquisition from the 3 Inepro PRO380 meters.

I do not understand why problem never occured when using port 1.3 of same PLC, nor with my previous M505.
What is the difference between ports 1.1 and 1.3 ?
If same hardware, could it be a hardware issue with port 1.1 on my M525?

22:13:08.5230 => Send: 03 03 60 00 00 02 DB E9 
22:13:08.5530 <= Recv: CA CB 
22:13:08.5530 Timeout
22:13:08.5630 => Send: 03 03 60 00 00 02 DB E9 
22:13:08.5930 <= Recv: CA CB 
22:13:08.5930 Timeout
22:13:08.6230 => Send: 01 03 50 02 00 06 75 08 
22:13:08.6530 <= Recv: CA CB 
22:13:08.6530 Timeout
22:13:08.6630 => Send: 01 03 50 02 00 06 75 08 
22:13:08.6630 <= Recv: 43 71 CC CD 43 
22:13:08.7040 <= Recv: 72 E6 66 
22:13:08.7040 Wrong telegram: ???
22:13:08.7140 => Send: 01 03 50 02 00 06 75 08 
22:13:09.0140 Timeout
22:13:09.0390 => Send: 01 03 50 12 00 02 75 0E 
22:13:09.0390 <= Recv: 43 71 CC CD 43 
22:13:09.0390 <= Recv: 72 E6 66 52 
22:13:09.0390 Wrong telegram: ???
22:13:09.0490 => Send: 01 03 50 12 00 02 75 0E 
22:13:09.3490 Timeout
22:13:09.3590 => Send: 01 03 50 12 00 02 75 0E 
22:13:09.3890 <= Recv: 4B E3 
22:13:09.3890 Timeout
22:13:09.4140 => Send: 01 03 60 0C 00 02 1A 08 
22:13:09.4440 <= Recv: EC 4B E3 
22:13:09.4440 Timeout
22:13:09.4540 => Send: 01 03 60 0C 00 02 1A 08 
22:13:09.4840 <= Recv: CC D8 
22:13:09.4840 Timeout
22:13:09.4940 => Send: 01 03 60 0C 00 02 1A 08 
22:13:09.5240 <= Recv: CC D8 
22:13:09.5240 Timeout
22:13:09.5490 => Send: 01 03 60 18 00 02 5A 0C 
22:13:09.5790 <= Recv: CC D8 
22:13:09.5790 Timeout
22:13:09.5890 => Send: 01 03 60 18 00 02 5A 0C 
22:13:09.6190 <= Recv: 72 00 
22:13:09.6190 Timeout
22:13:09.6290 => Send: 01 03 60 18 00 02 5A 0C 
22:13:09.6590 <= Recv: 72 00 
22:13:09.6590 Timeout
22:13:09.6840 => Send: 02 03 50 12 00 02 75 3D 
22:13:09.7140 <= Recv: 72 00 
22:13:09.7140 Timeout
22:13:09.7240 => Send: 02 03 50 12 00 02 75 3D 
22:13:09.7540 <= Recv: 46 88 
22:13:09.7540 Timeout
22:13:09.7640 => Send: 02 03 50 12 00 02 75 3D 
22:13:09.7940 <= Recv: 46 88 
22:13:09.7940 Timeout
22:13:09.8190 => Send: 02 03 60 00 00 02 DA 38 
22:13:09.8490 <= Recv: 46 88 
22:13:09.8490 Timeout
22:13:09.8590 => Send: 02 03 60 00 00 02 DA 38 
22:13:09.8890 <= Recv: 85 9B 
22:13:09.8890 Timeout
22:13:09.8990 => Send: 02 03 60 00 00 02 DA 38 
22:13:09.9290 <= Recv: 85 9B 
22:13:09.9290 Timeout
22:13:09.9540 => Send: 03 03 50 12 00 02 74 EC 
22:13:09.9840 <= Recv: 85 9B 
22:13:09.9840 Timeout
22:13:09.9940 => Send: 03 03 50 12 00 02 74 EC 
22:13:10.0240 <= Recv: 68 5B 
22:13:10.0240 Timeout
22:13:10.0340 => Send: 03 03 50 12 00 02 74 EC 
22:13:10.0640 <= Recv: 68 5B 
22:13:10.0640 Timeout
22:13:10.0890 => Send: 03 03 60 00 00 02 DB E9 
22:13:10.1190 <= Recv: 68 5B 
22:13:10.1190 Timeout
22:13:10.1290 => Send: 03 03 60 00 00 02 DB E9 
22:13:10.1700 <= Recv: 04 
22:13:10.1700 Timeout
22:13:10.1800 => Send: 03 03 60 00 00 02 DB E9 
22:13:10.4800 Timeout
22:13:10.5100 => Send: 01 03 50 02 00 06 75 08 
22:13:10.5623 <= Recv: CA CB 
22:13:10.5623 Timeout
22:13:10.5723 => Send: 01 03 50 02 00 06 75 08 
22:13:10.5723 <= Recv: 43 71 CC CD 43 
22:13:10.6133 <= Recv: 72 E6 66 52 
22:13:10.6133 Wrong telegram: ???
22:13:10.6233 => Send: 01 03 50 02 00 06 75 08 
22:13:10.9233 Timeout
22:13:10.9483 => Send: 01 03 50 12 00 02 75 0E 
22:13:10.9483 <= Recv: 43 71 CC CD 43 
22:13:10.9483 <= Recv: 72 E6 66 52 
22:13:10.9483 Wrong telegram: ???
22:13:10.9583 => Send: 01 03 50 12 00 02 75 0E 
22:13:11.2583 Timeout
22:13:11.2683 => Send: 01 03 50 12 00 02 75 0E 
22:13:11.2993 <= Recv: 4B E3 
22:13:11.2993 Timeout
22:13:11.3243 => Send: 01 03 60 0C 00 02 1A 08 
22:13:11.3543 <= Recv: 4B E3 
22:13:11.3543 Timeout
22:13:11.3643 => Send: 01 03 60 0C 00 02 1A 08 
22:13:11.3943 <= Recv: CC D8 
22:13:11.3943 Timeout
22:13:11.4043 => Send: 01 03 60 0C 00 02 1A 08 
22:13:11.4343 <= Recv: CC D8 
22:13:11.4343 Timeout
22:13:11.4593 => Send: 01 03 60 18 00 02 5A 0C 
22:13:11.4893 <= Recv: CC D8 
22:13:11.4893 Timeout
22:13:11.4993 => Send: 01 03 60 18 00 02 5A 0C 
22:13:11.5293 <= Recv: 72 00 
22:13:11.5293 Timeout
22:13:11.5393 => Send: 01 03 60 18 00 02 5A 0C 
22:13:11.5693 <= Recv: E1 72 00 
22:13:11.5693 Timeout
22:13:11.5943 => Send: 02 03 50 12 00 02 75 3D 
22:13:11.6243 <= Recv: 72 00 
22:13:11.6243 Timeout
22:13:11.6343 => Send: 02 03 50 12 00 02 75 3D 
22:13:11.6643 <= Recv: 46 88 
22:13:11.6643 Timeout
22:13:11.6743 => Send: 02 03 50 12 00 02 75 3D 
22:13:11.7043 <= Recv: 46 88 
22:13:11.7043 Timeout
22:13:11.7293 => Send: 02 03 60 00 00 02 DA 38 
22:13:11.7593 <= Recv: 46 88 
22:13:11.7593 Timeout
22:13:11.7693 => Send: 02 03 60 00 00 02 DA 38 
22:13:11.7993 <= Recv: 85 9B 
22:13:11.7993 Timeout
22:13:11.8093 => Send: 02 03 60 00 00 02 DA 38 
22:13:11.8393 <= Recv: 85 9B 
22:13:11.8393 Timeout
22:13:11.8643 => Send: 03 03 50 12 00 02 74 EC 
22:13:11.8943 <= Recv: 85 9B 
22:13:11.8943 Timeout
22:13:11.9043 => Send: 03 03 50 12 00 02 74 EC 
22:13:11.9343 <= Recv: 68 5B 
22:13:11.9343 Timeout
22:13:11.9443 => Send: 03 03 50 12 00 02 74 EC 
22:13:11.9743 <= Recv: 68 5B 
22:13:11.9743 Timeout
22:13:11.9993 => Send: 03 03 60 00 00 02 DB E9 
22:13:12.0293 <= Recv: 68 5B 
22:13:12.0293 Timeout
22:13:12.0393 => Send: 03 03 60 00 00 02 DB E9 
22:13:12.0693 <= Recv: CA CB 
22:13:12.0693 Timeout
22:13:12.0793 => Send: 03 03 60 00 00 02 DB E9 
22:13:12.1093 <= Recv: CA CB 
22:13:12.1093 Timeout
22:13:12.1393 => Send: 01 03 50 02 00 06 75 08 
22:13:12.1693 <= Recv: CA CB 
22:13:12.1693 Timeout
22:13:12.1793 => Send: 01 03 50 02 00 06 75 08 
22:13:12.1793 <= Recv: 43 71 B3 33 43 
22:13:12.2193 <= Recv: 72 E6 66 70 
22:13:12.2193 Wrong telegram: ???
22:13:12.2293 => Send: 01 03 50 02 00 06 75 08 
22:13:12.5293 Timeout
22:13:12.5543 => Send: 01 03 50 12 00 02 75 0E 
22:13:12.5543 <= Recv: 43 71 B3 33 43 
22:13:12.5543 <= Recv: 72 E6 66 70 
22:13:12.5543 Wrong telegram: ???
22:13:12.5643 => Send: 01 03 50 12 00 02 75 0E 
22:13:12.8643 Timeout
22:13:12.8743 => Send: 01 03 50 12 00 02 75 0E 
22:13:12.9043 <= Recv: 4B E3 
22:13:12.9043 Timeout
22:13:12.9293 => Send: 01 03 60 0C 00 02 1A 08 
22:13:12.9593 <= Recv: 4B E3 
22:13:12.9593 Timeout
22:13:12.9693 => Send: 01 03 60 0C 00 02 1A 08 
22:13:12.9993 <= Recv: CC D8 
22:13:12.9993 Timeout
22:13:13.0093 => Send: 01 03 60 0C 00 02 1A 08 
22:13:13.0393 <= Recv: CC D8 
22:13:13.0393 Timeout
22:13:13.0643 => Send: 01 03 60 18 00 02 5A 0C 
22:13:13.0943 <= Recv: CC D8 
22:13:13.0943 Timeout
22:13:13.1043 => Send: 01 03 60 18 00 02 5A 0C 
22:13:13.1557 <= Recv: 72 00 
22:13:13.1557 Timeout
22:13:13.1657 => Send: 01 03 60 18 00 02 5A 0C 
22:13:13.1957 <= Recv: 72 00 
22:13:13.1957 Timeout
22:13:13.2207 => Send: 02 03 50 12 00 02 75 3D 
22:13:13.2507 <= Recv: 72 00 
22:13:13.2507 Timeout
22:13:13.2607 => Send: 02 03 50 12 00 02 75 3D 
22:13:13.2907 <= Recv: 46 88 
22:13:13.2907 Timeout
22:13:13.3007 => Send: 02 03 50 12 00 02 75 3D 
22:13:13.3307 <= Recv: 46 88 
22:13:13.3307 Timeout
22:13:13.3557 => Send: 02 03 60 00 00 02 DA 38 
22:13:13.3857 <= Recv: 46 88 
22:13:13.3857 Timeout
22:13:13.3957 => Send: 02 03 60 00 00 02 DA 38 
22:13:13.4257 <= Recv: 85 9B 
22:13:13.4257 Timeout
22:13:13.4357 => Send: 02 03 60 00 00 02 DA 38 
22:13:13.4657 <= Recv: 85 9B 
22:13:13.4657 Timeout
22:13:13.4907 => Send: 03 03 50 12 00 02 74 EC 
22:13:13.5207 <= Recv: 85 9B 
22:13:13.5207 Timeout
22:13:13.5307 => Send: 03 03 50 12 00 02 74 EC 
22:13:13.5607 <= Recv: 68 5B 
22:13:13.5607 Timeout
22:13:13.5707 => Send: 03 03 50 12 00 02 74 EC 
22:13:13.6007 <= Recv: 68 5B 
22:13:13.6007 Timeout
22:13:13.6257 => Send: 03 03 60 00 00 02 DB E9 
22:13:13.6557 <= Recv: 68 5B 
22:13:13.6557 Timeout
22:13:13.6657 => Send: 03 03 60 00 00 02 DB E9 
22:13:13.6957 <= Recv: CA CB 
22:13:13.6957 Timeout
22:13:13.7057 => Send: 03 03 60 00 00 02 DB E9 
22:13:13.7357 <= Recv: 0A CA CB 
22:13:13.7357 Timeout
22:13:13.7657 => Send: 01 03 50 02 00 06 75 08 
22:13:13.7957 <= Recv: CA CB 
22:13:13.7957 Timeout
22:13:13.8057 => Send: 01 03 50 02 00 06 75 08 
22:13:13.8057 <= Recv: 43 71 CC CD 43 
22:13:13.8447 <= Recv: 72 CC CD BF 07 
22:13:13.8447 Wrong telegram: ???
22:13:13.8547 => Send: 01 03 50 02 00 06 75 08 
22:13:13.8547 <= Recv: 43 71 CC CD 43 
22:13:13.8847 <= Recv: 72 CC CD BF 07 
22:13:13.8847 Wrong telegram: ???
22:13:13.9097 => Send: 01 03 50 12 00 02 75 0E 
22:13:13.9097 <= Recv: 43 71 CC CD 43 
22:13:13.9097 <= Recv: 72 CC CD BF 
22:13:13.9097 Wrong telegram: ???
22:13:13.9197 => Send: 01 03 50 12 00 02 75 0E 
22:13:14.2197 Timeout
22:13:14.2297 => Send: 01 03 50 12 00 02 75 0E 
22:13:14.2597 <= Recv: 4B E3 
22:13:14.2597 Timeout
22:13:14.2847 => Send: 01 03 60 0C 00 02 1A 08 
22:13:14.3147 <= Recv: 4B E3 
22:13:14.3147 Timeout
22:13:14.3247 => Send: 01 03 60 0C 00 02 1A 08 
22:13:14.3547 <= Recv: CC D8 
22:13:14.3547 Timeout
22:13:14.3647 => Send: 01 03 60 0C 00 02 1A 08 
22:13:14.3947 <= Recv: CC D8 
22:13:14.3947 Timeout
22:13:14.4197 => Send: 01 03 60 18 00 02 5A 0C 
22:13:14.4497 <= Recv: CC D8 
22:13:14.4497 Timeout
22:13:14.4597 => Send: 01 03 60 18 00 02 5A 0C 
22:13:14.4897 <= Recv: 72 00 
22:13:14.4897 Timeout
22:13:14.4997 => Send: 01 03 60 18 00 02 5A 0C 
22:13:14.5297 <= Recv: 72 00 
22:13:14.5297 Timeout
22:13:14.5547 => Send: 02 03 50 12 00 02 75 3D 
22:13:14.5847 <= Recv: 72 00 
22:13:14.5847 Timeout
22:13:14.5947 => Send: 02 03 50 12 00 02 75 3D 
22:13:14.6247 <= Recv: 46 88 
22:13:14.6247 Timeout
22:13:14.6347 => Send: 02 03 50 12 00 02 75 3D 
22:13:14.6647 <= Recv: 46 88 
22:13:14.6647 Timeout
22:13:14.6897 => Send: 02 03 60 00 00 02 DA 38 
22:13:14.7197 <= Recv: 46 88 
22:13:14.7197 Timeout
22:13:14.7297 => Send: 02 03 60 00 00 02 DA 38 
22:13:14.7597 <= Recv: 85 9B 
22:13:14.7597 Timeout
22:13:14.7697 => Send: 02 03 60 00 00 02 DA 38 
22:13:14.7997 <= Recv: 85 9B 
22:13:14.7997 Timeout
22:13:14.8247 => Send: 03 03 50 12 00 02 74 EC 
22:13:14.8547 <= Recv: 85 9B 
22:13:14.8547 Timeout
22:13:14.8647 => Send: 03 03 50 12 00 02 74 EC 
22:13:14.8947 <= Recv: 68 5B 
22:13:14.9166 Timeout
22:13:14.9266 => Send: 03 03 50 12 00 02 74 EC 
22:13:14.9566 <= Recv: 68 5B 
22:13:14.9566 Timeout
22:13:14.9816 => Send: 03 03 60 00 00 02 DB E9 
22:13:15.0116 <= Recv: 68 5B 
22:13:15.0116 Timeout
22:13:15.0216 => Send: 03 03 60 00 00 02 DB E9 
22:13:15.0516 <= Recv: CA CB 
22:13:15.0516 Timeout
22:13:15.0616 => Send: 03 03 60 00 00 02 DB E9 
22:13:15.0916 <= Recv: CA CB 
22:13:15.0916 Timeout
22:13:15.1216 => Send: 01 03 50 02 00 06 75 08 
22:13:15.1516 <= Recv: CA CB 
22:13:15.1516 Timeout
22:13:15.1616 => Send: 01 03 50 02 00 06 75 08 
22:13:15.1616 <= Recv: 43 71 B3 33 43 
22:13:15.2016 <= Recv: 72 E6 66 C3 C3 
22:13:15.2016 Wrong telegram: ???
22:13:15.2116 => Send: 01 03 50 02 00 06 75 08 
22:13:15.2116 <= Recv: 43 71 B3 33 43 
22:13:15.2416 <= Recv: 72 E6 66 C3 C3 
22:13:15.2416 Wrong telegram: ???
22:13:15.2666 => Send: 01 03 50 12 00 02 75 0E 
22:13:15.2666 <= Recv: 43 71 B3 33 43 
22:13:15.2666 <= Recv: 72 E6 66 C3 
22:13:15.2666 Wrong telegram: ???
22:13:15.2766 => Send: 01 03 50 12 00 02 75 0E 
22:13:15.5766 Timeout
22:13:15.5866 => Send: 01 03 50 12 00 02 75 0E 
22:13:15.6166 <= Recv: 4B E3 
22:13:15.6166 Timeout
22:13:15.6416 => Send: 01 03 60 0C 00 02 1A 08 
22:13:15.6716 <= Recv: 4B E3 
22:13:15.6716 Timeout
22:13:15.6816 => Send: 01 03 60 0C 00 02 1A 08 
22:13:15.7116 <= Recv: CC D8 
22:13:15.7116 Timeout
22:13:15.7216 => Send: 01 03 60 0C 00 02 1A 08 
22:13:15.7516 <= Recv: CC D8 
22:13:15.7516 Timeout
22:13:15.7766 => Send: 01 03 60 18 00 02 5A 0C 
22:13:15.8066 <= Recv: CC D8 
22:13:15.8066 Timeout
22:13:15.8166 => Send: 01 03 60 18 00 02 5A 0C 
22:13:15.8466 <= Recv: 72 00 
22:13:15.8466 Timeout
22:13:15.8566 => Send: 01 03 60 18 00 02 5A 0C 
22:13:15.8866 <= Recv: 72 00 
22:13:15.8866 Timeout
22:13:15.9116 => Send: 02 03 50 12 00 02 75 3D 
22:13:15.9416 <= Recv: 72 00 
22:13:15.9416 Timeout
22:13:15.9516 => Send: 02 03 50 12 00 02 75 3D 
22:13:15.9926 <= Recv: 04 
22:13:15.9926 Timeout
22:13:16.0026 => Send: 02 03 50 12 00 02 75 3D 
22:13:16.3026 Timeout
22:13:16.3276 => Send: 02 03 60 00 00 02 DA 38 
22:13:16.3576 <= Recv: 46 88 
22:13:16.3576 Timeout
22:13:16.3676 => Send: 02 03 60 00 00 02 DA 38 
22:13:16.3976 <= Recv: 85 9B 
22:13:16.3976 Timeout
22:13:16.4076 => Send: 02 03 60 00 00 02 DA 38 
22:13:16.4376 <= Recv: 85 9B 
22:13:16.4376 Timeout
22:13:16.4626 => Send: 03 03 50 12 00 02 74 EC 
22:13:16.4926 <= Recv: 85 9B 
22:13:16.4926 Timeout
22:13:16.5026 => Send: 03 03 50 12 00 02 74 EC 
22:13:16.5326 <= Recv: 68 5B 
22:13:16.5326 Timeout
22:13:16.5426 => Send: 03 03 50 12 00 02 74 EC 
22:13:16.5726 <= Recv: 68 5B 
22:13:16.5726 Timeout
22:13:16.5976 => Send: 03 03 60 00 00 02 DB E9 
22:13:16.6276 <= Recv: 68 5B 
22:13:16.6276 Timeout
22:13:16.6376 => Send: 03 03 60 00 00 02 DB E9 
22:13:16.6676 <= Recv: CA CB 
22:13:16.6676 Timeout
22:13:16.6776 => Send: 03 03 60 00 00 02 DB E9 
22:13:16.7076 <= Recv: CA CB 
22:13:16.7076 Timeout
22:13:16.7376 => Send: 01 03 50 02 00 06 75 08 
22:13:16.7686 <= Recv: CA CB 
22:13:16.7686 Timeout

martin_triska

Hello @pascal37,

thank you for the information sent. Unfortunately, based on it, we are not able to determine the exact source of the issue.
Can you please check the following:

Again, make sure you have the latest firmware in the I/O board flashed. Type:

/opt/unipi/tools/fwspi -i 0

Can you confirm that the i2c-0 error in the dmesg log appears always in the moment of communication goes wrong? It seems there is a correlation (863475 seconds is almost exactly 10 days).
Are you sure that no other running processes access the interface (/dev/ttyNS0)? Please check also your time-scheduled tasks (e.g. executed from cron).
Are you sure that no noise or interference from the environment or power supply can infiltrate the unit?

As the last resort, we are able to simulate your application with the same HW, but it will be pretty time-consuming. In the meantime, please use another RS485 interface.

pascal37

Hi @martin_triska,

I did a reflash (USB method) of my M525 with latest Mervis OS image (2.3.0.10), firmwares are already up-to-date:

unipi@M525-sn8:~$ sudo /opt/unipi/tools/fwspi -i 0
Boardset:     0 B-1000                         (v1.0)
Baseboard:    0 B-1000                         (v2.0)
Firmware: v5.50
unipi@M525-sn8:~$ sudo /opt/unipi/tools/fwspi -i 1
Boardset:    19 E-4Ai4Ao_U-R485Di4Ro5          (v1.0)
Baseboard:   11 E-4Ai4Ao                       (v1.0)
Firmware: v5.50

From what I've seen after serial communications "crashes", I did not see anymore an i2c-0 error in demsg, so I am not sure this is correlated.
Yes: it is a standard Mervis OS image + 2 pythons scripts running as daemons with pymodbus client over ModbusTCP, so no serial communications there.
Yes: I am using shielded cable specific for RS485, the bus is properly terminated at each end. Cable shield is properly grounded at PLC side and also to PLC itself. Environment is stable and, so long the issue is not occurring, communication is working perfectly.
With the increase of Max telegram duration to 300ms, there is absolutely no error during multiple days:

It seems most likely a driver or Mervis runtime issue related to M525 model and port 1.1 !?
As I said, I never had the issue when using port 1.3 instead of port 1.1
I have also another RS485 bus on port 2.1 and no issue either.
This issue also never occurred with port 1.1 on my previous M505 PLC

If you want to try reproduce the issue, you need 3 Inepro PRO380 meters and read constantly multiple values from them like power, energy, voltage.
In my case, the maximum uptime before a crash is about 10 days.

I can use port 1.3 in the meantime, but I will need the 3rd RS485 port later for my project...
But let's check again if it is really stable with port 1.3 over >10 days !

Many thanks in advance and kind regards

pascal37

Hi @martin_triska,

I confirm the solution is stable over >50 days when using port 1.3 :

... but not with port 1.1, where serial communications crashes < 10 days !
If you did not find a software issue related to port RS485-1.1 on Axon M525, there should be a hardware issue with my PLC !

I have bought this PLC because I need 3 RS485 ports, so using port 1.3 over 1.1 is not a solution...

Can you please replace this unit ?
Many thanks in advance and kind regards,

Pascal

martin_triska

Hello @pascal37,

thank you for the feedback. So, let us sum it up:

On ports other than 1.1, you've never seen the issue yet (tested over 50 days without reboot).
On port 1.1, the issue appears always in a few (approx. 10) days.
There is no correlation with any item in the system log (dmesg) recorded in the moment of failure.
Flash to the latest version 5.50 has not solved it.
An older PLC M505 does not have this problem (is the FW and image versions in it also the latest?)

If so, you are right - it seems to the issue is hardware-based and the unit has to be replaced.

pascal37

Hi @martin_triska,

Yes, that sums up the situation well, so I'll write to support@unipi.technology for replacement request.

Many thanks,

Pascal