As we had major schedule pressure to get things working, we ended up putting fans to push air through Neuron. This cooled down things so that CPU temp reported by Pi was around 50-55 C during our process. Neuron is installed inside electric cabin and air will circulate inside this cabin. But as cabin is made of steel, it will most likely provide enough radiation cooling. Cabin is installed to clean room environment with ambient temperature around ~22 C. This cooling however did not help to relay switching off.

We ended up making calls to command relay closed in few places during our process and when process is in standby, we command relay closed every 30 seconds. This workaround allowed us to proceed with process. Of course, we still do not know what the root issue is and would like to know that. Since the device now installed in clean room environment, debugging it is way harder. As for load, if I recall correctly (can't access the device remotely), it's around 2. So quite high. We are running closed loop control and drawing graph to remote UI while maintaining local UI also. So quite a lot of tasks for Pi. For some devices where we have
also used Neuron we made an C++ motion-core that keeps closed loop control for motion and separate process and UI program that talks via sockets to that motion-core. For this particular device where relay is opened everything is run in same program and we have double checked as best we can that there are no calls to open this particular relay. Installation was from latest version of Evok as of now.

Hello

We have a process control system where we control few relays among other things. One of the relays is switching off after random time. It's randomly switched off after 5 minutes or after 10 minutes. Sometimes it works fine for a longer period of time but usually it opens itself automatically after a while. We control the relay (set to closed) only once during startup of the process and then it should remain closed. Another relay that stays on for a shorter time works ok.

We don't have anything in our software that should cause it to be opened again.

We are thinking that this could be due to temperature, as we are noticing high temperatures in the CPU. In fact we have high temperature alarm (CPU in 80-85 C range) in the Raspberry Pi LCD that is connected to UniPi.

Any suggestions what to try next ? Unipi model is Neuron M503.

Henkka

@tomas_knot

Thanks, I will try that.

How could I ask for input states through websocket, without using "all" query message? For example ideally I would like to ask about the state of one spesific input ? Or perhaps state of all inputs.

--
Henkka

I'm wondering what is the proper way to set websocket response filters ? I have tried to set them with command "filter", parameters "devices" as "ai,input" as I read from /opt/evok/evok.py but still I'm not getting input values as websocket messages on every message.

  1 import websocket
  2 import time
  3 import json
  4
  5 url = "ws://127.0.0.1/ws"
  6
  7 def on_message(ws, message):
  8     print(message)
  9     print("---")
 10
 11 def on_error(ws, error):
 12     print error
 13
 14 def on_close(ws):
 15     print "Connection closed"
 16
 17 def on_open(ws):
 19         ws.send('{"cmd":"filter","devices":"ai,input"}')
 20         #ws.send('{"cmd":"all"}')
 21
 22 ws = websocket.WebSocketApp(url, on_message = on_message, on_error = on_error, on_close = on_close)
 23 ws.on_open = on_open
 24 ws.run_forever()

Reason I'm asking is because I have an button connected to one input and I would like to see it's counter value on every message. I have a certain point in my program that waits for user to press button. Now it seems that I'm missing some button presses since sometimes button press is not noticed, eg. no websocket message is ever received with increased button count. However if I press button long enough, then I get a valid websocket message with an increased counter. Usually this counter is then advanced by more than 1, say from 50 to 52 and thus I'm missing button press 51.

--
Henkka

@tomas_knot

Hi,

Thanks, I think now it's clear.

Right after boot I tried multiple times same command, with only few seconds in between;

Results are somehow changing;

pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
Binary file /sys/bus/i2c/devices/1-0057/eeprom matches
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
Binary file /sys/bus/i2c/devices/1-0057/eeprom matches
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
Binary file /sys/bus/i2c/devices/1-0057/eeprom matches
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom

See how last 3 lines there is suddenly no eeprom available ? So something fishy about i2c bus perhaps.

I then went on to think about connecting oscilloscope to see what's going on and what we have done so far. When thinking about where to connect it, it hit me - we have an external 7" Pi LCD connected to Raspberry Pi Display connector and it has those 2 extra SDA and SCL pins... Could they be reason for this... They are there for A and B model Pi's that don't have these integrated to display-cable and as this is version 3 Pi, they should not be connected as those are routed via display cable. So I disconnected them and after boot so far it has performed flawlessly!

Below are results of the same "test" as above, this time results are all predictable and ok, every time there is an eeprom device on the bus.

pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
Binary file /sys/bus/i2c/devices/1-0057/eeprom matches
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
Binary file /sys/bus/i2c/devices/1-0057/eeprom matches
pi@f:~ $ sudo grep M503 /sys/bus/i2c/devices/1-0057/eeprom
Binary file /sys/bus/i2c/devices/1-0057/eeprom matches

So, this was my error, not thinking that connection of LCD could anyway be in part of this... Sorry about that.

But many thanks for your excellent support! For me at least, I think the issue is now solved (fingers crossed...).

@tomas_knot

Hello, thanks for the reply.

Still no luck but few logbook entries what I have tried so far.

This is now my /etc/systemd/system/evok.service:

[Unit]
Description=Evok Modbus/Websocket/Rpc Server
#Requires=pigpio.service
Requires=neurontcp.service

[Service]
ExecStartPre=/bin/sleep 15
Type=simple
ExecStart=/usr/bin/python /opt/evok/evok.py
TimeoutStopSec=3
SyslogLevel=debug
SyslogIdentifier=evok
StandardOutput=null
WorkingDirectory=/opt/evok

#User=evok
#Group=evok

Restart=always

[Install]
WantedBy=multi-user.target
#Alias=evok.service

but still no luck, it still doesn't want to start the evok web-configuration and/or evok software at the boot of the pi.

I then added even more time, 30 seconds, and logged into Pi right after boot via ssh. From logs I can definitely verify that evok is trying to start after 30 seconds and this is what we get as a first rows in evok.log:

2017-11-27 15:42:19,697 - evok - ERROR - 'model'
Traceback (most recent call last):
  File "/opt/evok/neuron.py", line 180, in readboards
    yield board.parse_definition(self.hw_dict, i)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 307, in wrapper
    yielded = next(result)
  File "/opt/evok/neuron.py", line 560, in parse_definition
    if defin and defin['type'] in config.globals['model']:
KeyError: 'model'

I tried to restart evok service but it didn't this time help either.

So I went on to check other services you mentioned and saw that every time I try to restart evok service, I get following lines to nginx error.log:

2017/11/27 15:51:35 [error] 587#587: *22 connect() failed (111: Connection refused) while connecting to upstream, client: 10.73.73.205, server: _, request: "GET /ws HTTP/1.1", upstream: "http://[::1]:8080/ws", host: "10.73.73.237"
2017/11/27 15:51:35 [error] 587#587: *22 connect() failed (111: Connection refused) while connecting to upstream, client: 10.73.73.205, server: _, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "10.73.73.237"
2017/11/27 15:51:36 [error] 587#587: *25 no live upstreams while connecting to upstream, client: 10.73.73.205, server: _, request: "GET /ws HTTP/1.1", upstream: "http://localhost/ws", host: "10.73.73.237"
2017/11/27 15:51:37 [error] 587#587: *26 connect() failed (111: Connection refused) while connecting to upstream, client: 10.73.73.205, server: _, request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:8080/ws", host: "10.73.73.237"
2017/11/27 15:51:37 [error] 587#587: *26 connect() failed (111: Connection refused) while connecting to upstream, client: 10.73.73.205, server: _, request: "GET /ws HTTP/1.1", upstream: "http://[::1]:8080/ws", host: "10.73.73.237"
2017/11/27 15:51:38 [error] 587#587: *29 no live upstreams while connecting to upstream, client: 10.73.73.205, server: _, request: "GET /ws HTTP/1.1", upstream: "http://localhost/ws", host: "10.73.73.237"

Don't know if this matters or not.

Finally, I managed to get it running again by restarting nginx and restarting evok. I can certainly test any other remedies you come up if needed.

@tomas_knot
Hi, thanks for the reply

1. check, ok.
2. I'm missing device_tree_param. I have only lines below active, this is standard from evok-installation sd-image, from what I can recall we haven't modified this file.

/boot/config.txt active lines:

gpu_mem=32
dtparam=i2c_arm=on,watchdog=on
dtoverlay=i2c-rtc,mcp7941x
dtoverlay=neuron-spi-new
dtoverlay=neuronee
core_freq=250
dtoverlay=pi3-disable-bt

For me, it is working if we restart the evok service but not directly after reboot of the pi.

@tomas_knot Hi,

Thanks for reply!

Is there any way to upgrade existing previous image installation, even by hand installing packages or getting an new evok from git etc. ? We have quite a few other packages installed and configured already to our device that should be soon in production and it would be great if there is an way to upgrade without re-install.

I'm having similar issues with a fresh install of UniPian-Neuron-OS-2017-11-21.zip from https://files.unipi.technology/s/public?path=%2FSoftware%2FOpen-Source Images . UniPi control panel does not diplay devices and the log of evok starting shows this:

2017-11-27 06:33:51,904 - evok - ERROR - 'NoneType' object has no attribute 'do_scan'
Traceback (most recent call last):
File "/opt/evok/neuron.py", line 217, in scan_boards
yield self.modbus_cache_map.do_scan()
AttributeError: 'NoneType' object has no attribute 'do_scan'
2017-11-27 06:33:52,407 - evok - ERROR - 'NoneType' object has no attribute 'do_scan'
Traceback (most recent call last):
File "/opt/evok/neuron.py", line 217, in scan_boards
yield self.modbus_cache_map.do_scan()
AttributeError: 'NoneType' object has no attribute 'do_scan'

I found an similar issue from here:

https://forum.unipi.technology/topic/99/evok-service-startup-problem-during-cold-boot

Also similar issue here:

https://forum.unipi.technology/topic/488/internal-error-500-after-fresh-evok-installation/8

If I do an restart of the service evok right after Pi boots (with service evok restart), then it will not work, it does not recognice inputs and outputs. If I wait for a while and then do an restart, then it works ?