Coprocessor management troubleshooting grid

Some typical issues related to the management of a coprocessor are listed below. Solutions or debugging methods are proposed for these issues.

If your issue is not listed, try looking at in the articles in the Coprocessor management Linux, Coprocessor management STM32Cube or troubleshooting grids categories.

1. Coprocessor firmware loading and control[edit source]↑

Symptom

Resolution

The coprocessor traces are not available on the Linux side

 cat  /sys/kernel/debug/remoteproc/remoteproc0/trace0
No such file or directory

This may happen for two reasons:

The firmware does not include any resource table or the resource table does not define any trace.

Update the firmware resource table and rebuild the firmware

The ".resource_table" section is empty or not defined in the elf file.

Use following command to verify it in elf file generated :

  readelf -l <elf file>
 ...
 02     .data .resource_table .bss ._user_heap_stack
 ....

In case starting the coprocessor from the boot loader (u-boot) :

"unsupported fw ver: 0
Remote Processor 0 resource table Not found : 0x00000000-0x0 "

The firmware does not include any resource table. This is only a warning that do not prevent the firmware to start properly. "rproc load_rsc" step can be bypassed

2. Inter processor communication[edit source]↑

Symptom	Resolution
Frozen firmware as consequence of a deadlock in OpenAMP during IP communication withe the main processor.	Issue probably comes from rpmsg_virtio_rx_callback or rpmsg_virtio_send_offchannel_raw (rpmsg_virtio.c) functions which is called in interrupt context. They are using a mutex lock in rpmsg_device struct when accessing the index of the virtio queue index. Rework your code so theses functions are not called in interrupt context.
Linux kernel trace: stm32-ipcc 4c001000.mailbox: Try increasing MBOX_TX_QUEUE_LEN	On each IPCC interrupt, the coprocessor treats all the RPMsg buffered (one IPCC event for several RPMsg). On Linux Side one IPCC signal is programmed for each RPMsg sent. This can result is an overflow warning on Linux because too many IPCC events are queued. No RPMsg are dropped, but this message could be interpreted as the coprocessor reaches its capacity to treat the received messages in time. Consider to rework the code so the coprocessor to process messages more efficiently, or to decrease the message rate sent by the Linux.
Linux kernel trace ( example with rpmsg_tty driver): rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: timeout waiting for a tx buffer	This message means that there is no more TX buffer available to transmit a message to the remote processor. This may happen for two reasons: The firmware implementation is incomplete and the coprocessor does not process any IPCC interrupt (interrupt disabled, interrupt handler not defined, ...). Fix the code of the firmware : refer to IPCC_internal_peripheral for detail on the peripheral. The coprocessor is busy, frozen or crashed. This can be confirmed by a debug tool analysis.
Linux kernel trace: rpmsg_tty virtio0.rpmsg-tty-channel.-1.0: No memory for tty_prepare_flip_string	There is no more space in the TTY temporary buffer on reception. The root cause is probably that the Linux application does not read the tty device in time to treat the incoming TTY stream. Consider to rework the Linux application so it takes less time to process messages, or decrease the message exchange frequency.
Linux kernel trace: remoteproc remoteproc0: stm32_rproc_kick: failed (<mbx>, <error value>)	The Linux remoteproc driver can't use the IPCC mailbox which may happen for two reasons: The Linux kernel is built without the support of the stm32 IPCC mailbox: enable it, see IPCC configuration. The mailboxes are not / incorrectly defined in the Linux kernel DeviceTree: fix it as described in mailbox DeviceTree.