• Aishwarya Pothula

How to pass agent Observations from Unity to Python

Updated: Sep 15, 2021

Custom Side Channel of MLagents (Release 3)

To exchange data between C# (Unity) and Python (API) we need a side channel. While there are specific side channels for datatypes such as FloatPropertiesChannel, a custom side channel allows for the exchange of custom data structures.

While following the official documentation for creating a custom side channel by Unity, this blog highlights my use of custom side channel to communicate a floatlist from C# to Python and a String from Python to C#.

The most common errors while implementing side channels are due to version differences. Make sure to follow the releases document to ensure the use of compatible versions.

For the purpose of this blog, I have used MLAgents Release 3. For compatibility with this release I have used the following packages and corresponding versions.

To download com.unity.ml-agents and communicator, mlagents package inside Unity is to be installed. To install the ml-agents, ml-agents-envs and gym-unity use the following command in the terminal.

pip3 install mlagents == 0.17.0 

For my example, on the Unity side I created the following scripts

1. Created the FloatChannel class in my Unity project that contains methods to both listen to messages from Python and send messages to Python.

The FloatChannel class has to implement the SideChannel abstract class.

OnMessageReceived(IncomingMessage msg) method is implemented to read data from IncomingMessage.

The ChannelID, which is used to uniquely identify a channel, should be assigned in the constructor.

An OutgoingMessage instance is created, followed by a call to QueueMessageToSend(msg) in order to send messages to Python.

Please note that I have used msgOut.WriteFloatList because that is the data structure I want to send. Other datastructure can be float, int, string etc.

With this the custom side channel class has been created.

2. The next step is to register and instantiate the side channel class. The step can be performed wherever you need to pass/get values from the side channel. It needs to be ensured that this script is attached to an agent.

In this script, I am putting the three co-ordinates of collision points in a list that I want to send to Python. Since I need to access the side channel, I have instantiated the channel and registered the custom side channel class here by calling SideChannelsManager.RegisterSideChannel. Please note that in the Unity documentation SideChannelManager.RegisterSideChannel is mentioned instead of SideChannelsManager.RegisterSideChannel.

By calling floatChannel.SendDebugStatementToPython, I am sending the coordinates list to the method in FLoatChannel responsible for transmitting to Python.

For my example, on the Python side I created the following script

1. Imports

2. Creating the FloatSideChannel class which consists methods responsible for both listening to messages from C# and also sending messages to c#.

Please note that msg.read_float32_list and msg.write_string have been used to represent the data that they are receiving and sending, respectively.

Ensure that the same ChannelID used in C# is used in Python too.

3. Instantiate the side channel class and load the environment.

I assigned file_name as None so that I can run the Unity environment without using a build. After running this block, I play the Unity environment.

4. Write the program logic

2 views0 comments

Recent Posts

See All

Exporting Individual Fbx Files from Motive Recordings The goal of this blog is to understand how to export individual mocap files from multi-skeleton mocap Motive videos and apply them to a model. You